Commit Graph

362 Commits

Author SHA1 Message Date
Vitalii
149b59069f satellite/{web, console}: removed old object flow
Removed old flow along with a feature flag

Issue:
https://github.com/storj/storj/issues/5166

Change-Id: I85cfabbf94e910c9d5efb01ef850f6b888a13f71
2022-10-20 16:27:50 +03:00
Cameron
a52f766273 satellite/overlay: add email-sending functionality to overlay service
We want to send emails to SNOs. Node status changes go through the
overlay service, so it's a good place to add the mail service.
Add the mailservice.Service, satellite address, and satellite name to
overlay service. Also add feature flag --overlay.send-node-emails

Change-Id: I3bd2cb3bf22f9724954ce2374f8b651b902b3a24
2022-10-13 18:01:05 +00:00
Márton Elek
ea106f8894 go.mod: bump storj/private
Change-Id: I087b39c267c6492c4a5c85f470631b2f206cdf39
2022-10-13 11:23:57 +00:00
Fadila Khadar
02924d0ded satellite/metainfo: temporary feature flag for listing query testing
Fixes: https://github.com/storj/storj/issues/5144

Change-Id: I7650f4d5dd0378e2246339e79710a695996a845c
2022-09-28 17:46:52 +00:00
dlamarmorgan
88328ab240 satellite/payments/billing/billing.go: change default loop interval
Change the default loop interval for querying for new payments and adding them into the billing table from 1 minute to 15 seconds.

Change-Id: I26cf4a764cbe1de4c9b839ad60352374d8231522
2022-09-27 11:42:28 -07:00
dlamarmorgan
0b468aca56 satellite/payments/storjscan/storjscan.go: change default block confirmations
Change the default number of required block confirmations for a payment to be confirmed from 12 to 15.

Change-Id: I44c258134c293e7691623bc00c504130aa69a96a
2022-09-23 14:30:05 +00:00
Michal Niewrzal
4e71dd302a satellite/{metainfo,metabase}: prepare feature flag MultipleVersions
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.

Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
2022-09-21 13:07:13 +00:00
Erik van Velzen
e6b5501f9b satellite/gc/sender: new service to send retain filters
Implement a new service to read retain filter from a bucket and
send them out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Paralellism (setting ConcurrentSends) and end-to-end garbage collection
tests will be restored in a subsequent commit.

Solves https://github.com/storj/team-metainfo/issues/121

Change-Id: Iaf8a33fbf6987676cc3cf74a18a8078916fe673d
2022-09-20 11:49:40 +00:00
Artur M. Wolff
5150ee5001 mod: bump storj.io/private
Change-Id: I534aeedead1f1a2c5099e80b49fa5a8ea599fac8
2022-09-19 13:09:31 +00:00
Michal Niewrzal
158eb2381e satellite/gc/bloomfilter: uploading bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket.

This change add main logic to new service. After collecting all bloom
filters with segment loop and piece tracker all filters are marshaled
and packed into zip files. Each zip contains up to "ZipBatchSize" bloom
filters and it's uploaded to specified in configuration bucket.

All uploaded objects have specified expiration time to not delete them
manually.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2b6bc02a7dd7c3a639e75810fd013ae4afdc80a2
2022-09-12 08:33:53 +00:00
Michal Niewrzal
68f6d93f29 satellite/gc/bloomfilter: add service to collect bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This this initial change to add such
service. Added service is joining segment loop and collects all
bloom filters.

Sending bloom filters to the bucket will be added as a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2551723605afa41bec84826b0c647cd1f61f3b14
2022-09-02 08:10:46 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00
paul cannon
0dcc0a9ee0 satellite/reputation: reconfigure lambda and alpha
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.

For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at

    https://forum.storj.io/t/tuning-audit-scoring/14084

In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.

Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.

Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
2022-08-17 18:52:53 +00:00
dlamarmorgan
f596f72f44 satellite/payments/storjscan: add billing update to chore
Update chore to add confirmed transactions to the billing table.

Change-Id: I3c9a98c2ddc96f8a8905250376a1f5490d810277
2022-08-16 11:18:15 -07:00
Vitalii
0ec28ca623 web/satellite: add token funds modal
Created new modal which shows user their native STORJ token wallet address.
There are QR and copy buttons.
It will be used only in new billing screen.

Change-Id: Icef3c8668c548b779c07fe2b85eb5761cd1221a3
2022-08-15 14:17:30 +03:00
Moby von Briesen
3f6bf64af9 satellite/console: Default NewProjectDashboard to "true"
We have enabled the new project dashboard in production. Change the
default to true so that we do not need an explicit configuration in
prod.

Change-Id: I0f93773965283e7b0682f6586685224281cbf78c
2022-08-06 01:00:31 +00:00
Vitalii
ad37ea4518 satellite/{web, console}: login captcha implemented
Implemented Recaptcha and Hcaptcha for login screen.
Slightly refactored registration page implementation.
Made 2 different login/registration captcha configs on server side to easily swap between captchas independently.

Issue: https://github.com/storj/storj/issues/4982

Change-Id: I362bd5db2d59010e90a22301893bc3e1d860293a
2022-08-03 23:02:27 +00:00
Jeff Wendling
f343b856dd go.mod: bump common/private/monkit/monkit-jaeger
this includes all of the fixes identified for end to end
tracing to finally work.

Change-Id: Ia433a2cf9130dc43b6810379af32f5c10f9d3888
2022-07-27 16:17:00 +00:00
Qweder93
2b24632610 {satellite/metainfo, satellite/accounting}: remove ValidateSegmentLimit config value and check
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now

Resolves: https://github.com/storj/storj/issues/4470

Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
2022-07-27 15:36:53 +00:00
Egon Elbre
82fede2132 satellite/satellitedb: batch delete storage node tallies
Currently we have a significant number of tallies that need to be
deleted together. Add a limit (by default 10k) to how many will
be deleted at the same time.

Change-Id: If530383f19b4d3bb83ed5fe956610a2e52f130a1
2022-07-20 15:42:35 +03:00
paul cannon
2f20bbf4d8 satellite/reputation: add a reputation write cache
This should lower the amount of database load coming from
reputation updates.

Change-Id: Iaacfb81480075261da77c5cc93e08b24f69f8949
2022-07-14 21:40:16 +00:00
Yaroslav Vorobiov
de5951331a satellite: wire storjscan chore to core process
Adds storjscan chore and storjscan client to satellite chore.
Storjscan chore interval added to storjscan config.

Change-Id: I13bb8d83b14ca90f4735c9a10c361ad24a56b256
2022-07-14 15:07:52 +00:00
Vitalii
69dc9a4731 satellite/console: added new email which is sent on unknown password reset
Added new email html template.
It is sent when user tries to reset password with unknown or unverified account.
Made a couple of minor config changes.

Issue: https://github.com/storj/storj/issues/4913

Change-Id: I730f48b3478e302d1e38e1f8a27c75f66a8ba6fd
2022-07-14 14:32:59 +00:00
littleskunk
cf750716ea
satellite/console: enable new access grant flow (#4934) 2022-06-30 00:06:39 +02:00
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Jeremy Wharton
58c5d44f44 satellite/console: integrate sessions into satellite UI
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.

Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
2022-06-13 08:02:02 +00:00
Yaroslav Vorobiov
62774768d9 satellite/payments/storjscan: update config and wrap service errors
Change-Id: I4f75375e9499119cbcce78c5e77d1950e4bae38b
2022-06-07 14:21:42 +00:00
Paul Willoughby
911cc1e163 satellite/contact: reject privateIPs in PingMe and CheckIn endpoints
prevent network enumeration by rejecting privateIPs in PingMe and
Checkin endpoints

Closes storj/storj-private#32

Change-Id: I63f00483ff4128ebd5fa9b7b8da826a5706748c9
2022-06-07 08:09:14 +00:00
Vitalii
ba58530089 {satellite}/web,console,testsuite: remove old navigation structure
Removed old satellite UI navigation structure.
Removed old feature flag.

Change-Id: Ic998886cf2e30ebd44e67a20fc53888103fe4b8d
2022-06-06 16:43:05 +03:00
dlamarmorgan
270204f352 satellite/{payments/storjscan,satellitedb}: Add wallet implementation
Add storjscan wallets implementation to the satellite. The wallets interface allows you to add and claim new wallets as called by the API. The storjscan specific implementation of this interface uses a wallets DB to associate the user to a wallet address, as well as a storjscan client to request and associate new wallets to the satellite.

Change-Id: I54081edb5545d4e3ee07cf1cce3d3e87cc00c4a1
2022-06-03 11:45:47 +00:00
cl-mitch
cbaca8b17e
web/satellite added new billing screen feature flag (#4836)
New feature flag implemented to enable the new billing flow.

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-06-02 09:30:27 -05:00
littleskunk
6cdd250019
satellite/console: enable new object flow / onboarding (#4851) 2022-06-01 14:04:03 +02:00
littleskunk
3d1b469f35
satellite/console: disable account activation reminder email (#4844)
* satellite/console: disable account activation reminder email
2022-05-31 17:36:40 +02:00
Stefan Benten
8673583582 satellite/admin: fix console config handling
An older change plummed the full console config as subconfig of
the admin api configuration in. This bloated the generated satellite
configuration unnecessarily while also allow for confusion/mistakes.

Change-Id: Icf49cc1f147711e37e85f6eac1143fab8ddf1659
2022-05-27 22:26:06 +00:00
Cameron
87f6a3dcda {web/satelliite, satellite/console}: logo redirects to homepage
logo redirects to homepage on login, signup, forgot password, reset
password, and activate account pages

Change-Id: I992aeae197004d620addd8d515cae1c1ca80a778
2022-05-23 16:26:19 -04:00
NickolaiYurchenko
99237d5c78 web/satellite: new bucket creation flow
old bucket creation flow removed
new flow added
name and passphrase splitted into separate views
demo bucket will not be created automatically
bucket creation progress bar added

Change-Id: I2a1d7d77c3038caaafb3c06bdb0ac5dd1ad17599
2022-05-22 11:55:55 +00:00
Malcolm Bouzi
087e57d037 web/satellite: create chore that will resend verification emails to unverified users
We want to remind unverified users to verify their emails:
once after 24 hours has passed and again after 5 days has passed.

Add mailservice.Service to satellite core because it is needed by the
chore for sending emails. To add the mailservice.Service to the core,
we create a helper function in satellite/peer.go to avoid duplicating
the code in both api.go and core.go. In addition to the chore, this
change adds methods to users.DB to get unverified users in need of
reminder.

Change-Id: I4e515bdf43f922788b4f965b2efb34fa32288bd1
2022-05-18 08:08:33 +00:00
JT Olio
5fb9ee3cfa uplink, satellite: use bgp hostnames
Change-Id: I58f5011d3019f8267fa8cbd3096b2cfe42eb5f8b
2022-05-16 18:14:37 +00:00
Cameron
0633aca607 satellite/console: create new consoleauth service
We want to send email verification reminders to users from the satellite
core, but some of the functionality required to do so exists in the
satellite console service. We could simply import the console service
into the core to achieve this, but the service requires a lot of
dependencies that would go unused just to be able to send these emails.

Instead, we break out the needed functionality into a new service which
can be imported separately by the console service and the future email
chore.

The consoleauth service creates, signs, and checks the expiration of auth
tokens.

Change-Id: I2ad794b7fd256f8af24c1a8d73a203d508069078
2022-05-13 16:27:07 +00:00
Moby von Briesen
763bfc0913 satellite/console,web/satellite: Implement hCaptcha
Adds a new configuration for hcaptcha enabled, secretkey, and sitekey.
If both reCAPTCHA and hCaptcha are configured as "enabled", reCAPTCHA
will be used.

Change-Id: I73cc6e133d8da3555e0ed8b2b377cf9eb263e6dc
2022-05-13 14:57:45 +00:00
Erik van Velzen
db1cc8ca95 satellite/repair/checker: buffer repair queue
Integrate previous changes. Speed up the segment loop by batch inserting
into repair queue.

Change-Id: Ib9f4962d91960d21bad298f7771345b0dd270276
2022-05-12 16:28:05 +00:00
Vitalii
dedccbd2e4 satellite/console, web/satellite: limit failed login attempts
Added account locking on 3 or more login attempts.
Includes both password and MFA failed attempts on login.
Unlock account on successful password reset.

Change-Id: If4899b40ab4a77d531c1f18bfe22cee2cffa72e0
2022-05-11 14:49:11 +00:00
Qweder93
384118ee1e satellite/metabase: use pgutil.NullTimestampTZArray
This code cleanup to use helper DB method.

Change-Id: Idc438234cb4f7da23d6ade2d69f5d889f15e9009
2022-05-03 19:30:21 +00:00
hovex023
58f957bd1d
web/satellite: Add new access grants flow (#4765)
* Added new feature Flag for new Access Grant Flow. 

* Added 3 cards to access grant view for S3, CLI and Access grant to replace old header

* Added new formatting, text and Icon for Access Grant Delete Popup modal
2022-04-29 10:31:52 -05:00
nadimhq
dd6aac9b39
satellite/metainfo: enable server-side copy in production (#4708)
these changes enable server-side copy in production
2022-04-26 09:58:16 -04:00
Moby von Briesen
ed5ebb2527 satellite: Rename "acct mgmt api" to "rest api"
"REST API" is a more accurate descriptor of the generated API in the
console package than "account management API". The generated API is very
flexible and will allow us to implement many more endpoints outside the
scope of "account management", and "account management" is not very well
defined to begin with.

Change-Id: Ie87faeaa3c743ef4371eaf0edd2826303d592da7
2022-04-25 18:51:46 +00:00
Michał Niewrzał
6188c4c0dd satellite/metainfo: enable segment limit by default
Change-Id: I26cc0bbd97d6cc2ffc96213788f261efb749d028
2022-04-14 06:12:09 +00:00
Cameron
84b522bc06 satellite/console: create account management api keys service
We are in the process of creating an api to allow users to manage their
accounts programmatically. We would like to use api keys for
authorization. We were originally going to create an entirely new table
for these api keys, but seeing as we already have 2 other tables for
keys/tokens, api_keys and oauth_tokens, we thought it might be better to
use one of these. We're using oauth_tokens.

We create a new oidc.OAuthTokenKind for account management api keys:
KindAccountManagementTokenV0. We made the key versioned because we
likely want to improve the implementation in the future, but we want to
get something functional out the door ASAP because the account management
api feature is highly desired.

Add a new method to oidc.OAuthTokens interface for revoking v0 account
management api keys, RevokeAccountManagementTokenV0. Add update method
to dbx implementation to allow updating the expiration. We will revoke
these keys by setting the expiration to 0 so they are expired.

Change-Id: Ideb8ae04b23aa55d5825b064b5e43e32eadc1fba
2022-03-23 17:02:20 +00:00
littleskunk
6fbc56bd06
satellite/payment: enable segment fee (#4628) 2022-03-18 14:25:14 +01:00
Mya
4a110b266e satellite/console: added oidc endpoints
This change adds endpoints for supporting OpenID Connect (OIDC) and
OAuth requests. This allows application developers to easily
develop apps with Storj using common mechanisms for authentication
and authorization.

Change-Id: I2a76d48bd1241367aa2d1e3309f6f65d6d6ea4dc
2022-03-16 12:01:26 +00:00