Commit Graph

7230 Commits

Author SHA1 Message Date
Erik van Velzen
3cf7ebfad0
satellite/metabase/rangedloop: database abstraction (#5337)
Add an abstraction rangedloop.SegmentProvider to fetch chunks of
segments from the metainfo database in parallel.

Part of https://github.com/storj/storj/issues/5223

Change-Id: Ife26467ea0c3be550bde0b05464ef1db62dd4d2a
2022-12-09 03:01:07 +01:00
Lizzy Thomson
9fedc21fea satellite/console account session management for enabling/disabling MFA
Adds DeleteAllSessionsByUserIDExcept which removes all sessions except the specified session from the database and applies this function to enableMFA and disableMFA

addresses https://github.com/storj/storj-private/issues/15

Change-Id: I5d8c620dadbbda4a1b430ccf8a6121e167dd0761
2022-12-08 18:32:35 +00:00
Jeff Wendling
f2fdd6ca33 cmd/uplink: fix some issues with share
Because --readonly is default true, passing something like
--disallow-deletes=false would not actually update that
value because the readonly flag would override. this makes it
so that the --disallow-* flags override the --readonly and
--writeonly flags.

Also fixes some minor formatting issues with share like an
extra space after the "Public Access:" entry.

Simplifies the handling of the explicit "none" by making the
flags for the dates optional and using nil to signify that
the value was left unset.

Bump the go.mod to go1.18 to enable the use of generics and
add a small generic function. This can easily be backed out
if it causes problems.

Change-Id: I1c5f1321ad17b8ace778ce55561cbbfc24321a68
2022-12-08 17:46:02 +00:00
Wilfred Asomani
66b4509276 web/satellite: use TableItem in FileEntry.vue
This change continues the update of the file browser to use the common table component.
It replaces the use of <tr>s with <table-item>.
It also removes bootstrap from the file browser and fixes resulting UI "bugs"

Issue: https://github.com/storj/storj/issues/5090

Change-Id: Idfe415e9338243ad4fcedc8a74621c9e13e1da29
2022-12-08 14:38:09 +00:00
Erik van Velzen
ff6d640fca
satellite/metabase/rangedloop: minimal loop (#5334)
Minimal implementation of the ranged (=threaded) segment loop
service, to improve performance over the existing loop.

Has tests with a an inmemory segment database
and example observer.

Does not have yet: database link, observer duration tracking,
suspicious processed ratio guard, rate limiting, minimum execution
interval per observer, etc.

Part of https://github.com/storj/storj/issues/5223

Change-Id: I08ffb392c3539e380f4e7b4f1afd56c4c394668d
2022-12-08 15:27:21 +01:00
Wilfred Asomani
bf106131b0 satellite/{web/payments}: show token balance on billing overview
This change shows STORJ token balance on the billing overview page instead of the Stripe balance it shows currently.
It changes the text on the "Available balance" card to reflect the new balance being displayed. Finally, it adds shortcuts to navigate straight to token history or add tokens modal when call to action on "Balance card"

Issue: https://github.com/storj/storj/issues/5204

Change-Id: Ic88e43c602e4949b6c6be4c7644c04f3c7d38585
2022-12-08 11:47:37 +00:00
Cameron
907c911f57 satellite/console/emailreminders: set Enable config default to true
Enable email verification reminders by default

Change-Id: I2f8143e2805ccfd2ffccfbc019591859326499c5
2022-12-08 11:02:43 +00:00
Fadila Khadar
7fd23d6864 satellite/metabase: add logic for verifying segments in given buckets
To be able to verify segments in a list of buckets, this change:
- adds method ListBucketsStreamIDs to list all stream ids belonging to a list of buckets provided using a ListVerifyBucketList on which Add(projectID, bucketName) is defined.
- allows to specify a list of streamIDs to check in ListVerifySegments

Fixes https://github.com/storj/storj-private/issues/101

Change-Id: I72a48a0873a3056ac54ad56c0e9242364b2ae918
2022-12-08 09:45:15 +00:00
Jeremy Wharton
d6bcde4672 docs/blueprints: update automatic account freezing/unfreezing blueprint
This change updates the automatic account freezing/unfreezing blueprint
to account for new feedback and changes in the implementation process.

Change-Id: If1a478d961b67aa4a946793168a7f525e06bb9e7
2022-12-07 16:32:47 -06:00
Wilfred Asomani
91b551134d web/satellite: fix mobile header not visible on ios safari and chrome
This change fixes an issue where the navigation header will not show on iOS (safari and chrome).
This is apparently a known issue where the Webkit viewport height is taller than the visible viewport.
More information here; https://bugs.webkit.org/show_bug.cgi?id=141832 , https://stackoverflow.com/a/56671096

This change applies the fix on all iOS browsers (using Webkit) except for Firefox, in which this fix introduces a strange behaviour
when a virtual keyboard is opened and dismissed, while not solving the issue because it doesn't exist here.

Issue: https://github.com/storj/storj/issues/5153

Change-Id: I19fc8018012bb70a5eda3eb21d44bfeedd08411c
2022-12-07 20:21:00 +00:00
Michal Niewrzal
4544eee72b Revert "satellite/metainfo: enable metainfo.multiple-versions flag by default"
This reverts commit f0ce8996c3.

We need to revert it until https://github.com/storj/storj/issues/5373 is
fixed.

Change-Id: Ibb22af100014724d1910d4871d8f4e159fdea391
2022-12-07 19:43:20 +00:00
paul cannon
35f60fd5eb satellite/audit: change signature of ReverifyPiece
First, adding a logger argument allows the caller to have a logger
already set up with whatever extra fields they want here.

Secondly, we need to return the Outcome instead of a simple boolean so
that it can be passed on to the Reporter later (need to make the right
decision on increasing reputation vs decreasing it).

Thirdly, we collect the cached reputation information from the overlay
when creating the Orders, and return it from ReverifyPiece. This will
allow the Reporter to determine what reputation-status fields need to be
updated, similarly to how we include a map of ReputationStatus objects
in an audit.Report.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: I5700b9ce543d18b857b81e684323b2d21c498cd8
2022-12-07 18:32:42 +00:00
paul cannon
378b8915c4 satellite/{satellitedb,audit}: add NewContainment
NewContainment will replace Containment later in this commit chain, but
for now it is not yet being used.

NewContainment will allow a node to be contained for multiple pending
reverify jobs at a time. It is implemented by way of the reverify queue.

Refs: https://github.com/storj/storj/issues/5231
Change-Id: I126eda0b3dfc4710a88fe4a5f41780618ec19101
2022-12-07 18:03:37 +00:00
Márton Elek
27191d5b80 mod: bump storj.io/uplink
Change-Id: I8548f6bfafb86646bbc71b040190fad28579d92e
2022-12-07 14:07:18 +00:00
CatInAHatIsBack
7d200d3962
web/storagenode: fix bandwidth graph (#5020)
fixes #4951

Co-authored-by: Clement Sam <clementsam75@gmail.com>
2022-12-07 14:31:13 +01:00
Andrew Harding
c6e48fb71d satellite/metabase/rangedloop: clarify observer docs
Change-Id: I171d39fd069186c2c275aab3a5e672427b34e38f
2022-12-07 11:27:35 +00:00
Michal Niewrzal
8e030f4e30 satellite/accounting/tally: fix looping over all buckets
We have a bug where if number of buckets in the system will be
multiplication of batch size (2500) then loop that is going over
all buckets can run indefinitely.

Fixes https://github.com/storj/storj/issues/5374

Change-Id: Idd4d97c638db83e46528acb9abf223c98ad46223
2022-12-07 10:32:23 +00:00
Cameron
a3ff3eb193 satellite/nodeevents: validate emails before notifying
Simple email validation before attempting to send notifications. If the
email is not valid, skip sending notifications and go to update
email_sent so we don't try it again. Also, move ValidateEmail function
into new package so it can be used in nodeevents without import cycle.

Change-Id: I63ce0fc84f7b1d964f7cc6da61206f54baaf1a21
2022-12-06 09:59:45 -05:00
Clement Sam
746c3b2d83 private/multinodepb: return at_rest_total_bytes and summary_bytes for StorageUsage endpoint
at_rest_total_bytes and summary_bytes are storage usages return as bytes
instead of bytes*hour. This is used for the disk space graph.

Updates https://github.com/storj/storj/issues/5355

Change-Id: I81f77fe9b9069cf3b29ab681586e506363e5b066
2022-12-06 12:27:34 +00:00
Wilfred Asomani
c1ed5c06e8 satellite/{admin,ui}: implement changes for oauth2 proxy
We want to put the OAuth2 proxy, https://github.com/oauth2-proxy/oauth2-proxy, in front of the satellite admin ui.
This change implements the changes required/necessary for this to work.

Issue: https://github.com/storj/storj/issues/5072

Change-Id: I6da0df090cc6f0c18f1bf41e48ae082493f53f20
2022-12-06 09:44:04 +00:00
Vitalii
9a0c2dd878 web/satellite: added manage passphrase modal
Added manage passphrase modal where user can create new, switch or clear their project level passphrase.
Removed debug buttons from navigation.

Issue:
https://github.com/storj/storj/issues/5325

Change-Id: I53f263a9697f197cee3f68634f5799d45b3eb193
2022-12-06 10:57:39 +02:00
nadimhq
4b72314e90
docs/testplan: Testplan for Storage-node Email Notification (#5338)
This testplan is going to cover the changes to storage-node email notifications. It will go over the storage-node email notification design doc.

Co-authored-by: Antonio Franco (He/Him) <antonio@storj.io>
2022-12-05 13:48:47 -05:00
paul cannon
d8df75186b satellite/satellitedb: change reverifyQueue.Insert()
It helps for the (*reverifyQueue).Insert() method to be idempotent (it
does not make sense for the same node to be under containment for the
same piece multiple times). This change allows for that, by adding an
`ON CONFLICT DO NOTHING` clause to the database query.

Refs: https://github.com/storj/storj/issues/5231
Change-Id: Id2839ee185d5396c0bc2f84ffad610df9786f6c7
2022-12-05 17:15:11 +00:00
Kaloyan Raev
1a8954daad CODEOWNERS: added package ownership for Team Integrations
Change-Id: Idec883cd6efe5bbc0466fef5615683b0520ea9a5
2022-12-05 16:25:53 +00:00
paul cannon
2617925f8d satellite/audit: Split Reverifier from Verifier
Adding a new worker comparable to Verifier, called Reverifier; as the
name suggests, it will be used for reverifications, whereas Verifier
will be used for verifications.

This allows distinct logging from the two classes, plus we can add some
configuration that is specific to the Reverifier.

There is a slight modification to GetNextJob that goes along with this.

This should have no impact on operational concerns.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: Ie60d2d833bc5db8660bb463dd93c764bb40fc49c
2022-12-05 09:55:49 -06:00
Egon Elbre
8777523255 private/testplanet: disable WAL for storagenodes
Change-Id: I1be4f7901c830e829118afeb90f04b87df555459
2022-12-05 11:41:06 +00:00
Giulio Fidente
5d956c9dc5
Expose LOG_LEVEL env variable via Dockerfile for storagenode (#5362)
The default 'info' level for the storagenode will dump dozens of
lines every second. This change adds the ability to configure
the log.level argument at run time using LOG_LEVEL env variable.

Co-authored-by: Clement Sam <clementsam75@gmail.com>
2022-12-02 22:25:13 +00:00
Cameron
4a3c2cb7d0 satellite/nodeevents: implement customer.io nodeevents.Notifier
The CustomerioNotifier sends node event data to customer.io.

Change-Id: I40a942d6666ae5723c6acb84b11638497f4423f8
2022-12-02 16:50:49 +00:00
Cameron
f8917c49aa satellite/satellitedb: generate random salt for new projects
Change-Id: I52970e1f32c6d4e9840d741c9ce4fbe61ac52b29
2022-12-02 01:24:14 +00:00
Andrew Harding
b2b8efff7d Makefile: disable postgres fsync in the test container
This change turns off fsync on the postgres container used for tests. This
reduces migration time significantly when initializing new satellite
databases.

The change also includes a new benchmark for satellite initialization in
testplanet.

$ benchstat old.txt new.txt name                       old time/op  new time/op  delta
Run_Satellite/Postgres-16   1.36s ± 0%   0.08s ± 0%   ~     (p=1.000 n=1+1)

Change-Id: Ic954767133864770cf652b0dfdcd6b109a167b5f
2022-12-01 22:03:31 +00:00
Andrew Harding
4fdea51d5c storagenode/storagenodedb: faster test db init
Running all of the migrations necessary to initialize a storage node
database takes a significant amount of time during runs.

The package current supports initializing a database from manually coalesced
migration data (i.e. snapshot) which improves the situation somewhat.

This change takes things a bit further by changing the snapshot code to
instead hydrate the database directory from a pre-generated snapshot zip
file.

name                                old time/op  new time/op  delta
Run_StorageNodeCount_4/Postgres-16   2.50s ± 0%   0.16s ± 0%   ~     (p=1.000 n=1+1)

Change-Id: I213bbba5f9199497fbe8ce889b627e853f8b29a0
2022-12-01 20:45:36 +00:00
Cameron
08c9d745f1 satellite/nodeevents: take last_attempted into account when selecting
Previously, the node events chore would select based on the earliest
created_at. However, if for some reason this batch fails, it would still
be the next item to select. If there is a consistent error, the chore
would be stuck retrying the same batch over and over. Now instead
GetNextBatch orders by `last_attempted NULLS FIRST ASC, created_at ASC`.
If a batch fails during Notify, last_attempted is updated so we can move
on to a new batch if one exists.

Change-Id: Ia8458e05ac358d85b2f2c6d690f3d607d631be61
2022-12-01 20:08:38 +00:00
Cameron
76c22fa572 satellite/nodeevents: add method UpdateLastAttempted
Add method to update last_attempted column for a group of rows.

Change-Id: Ib6eadc9efc21368fccb12d1e824c90612c26a8f7
2022-12-01 19:38:54 +00:00
Cameron
b27c6dd032 satellite/nodeevents: add method GetByID
Add method GetByID to get a node event by its ID. For easier testing.

Change-Id: Ic9e0d201ffd682a2ff3bd7b4d02b58b06d0f8282
2022-12-01 19:05:53 +00:00
Cameron
602f0cc132 satellite/satellitedb: add node events column last_attempted
Also add a method to read a node event by id.

Change-Id: I08ca010e18a08253015e0e1280f791d0cc04d239
2022-12-01 18:35:06 +00:00
paul cannon
601874f79a satellite/audit: retire audit.Queues
audit.Queues was the previous method of passing stacks of segments for
audit to the verifier. As of commit 68f9ce4a, this is now happening
by way of the auditor queue (database-backed, so that communication can
happen between multiple peers). audit.Queues is no longer needed.

Refs: https://github.com/storj/storj/issues/5228
Change-Id: I46f2d48d655fb66366c92146cdb6b85aef200552
2022-12-01 13:12:43 +00:00
paul cannon
ed0fa59f23 satellite/overlay: add SetNodeContained() method
SetNodeContained() will change the contained flag in the nodes table,
which will affect whether nodes are selected for new uploads. This flag
_should_ correlate with whether or not a given node has any entries in
the reverification queue. However, the reverification queue is intended
to be 'safely partitionable' from the nodes table, so we can't enforce
that characteristic transactionally. But this is ok; there are no dire
consequences if they are out of sync.

We will be adding a chore that updates the contained flag based on the
contents of the reverification queue periodically, if something fails
to set it directly when appropriate.

Refs: https://github.com/storj/storj/issues/5231
Change-Id: I26460d8718dee63fd55d00a44568b2065fc8fe30
2022-12-01 12:43:40 +00:00
paul cannon
fba39b72b8 satellite/audit: add GetByNodeID to ReverifyQueue
GetByNodeID will allow querying the reverification queue to see if there
are any pending jobs for a given node ID. And thus, to see if that node
ID should be contained or not.

Some parameters on the other methods of the ReverifyQueue interface have
been changed to accept pointers; this was done ahead of the rest of the
changes for the reverification queue to better match the signatures of
the methods that these will replace once ReverifyQueue is actually being
used (meaning fewer changes to tests).

Refs: https://github.com/storj/storj/issues/5251
Change-Id: Ic38ce6d2c650702b69f1c7244a224f00a34893a1
2022-12-01 12:14:49 +00:00
Jeremy Wharton
54a64e1e50 satellite/console: remove error type for incorrect password
This change removes the error type that is returned when a token
request contains an incorrect password. Instead, the generic error
type for invalid login credentials is used.

Change-Id: Ia7dbc38f4a08aeaeeac7ff5b5a801233e349b8b3
2022-11-30 17:04:35 +00:00
Yaroslav Vorobiov
bb1e86c790 satellite: remove unused coinpayments code and chores
issue: https://github.com/storj/storj/issues/4824

Change-Id: I2e3e63151d1def96270279719f6eceda0acba66c
2022-11-30 16:24:48 +00:00
Egon Elbre
4df1f5b50f private/testplanet: add benchmark for testplanet creation
Change-Id: I0f44ea417b6b570b2b3ea6dd82945f03119524ce
2022-11-30 17:08:33 +02:00
Márton Elek
b4d8cbfbbf cmd/uplink: add options to save pprof/trace information
Change-Id: I5bcc602366de4ebd9b761e641a3806ddaeb9ecba
2022-11-30 11:53:29 +00:00
Vitalii
b5aadff0f7 web/satellite: create project passphrase modal
Added create project passphrase modal.
User can select between generated and entered passphrase.

Issue:
https://github.com/storj/storj/issues/5324

Change-Id: I69887562230c7c8002e9bc763ac24c551a0caa1d
2022-11-30 10:39:18 +00:00
prerna-parashar
3fe6aee786
satellite/analytics: Added analytics to track project members addition/deletion (#5340)
satellite/analytics: Added analytics to track project members addition/deletion
2022-11-29 14:56:03 -08:00
Wilfred Asomani
2442ba415f satellite/{web,console}: token links expiry changes
This change reduces the token links expiry time from 24h to 30m and improves the UI to promt users of the expiration.

see: https://github.com/storj/storj-private/issues/17

Change-Id: Iac00f5740fa84069937fdf9bd30a739b6db2a9e0
2022-11-29 21:44:42 +00:00
paul cannon
b612ded9af satellite/audit: help performance of pushing to audit queue
The audit chore will be pushing a large number of segments to be
audited, and the db might choke on that large insert when under load.

This change divides the insert up into batches, which can be sized
however is optimal for the backing database. It also arranges for
segments to be inserted in the order of the primary key, which helps
performance on some systems.

Refs: https://github.com/storj/storj/issues/5228

Change-Id: I941f580f690d681b80c86faf4abca2995e37135d
2022-11-29 15:37:49 +00:00
Vitalii
ffd6ba8e9c web/satellite: add server-side encryption banner to new project dashboard
Added server-side encryption banner to new project dashboard (same as on buckets view).
Reduced notAfter AG caveat to 1 hour for deleting bucket.
Added bucket AG caveat for deleting bucket.

Change-Id: I2aba9e5db315123b8d37c24d80789320093db00e
2022-11-29 14:50:27 +00:00
Michal Niewrzal
47125300dc mod: bump dependencies
* storj/common
* storj/private

Latests common version requires small refactoring for names and types
used by metainfo code.

Change-Id: I224fe93b4751c996ba6e846be0e5677252cf830f
2022-11-29 13:47:34 +00:00
Márton Elek
e617db832e cmd/uplink: ability to set experimental flag from environment variable
Change-Id: I440764a54ac83e5a85e14f64843260d9c4f993fd
2022-11-29 12:11:18 +00:00
Michal Niewrzal
75b77d53ff satellite/gc/sender: avoid sending BF to disqualified and exited nodes
We don't want to waste our time on disqualified and exited nodes.

Change-Id: I11709350ad291c24f3b46670dd6a418c0ddbb44f
2022-11-29 09:56:32 +00:00