Commit Graph

7204 Commits

Author SHA1 Message Date
Giulio Fidente
5d956c9dc5
Expose LOG_LEVEL env variable via Dockerfile for storagenode (#5362)
The default 'info' level for the storagenode will dump dozens of
lines every second. This change adds the ability to configure
the log.level argument at run time using LOG_LEVEL env variable.

Co-authored-by: Clement Sam <clementsam75@gmail.com>
2022-12-02 22:25:13 +00:00
Cameron
4a3c2cb7d0 satellite/nodeevents: implement customer.io nodeevents.Notifier
The CustomerioNotifier sends node event data to customer.io.

Change-Id: I40a942d6666ae5723c6acb84b11638497f4423f8
2022-12-02 16:50:49 +00:00
Cameron
f8917c49aa satellite/satellitedb: generate random salt for new projects
Change-Id: I52970e1f32c6d4e9840d741c9ce4fbe61ac52b29
2022-12-02 01:24:14 +00:00
Andrew Harding
b2b8efff7d Makefile: disable postgres fsync in the test container
This change turns off fsync on the postgres container used for tests. This
reduces migration time significantly when initializing new satellite
databases.

The change also includes a new benchmark for satellite initialization in
testplanet.

$ benchstat old.txt new.txt name                       old time/op  new time/op  delta
Run_Satellite/Postgres-16   1.36s ± 0%   0.08s ± 0%   ~     (p=1.000 n=1+1)

Change-Id: Ic954767133864770cf652b0dfdcd6b109a167b5f
2022-12-01 22:03:31 +00:00
Andrew Harding
4fdea51d5c storagenode/storagenodedb: faster test db init
Running all of the migrations necessary to initialize a storage node
database takes a significant amount of time during runs.

The package current supports initializing a database from manually coalesced
migration data (i.e. snapshot) which improves the situation somewhat.

This change takes things a bit further by changing the snapshot code to
instead hydrate the database directory from a pre-generated snapshot zip
file.

name                                old time/op  new time/op  delta
Run_StorageNodeCount_4/Postgres-16   2.50s ± 0%   0.16s ± 0%   ~     (p=1.000 n=1+1)

Change-Id: I213bbba5f9199497fbe8ce889b627e853f8b29a0
2022-12-01 20:45:36 +00:00
Cameron
08c9d745f1 satellite/nodeevents: take last_attempted into account when selecting
Previously, the node events chore would select based on the earliest
created_at. However, if for some reason this batch fails, it would still
be the next item to select. If there is a consistent error, the chore
would be stuck retrying the same batch over and over. Now instead
GetNextBatch orders by `last_attempted NULLS FIRST ASC, created_at ASC`.
If a batch fails during Notify, last_attempted is updated so we can move
on to a new batch if one exists.

Change-Id: Ia8458e05ac358d85b2f2c6d690f3d607d631be61
2022-12-01 20:08:38 +00:00
Cameron
76c22fa572 satellite/nodeevents: add method UpdateLastAttempted
Add method to update last_attempted column for a group of rows.

Change-Id: Ib6eadc9efc21368fccb12d1e824c90612c26a8f7
2022-12-01 19:38:54 +00:00
Cameron
b27c6dd032 satellite/nodeevents: add method GetByID
Add method GetByID to get a node event by its ID. For easier testing.

Change-Id: Ic9e0d201ffd682a2ff3bd7b4d02b58b06d0f8282
2022-12-01 19:05:53 +00:00
Cameron
602f0cc132 satellite/satellitedb: add node events column last_attempted
Also add a method to read a node event by id.

Change-Id: I08ca010e18a08253015e0e1280f791d0cc04d239
2022-12-01 18:35:06 +00:00
paul cannon
601874f79a satellite/audit: retire audit.Queues
audit.Queues was the previous method of passing stacks of segments for
audit to the verifier. As of commit 68f9ce4a, this is now happening
by way of the auditor queue (database-backed, so that communication can
happen between multiple peers). audit.Queues is no longer needed.

Refs: https://github.com/storj/storj/issues/5228
Change-Id: I46f2d48d655fb66366c92146cdb6b85aef200552
2022-12-01 13:12:43 +00:00
paul cannon
ed0fa59f23 satellite/overlay: add SetNodeContained() method
SetNodeContained() will change the contained flag in the nodes table,
which will affect whether nodes are selected for new uploads. This flag
_should_ correlate with whether or not a given node has any entries in
the reverification queue. However, the reverification queue is intended
to be 'safely partitionable' from the nodes table, so we can't enforce
that characteristic transactionally. But this is ok; there are no dire
consequences if they are out of sync.

We will be adding a chore that updates the contained flag based on the
contents of the reverification queue periodically, if something fails
to set it directly when appropriate.

Refs: https://github.com/storj/storj/issues/5231
Change-Id: I26460d8718dee63fd55d00a44568b2065fc8fe30
2022-12-01 12:43:40 +00:00
paul cannon
fba39b72b8 satellite/audit: add GetByNodeID to ReverifyQueue
GetByNodeID will allow querying the reverification queue to see if there
are any pending jobs for a given node ID. And thus, to see if that node
ID should be contained or not.

Some parameters on the other methods of the ReverifyQueue interface have
been changed to accept pointers; this was done ahead of the rest of the
changes for the reverification queue to better match the signatures of
the methods that these will replace once ReverifyQueue is actually being
used (meaning fewer changes to tests).

Refs: https://github.com/storj/storj/issues/5251
Change-Id: Ic38ce6d2c650702b69f1c7244a224f00a34893a1
2022-12-01 12:14:49 +00:00
Jeremy Wharton
54a64e1e50 satellite/console: remove error type for incorrect password
This change removes the error type that is returned when a token
request contains an incorrect password. Instead, the generic error
type for invalid login credentials is used.

Change-Id: Ia7dbc38f4a08aeaeeac7ff5b5a801233e349b8b3
2022-11-30 17:04:35 +00:00
Yaroslav Vorobiov
bb1e86c790 satellite: remove unused coinpayments code and chores
issue: https://github.com/storj/storj/issues/4824

Change-Id: I2e3e63151d1def96270279719f6eceda0acba66c
2022-11-30 16:24:48 +00:00
Egon Elbre
4df1f5b50f private/testplanet: add benchmark for testplanet creation
Change-Id: I0f44ea417b6b570b2b3ea6dd82945f03119524ce
2022-11-30 17:08:33 +02:00
Márton Elek
b4d8cbfbbf cmd/uplink: add options to save pprof/trace information
Change-Id: I5bcc602366de4ebd9b761e641a3806ddaeb9ecba
2022-11-30 11:53:29 +00:00
Vitalii
b5aadff0f7 web/satellite: create project passphrase modal
Added create project passphrase modal.
User can select between generated and entered passphrase.

Issue:
https://github.com/storj/storj/issues/5324

Change-Id: I69887562230c7c8002e9bc763ac24c551a0caa1d
2022-11-30 10:39:18 +00:00
prerna-parashar
3fe6aee786
satellite/analytics: Added analytics to track project members addition/deletion (#5340)
satellite/analytics: Added analytics to track project members addition/deletion
2022-11-29 14:56:03 -08:00
Wilfred Asomani
2442ba415f satellite/{web,console}: token links expiry changes
This change reduces the token links expiry time from 24h to 30m and improves the UI to promt users of the expiration.

see: https://github.com/storj/storj-private/issues/17

Change-Id: Iac00f5740fa84069937fdf9bd30a739b6db2a9e0
2022-11-29 21:44:42 +00:00
paul cannon
b612ded9af satellite/audit: help performance of pushing to audit queue
The audit chore will be pushing a large number of segments to be
audited, and the db might choke on that large insert when under load.

This change divides the insert up into batches, which can be sized
however is optimal for the backing database. It also arranges for
segments to be inserted in the order of the primary key, which helps
performance on some systems.

Refs: https://github.com/storj/storj/issues/5228

Change-Id: I941f580f690d681b80c86faf4abca2995e37135d
2022-11-29 15:37:49 +00:00
Vitalii
ffd6ba8e9c web/satellite: add server-side encryption banner to new project dashboard
Added server-side encryption banner to new project dashboard (same as on buckets view).
Reduced notAfter AG caveat to 1 hour for deleting bucket.
Added bucket AG caveat for deleting bucket.

Change-Id: I2aba9e5db315123b8d37c24d80789320093db00e
2022-11-29 14:50:27 +00:00
Michal Niewrzal
47125300dc mod: bump dependencies
* storj/common
* storj/private

Latests common version requires small refactoring for names and types
used by metainfo code.

Change-Id: I224fe93b4751c996ba6e846be0e5677252cf830f
2022-11-29 13:47:34 +00:00
Márton Elek
e617db832e cmd/uplink: ability to set experimental flag from environment variable
Change-Id: I440764a54ac83e5a85e14f64843260d9c4f993fd
2022-11-29 12:11:18 +00:00
Michal Niewrzal
75b77d53ff satellite/gc/sender: avoid sending BF to disqualified and exited nodes
We don't want to waste our time on disqualified and exited nodes.

Change-Id: I11709350ad291c24f3b46670dd6a418c0ddbb44f
2022-11-29 09:56:32 +00:00
Ivan Fraixedes
ef4b564b82
cmd/uplink: Update error message referring to 'import'
Uplink doesn't have a `save` command, however, it's referred on an error
message that's returned when the `access register` command is executed
without having any default access configured.

The correct command to mention is `import`.

Change-Id: Ia2092d02965737f421683fc98c52a51c9529b86e
2022-11-25 18:54:51 +01:00
Moby von Briesen
3501656e98 satellite/repair: Add flag to allow disabling reputation updates
Reputation updates during repair currently consumes a lot of database
resources. Sometimes increasing the rate of repair is more important
than auditing a node based on whether they have or don't have the
correct piece during repair. This is the job of the audit service.

This commit is to implement an intermediate solution from this issue: https://github.com/storj/storj/issues/5089
This commit does not address the more in-depth fix discussed here: https://github.com/storj/storj/issues/4939

Change-Id: I4163b18d78a96fadf5265789fd73c8aa8def0e9f
2022-11-24 08:31:11 -05:00
dlamarmorgan
94dcfd77ee Revert "web/satellite: allow execute permissions on wasm_exec"
This reverts commit 9022506292.

Reason for revert: Execute permissions are not required for this file. The error "Uncaught DOMException: Failed to execute 'importScripts' on 'WorkerGlobalScope':" was instead caused by a browser cached version of import returning an incorrect response.

Change-Id: Ib5f6bf30dc5a9919bb7d71fb2b04075486c74f89
2022-11-23 20:26:07 +00:00
Jeremy Wharton
7a2be3e6f6 private/web,satellite/console/.../consoleapi: serve rate limiting errors as JSON
This change causes rate limiting errors to be returned to the client
as JSON objects rather than plain text to prevent the satellite UI from
encountering issues when trying to parse them.

Resolves storj/customer-issues#88

Change-Id: I11abd19068927a22f1c28d18fc99e7dad8461834
2022-11-23 17:56:07 +00:00
Michal Niewrzal
f0ce8996c3 satellite/metainfo: enable metainfo.multiple-versions flag by default
We tested new upload flow (with multiple versions) to fix inconsistency
while uploading object on QA/EUN1/SLC. Now we would like to enable it
for all satellites by default. Tests required small adjustments.

Fixes https://github.com/storj/storj/issues/5283

Change-Id: I0d53c041abebc0d182ba5a88bb1dac906c29caf0
2022-11-23 17:05:22 +00:00
Vitalii
93bd1e2664 web/satellite: reworked bucket tables
Reworked bucket tables to be the same on buckets view and project dashboard.
Improved table adaptation.
User can open/delete bucket from project dashboard.
User can view bucket details from project dashboard.
User can search through buckets from buckets view.

Issues:
https://github.com/storj/storj/issues/5176
https://github.com/storj/storj/issues/5174

Change-Id: Ie4599ab62dc7eeba9a3349ab188cde8a198cc0f3
2022-11-22 17:49:54 +00:00
paul cannon
8b494f3740 satellite/audit: use db for auditor queue
As part of the effort of splitting out the auditor workers to their own
process, we are transitioning the communication between the auditor
chore and the verification workers to a queue implemented in the
database, rather than the sequence of in-memory queues we used to use.

This logical database is safely partitionable from the rest of
satelliteDB.

Refs: https://github.com/storj/storj/issues/5251

Change-Id: I6cd31ac5265423271fbafe6127a86172c5cb53dc
2022-11-22 14:04:00 +00:00
Erik van Velzen
b574ee5e6d satellite/metabase/rangedloop: service skeleton
Create skeleton for multi-threaded segment loop, peer, cmd command for rangedloop.

Change-Id: I52c78a313f15070d43207c52ea94e53169821654
2022-11-22 15:21:41 +02:00
Michal Niewrzal
d5eea2db61 satellite/accounting: use custom query for bucket tally by default
We added alternative way to calculate bucket tallies for accounting and
now it's tested and we will enable it by default.

CollectBucketTallies was extended to support overriding current time
to be able to test handling expired objects.

Change-Id: I738b99a33fd2e086245f92d874c1cbb806e834c0
2022-11-22 10:23:40 +00:00
Egon Elbre
954d703533 ci: use check-cross-compile tool
Change-Id: I2b858222c33eb0c9d1081ba8853a852ee14b32bd
2022-11-22 09:38:46 +00:00
Michal Niewrzal
8e9b7736cc cmd/satellite: repair-segment; don't stop processing if segment is not found
If we are processing list of segments (csv) we should not stop if one of
segments is not found in DB.

Change-Id: I720f85dc7601c2ca77032e20c1577de55092bd9b
2022-11-22 08:31:16 +00:00
dlamarmorgan
9022506292 web/satellite: allow execute permissions on wasm_exec
Execute permissions are needed on the wasm_exec.js file but nothing seems to set this by default. Once set, it will keep permissions even after rebuilds, but this change will set the initial permissions if required for any new environments.

Change-Id: Ic6848e561210231e67dd2e17a286abdf885926ed
2022-11-21 11:20:53 -08:00
Clement Sam
3378215adf satellite/orders: decrease order expiration time to 24hours
Closes https://github.com/storj/storj/issues/5202

Change-Id: I55d1a84c46dd610eeb00dd79df8f4f7e699499a0
2022-11-21 14:52:32 +00:00
Wilfred Asomani
6c7f412124 web/satellite: fix pop up position on safari
This change fixes wrong positioning of pop ups (the bucket guide, and bucket item drop down) on Safari.

Change-Id: Ia0abcc1a1450cb21e38a435218fb6c788ce546e2
2022-11-21 13:49:05 +00:00
Ivan Fraixedes
567557abc3 satellite/orders: Remove period logs messages
Remove the final period of two log messages to be consistent with the
other logs messages.

Change-Id: I9253a4d5fb293c95d3baf8e093dc5744387c1516
2022-11-21 13:19:13 +00:00
Michal Niewrzal
2ac5d16faf cmd/satellite: fix repair-segment command args validation
After adding option to input only CSV file number of allowed input parameters was not adjusted.

Change-Id: I55096be02d8e692de2f04571309be6b56d18bf67
2022-11-21 12:23:31 +00:00
NickolaiYurchenko
c27563b519 web/satellite: analytics api calls moved to pinia store
Change-Id: Icb17f51f0ebe625fe639cc10f7ff077f68520313
2022-11-21 10:47:26 +00:00
Moby von Briesen
1b67983130 web/satellite: Fix filebrowser bugs in Firefox
When attempting to upload a file on Firefox mobile, an error
"item.webkitRelativePath is undefined" prevents the client from
proceeding with the upload after the file is selected. This commit
contains a small check to protect against this issue.

Additionally, in Firefox (desktop, maybe mobile), links in the file
browser to enter subfolders, exit subfolders, or navigate out of folders
via breadcrumbs has default behavior of navigating the user to an empty
page that only displays "null". This changes adds `@click.prevent` in
some key locations to prevent this undesired behavior.

Change-Id: I83ee6fcc5a7d0ce9996dacc3f966e38a4936a9fe
2022-11-18 17:57:23 -05:00
Jeremy Wharton
f84111ee69 web/satellite: prevent token card text from overlapping elements
This change resolves an issue where text inside the Billing page's
Storj Token card would appear over the project dropdown menu.
The issue was caused by using z-indexing to prevent an absolutely
positioned cosmetic element from displaying over its non-positioned
siblings. An alternate solution is presented in this change:
specifying a position for the non-cosmetic elements allows them to
display properly over preceding positioned elements without the usage
of z-indexing.

Resolves #5219

Change-Id: Icebbda33dcef9418020b025fe8ab3f461cd56533
2022-11-18 21:05:21 +00:00
Michal Niewrzal
8d5a2a90f2 cmd/satellite: repair-segment; add option to process csv file directly
Current option is to put stream id and position as an input but
it's not very efficient when we have long list of segments to repair.
This change adds option to read whole csv file and process each entry
one by one.

If command will have single argument then it will treat it as csv file
location and if will have two arguments then it will parse it just as
stream id and position.

Change-Id: I1e91cf57a794d81d74af0091c24a2e7d00d1fab9
2022-11-18 17:40:17 +00:00
Cameron
87660bd9b3 satellite/overlay/offlinenodes: insert offline nodes into node events
Add a new chore to periodically insert nodes who are offline and
have not gotten an offline email in a certain amount of time into node
events

Change-Id: I658b385bb777b0240c98092946a93d65bee94abc
2022-11-18 12:10:06 -05:00
Michal Niewrzal
ec777855e1 cmd/satellite: add segment-repair command
Implements logic for satellite command to repair a single segment.
Segment will be repaired even if it's healthy. This is not checked
during this process. As a part of repair command will download whole
segment into memory and will try to reupload segment to number of new
nodes that equal to existing number of pieces. After successful
upload new pieces will completely replace existing pieces. 

Command:
    satellite repair-segment <streamid> <position>

https://github.com/storj/storj/issues/5254

Change-Id: I8e329718ecf8e457dbee3434e1c68951699007d9
2022-11-18 16:18:08 +01:00
Cameron
705dfa0875 satellite/nodeevents: add method Name to Type
Add method Name to Type for a more human-readable format

Change-Id: I10050b9ef3691922d9dfccdbb6f046769a649321
2022-11-18 12:31:02 +00:00
Cameron
57be07f60a satellite/nodeevents: add Chore
Create NodeEvents Chore on satellite core to read nodeevents DB and
notify node operators on node events. The chore sends notifications
grouped by email and event type: it selects the oldest entry in
nodeevents.DB and also any other event with the same email and event
type no matter how old it is. The oldest entry of a group must exist for
a minimum amount of time before that group can be selected, however.
This minimum amount of time is a configurable value:
--node-events.selection-wait-period. This wait period allows us to
combine events of the same time and same email address into a singular
email.

Change-Id: I8b444aa324d2dae265cc27d9e9e85faef79195d8
2022-11-18 12:00:14 +00:00
Clement Sam
3e0a4230a5 storagenode/payout: fix disk space value at payout
Payout is still calculating using the tb*h. 0So it’s getting the total disk space used this month and dividing by (24*30) instead of just 30.

More context here: https://forum.storj.io/t/current-month-earnings-in-node-v1-67-1/20319/5

Follow up PR for https://github.com/storj/storj/issues/5146

Change-Id: Ie2d48497f2a9bdbc995c99ee27e70b46580ff638
2022-11-18 01:04:20 +00:00
Erik van Velzen
9fb18a43d8 satellite/metabase/rangedloop: observer interface
New interface for parallel segment loop.

Closes https://github.com/storj/storj/issues/5239

Change-Id: I2bcce6f836f6625da8ceb4fc0fc030c0ea4380e7
2022-11-17 20:12:23 +00:00