Commit Graph

7194 Commits

Author SHA1 Message Date
paul cannon
ed0fa59f23 satellite/overlay: add SetNodeContained() method
SetNodeContained() will change the contained flag in the nodes table,
which will affect whether nodes are selected for new uploads. This flag
_should_ correlate with whether or not a given node has any entries in
the reverification queue. However, the reverification queue is intended
to be 'safely partitionable' from the nodes table, so we can't enforce
that characteristic transactionally. But this is ok; there are no dire
consequences if they are out of sync.

We will be adding a chore that updates the contained flag based on the
contents of the reverification queue periodically, if something fails
to set it directly when appropriate.

Refs: https://github.com/storj/storj/issues/5231
Change-Id: I26460d8718dee63fd55d00a44568b2065fc8fe30
2022-12-01 12:43:40 +00:00
paul cannon
fba39b72b8 satellite/audit: add GetByNodeID to ReverifyQueue
GetByNodeID will allow querying the reverification queue to see if there
are any pending jobs for a given node ID. And thus, to see if that node
ID should be contained or not.

Some parameters on the other methods of the ReverifyQueue interface have
been changed to accept pointers; this was done ahead of the rest of the
changes for the reverification queue to better match the signatures of
the methods that these will replace once ReverifyQueue is actually being
used (meaning fewer changes to tests).

Refs: https://github.com/storj/storj/issues/5251
Change-Id: Ic38ce6d2c650702b69f1c7244a224f00a34893a1
2022-12-01 12:14:49 +00:00
Jeremy Wharton
54a64e1e50 satellite/console: remove error type for incorrect password
This change removes the error type that is returned when a token
request contains an incorrect password. Instead, the generic error
type for invalid login credentials is used.

Change-Id: Ia7dbc38f4a08aeaeeac7ff5b5a801233e349b8b3
2022-11-30 17:04:35 +00:00
Yaroslav Vorobiov
bb1e86c790 satellite: remove unused coinpayments code and chores
issue: https://github.com/storj/storj/issues/4824

Change-Id: I2e3e63151d1def96270279719f6eceda0acba66c
2022-11-30 16:24:48 +00:00
Egon Elbre
4df1f5b50f private/testplanet: add benchmark for testplanet creation
Change-Id: I0f44ea417b6b570b2b3ea6dd82945f03119524ce
2022-11-30 17:08:33 +02:00
Márton Elek
b4d8cbfbbf cmd/uplink: add options to save pprof/trace information
Change-Id: I5bcc602366de4ebd9b761e641a3806ddaeb9ecba
2022-11-30 11:53:29 +00:00
Vitalii
b5aadff0f7 web/satellite: create project passphrase modal
Added create project passphrase modal.
User can select between generated and entered passphrase.

Issue:
https://github.com/storj/storj/issues/5324

Change-Id: I69887562230c7c8002e9bc763ac24c551a0caa1d
2022-11-30 10:39:18 +00:00
prerna-parashar
3fe6aee786
satellite/analytics: Added analytics to track project members addition/deletion (#5340)
satellite/analytics: Added analytics to track project members addition/deletion
2022-11-29 14:56:03 -08:00
Wilfred Asomani
2442ba415f satellite/{web,console}: token links expiry changes
This change reduces the token links expiry time from 24h to 30m and improves the UI to promt users of the expiration.

see: https://github.com/storj/storj-private/issues/17

Change-Id: Iac00f5740fa84069937fdf9bd30a739b6db2a9e0
2022-11-29 21:44:42 +00:00
paul cannon
b612ded9af satellite/audit: help performance of pushing to audit queue
The audit chore will be pushing a large number of segments to be
audited, and the db might choke on that large insert when under load.

This change divides the insert up into batches, which can be sized
however is optimal for the backing database. It also arranges for
segments to be inserted in the order of the primary key, which helps
performance on some systems.

Refs: https://github.com/storj/storj/issues/5228

Change-Id: I941f580f690d681b80c86faf4abca2995e37135d
2022-11-29 15:37:49 +00:00
Vitalii
ffd6ba8e9c web/satellite: add server-side encryption banner to new project dashboard
Added server-side encryption banner to new project dashboard (same as on buckets view).
Reduced notAfter AG caveat to 1 hour for deleting bucket.
Added bucket AG caveat for deleting bucket.

Change-Id: I2aba9e5db315123b8d37c24d80789320093db00e
2022-11-29 14:50:27 +00:00
Michal Niewrzal
47125300dc mod: bump dependencies
* storj/common
* storj/private

Latests common version requires small refactoring for names and types
used by metainfo code.

Change-Id: I224fe93b4751c996ba6e846be0e5677252cf830f
2022-11-29 13:47:34 +00:00
Márton Elek
e617db832e cmd/uplink: ability to set experimental flag from environment variable
Change-Id: I440764a54ac83e5a85e14f64843260d9c4f993fd
2022-11-29 12:11:18 +00:00
Michal Niewrzal
75b77d53ff satellite/gc/sender: avoid sending BF to disqualified and exited nodes
We don't want to waste our time on disqualified and exited nodes.

Change-Id: I11709350ad291c24f3b46670dd6a418c0ddbb44f
2022-11-29 09:56:32 +00:00
Ivan Fraixedes
ef4b564b82
cmd/uplink: Update error message referring to 'import'
Uplink doesn't have a `save` command, however, it's referred on an error
message that's returned when the `access register` command is executed
without having any default access configured.

The correct command to mention is `import`.

Change-Id: Ia2092d02965737f421683fc98c52a51c9529b86e
2022-11-25 18:54:51 +01:00
Moby von Briesen
3501656e98 satellite/repair: Add flag to allow disabling reputation updates
Reputation updates during repair currently consumes a lot of database
resources. Sometimes increasing the rate of repair is more important
than auditing a node based on whether they have or don't have the
correct piece during repair. This is the job of the audit service.

This commit is to implement an intermediate solution from this issue: https://github.com/storj/storj/issues/5089
This commit does not address the more in-depth fix discussed here: https://github.com/storj/storj/issues/4939

Change-Id: I4163b18d78a96fadf5265789fd73c8aa8def0e9f
2022-11-24 08:31:11 -05:00
dlamarmorgan
94dcfd77ee Revert "web/satellite: allow execute permissions on wasm_exec"
This reverts commit 9022506292.

Reason for revert: Execute permissions are not required for this file. The error "Uncaught DOMException: Failed to execute 'importScripts' on 'WorkerGlobalScope':" was instead caused by a browser cached version of import returning an incorrect response.

Change-Id: Ib5f6bf30dc5a9919bb7d71fb2b04075486c74f89
2022-11-23 20:26:07 +00:00
Jeremy Wharton
7a2be3e6f6 private/web,satellite/console/.../consoleapi: serve rate limiting errors as JSON
This change causes rate limiting errors to be returned to the client
as JSON objects rather than plain text to prevent the satellite UI from
encountering issues when trying to parse them.

Resolves storj/customer-issues#88

Change-Id: I11abd19068927a22f1c28d18fc99e7dad8461834
2022-11-23 17:56:07 +00:00
Michal Niewrzal
f0ce8996c3 satellite/metainfo: enable metainfo.multiple-versions flag by default
We tested new upload flow (with multiple versions) to fix inconsistency
while uploading object on QA/EUN1/SLC. Now we would like to enable it
for all satellites by default. Tests required small adjustments.

Fixes https://github.com/storj/storj/issues/5283

Change-Id: I0d53c041abebc0d182ba5a88bb1dac906c29caf0
2022-11-23 17:05:22 +00:00
Vitalii
93bd1e2664 web/satellite: reworked bucket tables
Reworked bucket tables to be the same on buckets view and project dashboard.
Improved table adaptation.
User can open/delete bucket from project dashboard.
User can view bucket details from project dashboard.
User can search through buckets from buckets view.

Issues:
https://github.com/storj/storj/issues/5176
https://github.com/storj/storj/issues/5174

Change-Id: Ie4599ab62dc7eeba9a3349ab188cde8a198cc0f3
2022-11-22 17:49:54 +00:00
paul cannon
8b494f3740 satellite/audit: use db for auditor queue
As part of the effort of splitting out the auditor workers to their own
process, we are transitioning the communication between the auditor
chore and the verification workers to a queue implemented in the
database, rather than the sequence of in-memory queues we used to use.

This logical database is safely partitionable from the rest of
satelliteDB.

Refs: https://github.com/storj/storj/issues/5251

Change-Id: I6cd31ac5265423271fbafe6127a86172c5cb53dc
2022-11-22 14:04:00 +00:00
Erik van Velzen
b574ee5e6d satellite/metabase/rangedloop: service skeleton
Create skeleton for multi-threaded segment loop, peer, cmd command for rangedloop.

Change-Id: I52c78a313f15070d43207c52ea94e53169821654
2022-11-22 15:21:41 +02:00
Michal Niewrzal
d5eea2db61 satellite/accounting: use custom query for bucket tally by default
We added alternative way to calculate bucket tallies for accounting and
now it's tested and we will enable it by default.

CollectBucketTallies was extended to support overriding current time
to be able to test handling expired objects.

Change-Id: I738b99a33fd2e086245f92d874c1cbb806e834c0
2022-11-22 10:23:40 +00:00
Egon Elbre
954d703533 ci: use check-cross-compile tool
Change-Id: I2b858222c33eb0c9d1081ba8853a852ee14b32bd
2022-11-22 09:38:46 +00:00
Michal Niewrzal
8e9b7736cc cmd/satellite: repair-segment; don't stop processing if segment is not found
If we are processing list of segments (csv) we should not stop if one of
segments is not found in DB.

Change-Id: I720f85dc7601c2ca77032e20c1577de55092bd9b
2022-11-22 08:31:16 +00:00
dlamarmorgan
9022506292 web/satellite: allow execute permissions on wasm_exec
Execute permissions are needed on the wasm_exec.js file but nothing seems to set this by default. Once set, it will keep permissions even after rebuilds, but this change will set the initial permissions if required for any new environments.

Change-Id: Ic6848e561210231e67dd2e17a286abdf885926ed
2022-11-21 11:20:53 -08:00
Clement Sam
3378215adf satellite/orders: decrease order expiration time to 24hours
Closes https://github.com/storj/storj/issues/5202

Change-Id: I55d1a84c46dd610eeb00dd79df8f4f7e699499a0
2022-11-21 14:52:32 +00:00
Wilfred Asomani
6c7f412124 web/satellite: fix pop up position on safari
This change fixes wrong positioning of pop ups (the bucket guide, and bucket item drop down) on Safari.

Change-Id: Ia0abcc1a1450cb21e38a435218fb6c788ce546e2
2022-11-21 13:49:05 +00:00
Ivan Fraixedes
567557abc3 satellite/orders: Remove period logs messages
Remove the final period of two log messages to be consistent with the
other logs messages.

Change-Id: I9253a4d5fb293c95d3baf8e093dc5744387c1516
2022-11-21 13:19:13 +00:00
Michal Niewrzal
2ac5d16faf cmd/satellite: fix repair-segment command args validation
After adding option to input only CSV file number of allowed input parameters was not adjusted.

Change-Id: I55096be02d8e692de2f04571309be6b56d18bf67
2022-11-21 12:23:31 +00:00
NickolaiYurchenko
c27563b519 web/satellite: analytics api calls moved to pinia store
Change-Id: Icb17f51f0ebe625fe639cc10f7ff077f68520313
2022-11-21 10:47:26 +00:00
Moby von Briesen
1b67983130 web/satellite: Fix filebrowser bugs in Firefox
When attempting to upload a file on Firefox mobile, an error
"item.webkitRelativePath is undefined" prevents the client from
proceeding with the upload after the file is selected. This commit
contains a small check to protect against this issue.

Additionally, in Firefox (desktop, maybe mobile), links in the file
browser to enter subfolders, exit subfolders, or navigate out of folders
via breadcrumbs has default behavior of navigating the user to an empty
page that only displays "null". This changes adds `@click.prevent` in
some key locations to prevent this undesired behavior.

Change-Id: I83ee6fcc5a7d0ce9996dacc3f966e38a4936a9fe
2022-11-18 17:57:23 -05:00
Jeremy Wharton
f84111ee69 web/satellite: prevent token card text from overlapping elements
This change resolves an issue where text inside the Billing page's
Storj Token card would appear over the project dropdown menu.
The issue was caused by using z-indexing to prevent an absolutely
positioned cosmetic element from displaying over its non-positioned
siblings. An alternate solution is presented in this change:
specifying a position for the non-cosmetic elements allows them to
display properly over preceding positioned elements without the usage
of z-indexing.

Resolves #5219

Change-Id: Icebbda33dcef9418020b025fe8ab3f461cd56533
2022-11-18 21:05:21 +00:00
Michal Niewrzal
8d5a2a90f2 cmd/satellite: repair-segment; add option to process csv file directly
Current option is to put stream id and position as an input but
it's not very efficient when we have long list of segments to repair.
This change adds option to read whole csv file and process each entry
one by one.

If command will have single argument then it will treat it as csv file
location and if will have two arguments then it will parse it just as
stream id and position.

Change-Id: I1e91cf57a794d81d74af0091c24a2e7d00d1fab9
2022-11-18 17:40:17 +00:00
Cameron
87660bd9b3 satellite/overlay/offlinenodes: insert offline nodes into node events
Add a new chore to periodically insert nodes who are offline and
have not gotten an offline email in a certain amount of time into node
events

Change-Id: I658b385bb777b0240c98092946a93d65bee94abc
2022-11-18 12:10:06 -05:00
Michal Niewrzal
ec777855e1 cmd/satellite: add segment-repair command
Implements logic for satellite command to repair a single segment.
Segment will be repaired even if it's healthy. This is not checked
during this process. As a part of repair command will download whole
segment into memory and will try to reupload segment to number of new
nodes that equal to existing number of pieces. After successful
upload new pieces will completely replace existing pieces. 

Command:
    satellite repair-segment <streamid> <position>

https://github.com/storj/storj/issues/5254

Change-Id: I8e329718ecf8e457dbee3434e1c68951699007d9
2022-11-18 16:18:08 +01:00
Cameron
705dfa0875 satellite/nodeevents: add method Name to Type
Add method Name to Type for a more human-readable format

Change-Id: I10050b9ef3691922d9dfccdbb6f046769a649321
2022-11-18 12:31:02 +00:00
Cameron
57be07f60a satellite/nodeevents: add Chore
Create NodeEvents Chore on satellite core to read nodeevents DB and
notify node operators on node events. The chore sends notifications
grouped by email and event type: it selects the oldest entry in
nodeevents.DB and also any other event with the same email and event
type no matter how old it is. The oldest entry of a group must exist for
a minimum amount of time before that group can be selected, however.
This minimum amount of time is a configurable value:
--node-events.selection-wait-period. This wait period allows us to
combine events of the same time and same email address into a singular
email.

Change-Id: I8b444aa324d2dae265cc27d9e9e85faef79195d8
2022-11-18 12:00:14 +00:00
Clement Sam
3e0a4230a5 storagenode/payout: fix disk space value at payout
Payout is still calculating using the tb*h. 0So it’s getting the total disk space used this month and dividing by (24*30) instead of just 30.

More context here: https://forum.storj.io/t/current-month-earnings-in-node-v1-67-1/20319/5

Follow up PR for https://github.com/storj/storj/issues/5146

Change-Id: Ie2d48497f2a9bdbc995c99ee27e70b46580ff638
2022-11-18 01:04:20 +00:00
Erik van Velzen
9fb18a43d8 satellite/metabase/rangedloop: observer interface
New interface for parallel segment loop.

Closes https://github.com/storj/storj/issues/5239

Change-Id: I2bcce6f836f6625da8ceb4fc0fc030c0ea4380e7
2022-11-17 20:12:23 +00:00
Cameron
c86340e6b9 satellite/satellitedb/dbx: change node_event read one to read first
read one is the wrong method when trying to select one row when there
are multiple. It returns TooManyRows error. Read first is the correct
method.

Change-Id: Ic6c92795486892ac041befd118b6945314bffeaa
2022-11-17 19:31:53 +00:00
Cameron
8681a36164 satellite/overlay: add ability to get offline nodes in need of email
Add LastOfflineEmail to overlay.NodeDossier. This is the last time a
node got an offline email. Add two new overlay db methods,
GetOfflineNodesForEmail and UpdateLastOfflineEmail. Edit db method
UpdateCheckIn to nullify last_offline_email if node is up.

Change-Id: I1ee60e7d98dd1b68348a57f9a4fb77c6c9895d6d
2022-11-17 19:03:04 +00:00
Michal Niewrzal
1ad91265bc satellite/metainfo: reduce log to warn
We have code that is used only by old uplinks and can fail at some point
but we don't interrupt anything and only log message about failure.
Until now it was logged as error but it's nothing critial so we can
reduce it to warning.

As an addition log entry was extended with more information about client
that is using this backward compatibility code.

Change-Id: Ie21c673ee59eb10de065cc371132f8f9505e2220
2022-11-17 18:20:59 +00:00
Jeremy Wharton
e339f1a785 satellite/console: enable session timeout by default
This change causes the session inactivity timer to be enabled unless
expressly specified otherwise.

Change-Id: I85b4014394afac2feb21f383cac414cddb09ca8f
2022-11-17 17:41:15 +00:00
Moby von Briesen
776f1128b3 web/satellite: project level passphrase vuex logic
Added new feature flag.
Reworked vuex logic to work properly with project level passphrase.
Implemented new simple set project level passphrase modal.

Issue:
https://github.com/storj/storj/issues/5280

Change-Id: I6a15e90ee9fa7aa8a09c67022466787090120f9c
2022-11-17 16:53:25 +00:00
Michal Niewrzal
1d23c264f6 satellite/metainfo: add storj-downloader as a known user agent
Change-Id: I9977770c43b85aca235d2bc18a454e400dce6e53
2022-11-17 16:03:54 +00:00
Vitalii
28b4e57e9e web/satellite: fix upload buttons
Fixed upload buttons click in file browser for Safari

Change-Id: I9c6195fe231bd26eb9ce99f1f7a9d2073b8207bf
2022-11-17 15:24:08 +00:00
Wilfred Asomani
6fe89d5244 web/satellite: add a button to claim wallet from storjscan
Previously, a wallet is claimed automatically on opening the payment methods page.
This will result in users who may not even use the wallets claiming them.
This change introduces a button that will be clicked before a wallet is claimed.

Issue: https://github.com/storj/storj/issues/5326

Change-Id: Iab2020d80a30315b6b43ad93d33a984663770279
2022-11-17 14:47:57 +00:00
NickolaiYurchenko
5ef109f45a web/satellite: VTable default props console error fixed
Change-Id: I46bea6fd7a2d676f503747c9d9f25f87f394ec09
2022-11-17 14:44:10 +02:00
Stefan Benten
70d42ead1c
satellite/orders/endpoint.go: order rollups by bucket name first
Currently the primary key of the underlying rollup table has the
primary key being the bucket name, but we used to sort by projectID.
This caused dead locks due to the contention during updates/inserts.

We should reevalute if bucket name being the primary key is the right
way for this table, this should stop the long running and failing attempts tho.

Change-Id: Ie7d0f86944da48ad9cbd92eb162226882a2fb954
2022-11-16 19:48:43 +01:00