It helps for the (*reverifyQueue).Insert() method to be idempotent (it
does not make sense for the same node to be under containment for the
same piece multiple times). This change allows for that, by adding an
`ON CONFLICT DO NOTHING` clause to the database query.
Refs: https://github.com/storj/storj/issues/5231
Change-Id: Id2839ee185d5396c0bc2f84ffad610df9786f6c7
Adding a new worker comparable to Verifier, called Reverifier; as the
name suggests, it will be used for reverifications, whereas Verifier
will be used for verifications.
This allows distinct logging from the two classes, plus we can add some
configuration that is specific to the Reverifier.
There is a slight modification to GetNextJob that goes along with this.
This should have no impact on operational concerns.
Refs: https://github.com/storj/storj/issues/5251
Change-Id: Ie60d2d833bc5db8660bb463dd93c764bb40fc49c
The default 'info' level for the storagenode will dump dozens of
lines every second. This change adds the ability to configure
the log.level argument at run time using LOG_LEVEL env variable.
Co-authored-by: Clement Sam <clementsam75@gmail.com>
This change turns off fsync on the postgres container used for tests. This
reduces migration time significantly when initializing new satellite
databases.
The change also includes a new benchmark for satellite initialization in
testplanet.
$ benchstat old.txt new.txt name old time/op new time/op delta
Run_Satellite/Postgres-16 1.36s ± 0% 0.08s ± 0% ~ (p=1.000 n=1+1)
Change-Id: Ic954767133864770cf652b0dfdcd6b109a167b5f
Running all of the migrations necessary to initialize a storage node
database takes a significant amount of time during runs.
The package current supports initializing a database from manually coalesced
migration data (i.e. snapshot) which improves the situation somewhat.
This change takes things a bit further by changing the snapshot code to
instead hydrate the database directory from a pre-generated snapshot zip
file.
name old time/op new time/op delta
Run_StorageNodeCount_4/Postgres-16 2.50s ± 0% 0.16s ± 0% ~ (p=1.000 n=1+1)
Change-Id: I213bbba5f9199497fbe8ce889b627e853f8b29a0
Previously, the node events chore would select based on the earliest
created_at. However, if for some reason this batch fails, it would still
be the next item to select. If there is a consistent error, the chore
would be stuck retrying the same batch over and over. Now instead
GetNextBatch orders by `last_attempted NULLS FIRST ASC, created_at ASC`.
If a batch fails during Notify, last_attempted is updated so we can move
on to a new batch if one exists.
Change-Id: Ia8458e05ac358d85b2f2c6d690f3d607d631be61
audit.Queues was the previous method of passing stacks of segments for
audit to the verifier. As of commit 68f9ce4a, this is now happening
by way of the auditor queue (database-backed, so that communication can
happen between multiple peers). audit.Queues is no longer needed.
Refs: https://github.com/storj/storj/issues/5228
Change-Id: I46f2d48d655fb66366c92146cdb6b85aef200552
SetNodeContained() will change the contained flag in the nodes table,
which will affect whether nodes are selected for new uploads. This flag
_should_ correlate with whether or not a given node has any entries in
the reverification queue. However, the reverification queue is intended
to be 'safely partitionable' from the nodes table, so we can't enforce
that characteristic transactionally. But this is ok; there are no dire
consequences if they are out of sync.
We will be adding a chore that updates the contained flag based on the
contents of the reverification queue periodically, if something fails
to set it directly when appropriate.
Refs: https://github.com/storj/storj/issues/5231
Change-Id: I26460d8718dee63fd55d00a44568b2065fc8fe30
GetByNodeID will allow querying the reverification queue to see if there
are any pending jobs for a given node ID. And thus, to see if that node
ID should be contained or not.
Some parameters on the other methods of the ReverifyQueue interface have
been changed to accept pointers; this was done ahead of the rest of the
changes for the reverification queue to better match the signatures of
the methods that these will replace once ReverifyQueue is actually being
used (meaning fewer changes to tests).
Refs: https://github.com/storj/storj/issues/5251
Change-Id: Ic38ce6d2c650702b69f1c7244a224f00a34893a1
This change removes the error type that is returned when a token
request contains an incorrect password. Instead, the generic error
type for invalid login credentials is used.
Change-Id: Ia7dbc38f4a08aeaeeac7ff5b5a801233e349b8b3
Added create project passphrase modal.
User can select between generated and entered passphrase.
Issue:
https://github.com/storj/storj/issues/5324
Change-Id: I69887562230c7c8002e9bc763ac24c551a0caa1d
This change reduces the token links expiry time from 24h to 30m and improves the UI to promt users of the expiration.
see: https://github.com/storj/storj-private/issues/17
Change-Id: Iac00f5740fa84069937fdf9bd30a739b6db2a9e0
The audit chore will be pushing a large number of segments to be
audited, and the db might choke on that large insert when under load.
This change divides the insert up into batches, which can be sized
however is optimal for the backing database. It also arranges for
segments to be inserted in the order of the primary key, which helps
performance on some systems.
Refs: https://github.com/storj/storj/issues/5228
Change-Id: I941f580f690d681b80c86faf4abca2995e37135d
Added server-side encryption banner to new project dashboard (same as on buckets view).
Reduced notAfter AG caveat to 1 hour for deleting bucket.
Added bucket AG caveat for deleting bucket.
Change-Id: I2aba9e5db315123b8d37c24d80789320093db00e
* storj/common
* storj/private
Latests common version requires small refactoring for names and types
used by metainfo code.
Change-Id: I224fe93b4751c996ba6e846be0e5677252cf830f
Uplink doesn't have a `save` command, however, it's referred on an error
message that's returned when the `access register` command is executed
without having any default access configured.
The correct command to mention is `import`.
Change-Id: Ia2092d02965737f421683fc98c52a51c9529b86e
Reputation updates during repair currently consumes a lot of database
resources. Sometimes increasing the rate of repair is more important
than auditing a node based on whether they have or don't have the
correct piece during repair. This is the job of the audit service.
This commit is to implement an intermediate solution from this issue: https://github.com/storj/storj/issues/5089
This commit does not address the more in-depth fix discussed here: https://github.com/storj/storj/issues/4939
Change-Id: I4163b18d78a96fadf5265789fd73c8aa8def0e9f
This reverts commit 9022506292.
Reason for revert: Execute permissions are not required for this file. The error "Uncaught DOMException: Failed to execute 'importScripts' on 'WorkerGlobalScope':" was instead caused by a browser cached version of import returning an incorrect response.
Change-Id: Ib5f6bf30dc5a9919bb7d71fb2b04075486c74f89
This change causes rate limiting errors to be returned to the client
as JSON objects rather than plain text to prevent the satellite UI from
encountering issues when trying to parse them.
Resolvesstorj/customer-issues#88
Change-Id: I11abd19068927a22f1c28d18fc99e7dad8461834
We tested new upload flow (with multiple versions) to fix inconsistency
while uploading object on QA/EUN1/SLC. Now we would like to enable it
for all satellites by default. Tests required small adjustments.
Fixes https://github.com/storj/storj/issues/5283
Change-Id: I0d53c041abebc0d182ba5a88bb1dac906c29caf0
Reworked bucket tables to be the same on buckets view and project dashboard.
Improved table adaptation.
User can open/delete bucket from project dashboard.
User can view bucket details from project dashboard.
User can search through buckets from buckets view.
Issues:
https://github.com/storj/storj/issues/5176https://github.com/storj/storj/issues/5174
Change-Id: Ie4599ab62dc7eeba9a3349ab188cde8a198cc0f3
As part of the effort of splitting out the auditor workers to their own
process, we are transitioning the communication between the auditor
chore and the verification workers to a queue implemented in the
database, rather than the sequence of in-memory queues we used to use.
This logical database is safely partitionable from the rest of
satelliteDB.
Refs: https://github.com/storj/storj/issues/5251
Change-Id: I6cd31ac5265423271fbafe6127a86172c5cb53dc
We added alternative way to calculate bucket tallies for accounting and
now it's tested and we will enable it by default.
CollectBucketTallies was extended to support overriding current time
to be able to test handling expired objects.
Change-Id: I738b99a33fd2e086245f92d874c1cbb806e834c0
If we are processing list of segments (csv) we should not stop if one of
segments is not found in DB.
Change-Id: I720f85dc7601c2ca77032e20c1577de55092bd9b
Execute permissions are needed on the wasm_exec.js file but nothing seems to set this by default. Once set, it will keep permissions even after rebuilds, but this change will set the initial permissions if required for any new environments.
Change-Id: Ic6848e561210231e67dd2e17a286abdf885926ed
This change fixes wrong positioning of pop ups (the bucket guide, and bucket item drop down) on Safari.
Change-Id: Ia0abcc1a1450cb21e38a435218fb6c788ce546e2
When attempting to upload a file on Firefox mobile, an error
"item.webkitRelativePath is undefined" prevents the client from
proceeding with the upload after the file is selected. This commit
contains a small check to protect against this issue.
Additionally, in Firefox (desktop, maybe mobile), links in the file
browser to enter subfolders, exit subfolders, or navigate out of folders
via breadcrumbs has default behavior of navigating the user to an empty
page that only displays "null". This changes adds `@click.prevent` in
some key locations to prevent this undesired behavior.
Change-Id: I83ee6fcc5a7d0ce9996dacc3f966e38a4936a9fe
This change resolves an issue where text inside the Billing page's
Storj Token card would appear over the project dropdown menu.
The issue was caused by using z-indexing to prevent an absolutely
positioned cosmetic element from displaying over its non-positioned
siblings. An alternate solution is presented in this change:
specifying a position for the non-cosmetic elements allows them to
display properly over preceding positioned elements without the usage
of z-indexing.
Resolves#5219
Change-Id: Icebbda33dcef9418020b025fe8ab3f461cd56533
Current option is to put stream id and position as an input but
it's not very efficient when we have long list of segments to repair.
This change adds option to read whole csv file and process each entry
one by one.
If command will have single argument then it will treat it as csv file
location and if will have two arguments then it will parse it just as
stream id and position.
Change-Id: I1e91cf57a794d81d74af0091c24a2e7d00d1fab9
Add a new chore to periodically insert nodes who are offline and
have not gotten an offline email in a certain amount of time into node
events
Change-Id: I658b385bb777b0240c98092946a93d65bee94abc
Implements logic for satellite command to repair a single segment.
Segment will be repaired even if it's healthy. This is not checked
during this process. As a part of repair command will download whole
segment into memory and will try to reupload segment to number of new
nodes that equal to existing number of pieces. After successful
upload new pieces will completely replace existing pieces.
Command:
satellite repair-segment <streamid> <position>
https://github.com/storj/storj/issues/5254
Change-Id: I8e329718ecf8e457dbee3434e1c68951699007d9
