Commit Graph

7340 Commits

Author SHA1 Message Date
Egon Elbre
b3448606fa satellite/satellitedb/dbx: document project_bucket.dbx
Updates https://github.com/storj/team-delivery/issues/1

Change-Id: I2931af073b7e2e014ce3881e6e5b6e05618ec49b
2023-01-10 17:28:29 +02:00
Lizzy Thomson
c3c239d1f7 web/satellite: update billing history logic
Update billing history logic to exclude empty status invoices

Fixes https://github.com/storj/storj/issues/5427

Change-Id: If600f3dbce511b1c427d8fcde577dd145e693bb1
2023-01-09 17:47:45 +00:00
Egon Elbre
1688e4965b satellite/satellitedb/dbx: split into multiple files
The current satellitedb.dbx has grown quite big over time.
Split the single file into multiple smaller ones.

Change-Id: I8d6ca851b834ac60820aff565906f04aab661875
2023-01-09 16:24:31 +02:00
NickolaiYurchenko
d806c5c49b web/satellite: CreditCardContainer.vue padding bug on 'Default' label fixed
Change-Id: I714614a5d92e4f3d994f03da94dd053e751bbf50
2023-01-06 18:09:38 +00:00
paul cannon
246c193145 cmd/tools/segment-verify: add timeout to VerifyWithExists
We have to wait until the slowest node is done being tested before we
can move on to the next of segments. Since the slowest node can be
arbitrarily slow, we'll set a timeout and treat too-slow nodes as
temporarily offline.

Change-Id: I80fe865dd4e8f826700430fb0140c2d3aefca381
2023-01-06 17:25:30 +00:00
paul cannon
23acee2df0 cmd/tools/segment-verify: better handle timeouts
When we are verifying pieces by downloading the first byte, if we
encounter a timeout, treat the node as if we failed to connect to it,
and log the error once instead of twice.

Change-Id: I70602d554183c98f1213f3ffb1bfec41100ea0e7
2023-01-06 16:52:28 +00:00
paul cannon
3a9ad48345 cmd/tools/segment-verify: fix problem-pieces.csv
This csv file was being closed as soon as the service was created.
All subsequent writes to the closed file handle produced errors,
which were logged but otherwise ignored.

Instead, we would like the file to remain open and writable, until
the service is destroyed.

Change-Id: Ib29944d25b2f5b2d0f90fdbdcde44fea8d769321
2023-01-06 16:23:34 +00:00
Wilfred Asomani
0773b4efbc satellite/console: support id and publicId
Update get usage-limits, daily-usage and salt endpoints to support
both project-ID and project-PublicID.

Issue: https://github.com/storj/storj/issues/5411

Change-Id: Iff0114a295d1a479b141bfffbfb31599844d1fc0
2023-01-06 15:42:45 +00:00
Wilfred Asomani
05c83a5539 satellite/{console,web}: support delete by ID and publicID
Update the delete API key by name and projectID to support project-ID
and project-publicID.

Issue: https://github.com/storj/storj/issues/5410

Change-Id: I3bd11b9c3ae1ad6ce662dfc18b42779d2e4edf9b
2023-01-06 13:40:10 +00:00
Erik van Velzen
23b92da490
satellite/metabase/rangedloop: live reporting (#5366)
Add an observer to monitor ranged segment loop progress.

Tested by running the segment loop in storj-up and navigating to
http://<container>:11111/mon/stats and there is the entry:

rangedloop-live,scope=storj.io/storj/satellite/metabase/rangedloop numSegments=364523630000.000000

part of https://github.com/storj/storj/issues/5223

Change-Id: If3d2774d2f17f51eac86f47c6dda1fb8ad696dfe
2023-01-06 09:49:14 +01:00
Andrew Harding
5efb08cd7b cmd/uplink: skip abort on committed multipart writes
The copyFile method has some safeguards to ensure that the multipart
write is aborted. This is accomplished by always calling abort on the
MultiWriteHandle when the copy is finished, whether or not there was a
failure or it was successfully committed. If the copy was committed,
then this RPC is a no-op on the metainfo server.

Regardless, the calls to abort to constitute an additional RPC to the
satellite for no benefit. This is exacerbated by the fact that the code
currently ends up calling abort twice.

This change updates the libuplink-backed MultiWriteHandle implementation
to not call abort if the write is committed and vice-versa. This
eliminates the two wasteful RPC calls.

Change-Id: I13679234f6f473e9a93179e6791fb57eac512f25
2023-01-05 18:30:23 +00:00
Lizzy Thomson
591615b90b satellite/satellitedb: remove references to last_verification_reminder
Removing all references to column last_verification_reminder which is to be removed, due to new column verification_reminders

Issue: https://github.com/storj/storj/issues/4560

Change-Id: I7c9a426e946c7aed58e62c1eef80629daf6b1272
2023-01-05 08:58:05 -07:00
Erik van Velzen
1d4411f166
satellite/metabase/rangedloop: cancellation (#5364)
Support interruption of the ranged segment loop through context.

Part of https://github.com/storj/storj/issues/5223

Change-Id: Iae0260e250f8ea33affed95c6592a1f42df384eb
2023-01-05 16:32:30 +01:00
Qweder93
8c69ee62fc {cmd/storj-sim, satellite/rangedloop}: added rangedloop to storj-sim, removed identity
added in storj-sim rangedloop for each satellite, to verify it works for metrics oveserver,
removed identity from rangedloop peer as we never use it, added logs on service run, added loop
to service instead of endless for loop, interval value to config

Closes: https://github.com/storj/storj/issues/5414

Change-Id: Ibc3b06071b68feda4a35b45da2bbe36e22a02fc8
2023-01-05 11:29:00 +00:00
Cameron
5138bb1e4f satellite/console: add public ID field to graphql Project
Add public ID field to graphql Project so it can be used on the front
end. Additionally public_id needed to be added to the ListByOwnerID sql
query which is called by graphql OwnedProjectsQuery.

github issue: https://github.com/storj/storj/issues/5408

Change-Id: I2ec04363c20493dc0f9c70b6d1610f724f18ec2f
2023-01-04 15:00:20 -05:00
paul cannon
6e1554652a cmd/tools/segment-verify: handle dq'd nodes
Previously, if any pieces are still on disqualified nodes, this tool
would treat those pieces as fine (if the disqualified node is still
online) or temporarily unavailable (if the disqualified node is
offline). Instead, we should treat such pieces as lost.

This also fixes a slight problem with the code that handles a broken
alias. This is not likely to happen, but if we do see an alias that is
not in the alias map, we return an error instead of nil.

Change-Id: Ib4e2e729ef0535dd7bd9ce2f621680d9f959891c
2023-01-04 17:54:03 +00:00
nadimhq
63639e862b
docs/testplan: Testplan for Scaling Audit Worker (#5335)
This testplan is going to cover the changes to allow for scaling audit workers. It will go over the scaling audit worker design doc found under blueprints.

Co-authored-by: Antonio Franco (He/Him) <antonio@storj.io>
2023-01-04 18:49:53 +01:00
Ethan Adams
d29abed2aa cmd/satellite: Add run auditor to satellite entrypoint.
Needed to support new auditor processess

Change-Id: I6687a667c123c69a335317216affad3a14ab7b9c
2023-01-04 15:31:06 +00:00
Wilfred Asomani
ab8f22a6d0 web/satellite: add zkSync warning message
Slightly modified the warning message on the "Add STORJ Tokens" modal;
And added a warning for zkSync.

Issue: https://github.com/storj/storj/issues/5418

Change-Id: I206e7b493c3fe04c69a3815a5f03bd6a07cfceae
2023-01-04 14:41:16 +00:00
Erik van Velzen
1da9fd1eee
satellite/metabase/rangedloop: monkit durations (#5365)
Wire up duration measurement of observers with monkit.

Tested by attaching a SleepObserver, starting the rangedloop in storj-up
and navigating to http://<container>:11111/mon/stats. It reports the
following statistic:

completed-observer-duration,observer=*rangedlooptest.SleepObserver,scope=storj.io/storj/satellite/metabase/rangedloop duration=10.000117

Change-Id: Ief131d34001dd5d3ba1d7be6f161986e1f66440d
2023-01-04 12:16:47 +00:00
Michal Niewrzal
77afdae741 satellite/metabase: handle target pending/committed objects while move
Before we introduced objects versions internally move operation was
always failing when under target location object exists. But then we
had only single version 1 all the time. With versions different than 1
we need to check all existing objects under target location.

To be backward compatible with our API new logic looks like this:
* if there is no object under target location use source object version
as target version
* if there are only pending objects find first free (highest) version
which could be used to move object there
* if there is committed object under target location reject move
operation

Fixes https://github.com/storj/storj/issues/5403

Change-Id: I717f3e7c42470b406287d6ec335f6f057d3fc3b5
2023-01-04 08:50:51 +00:00
paul cannon
2feb49afc3 cmd/tools/segment-verify: don't cache offline status forever
Because it was originally intended to work on only a few pieces from
each segment at a time, and would frequently have reset its list of
online nodes, segment-verify has been taking nodes out of its
onlineNodes set and never putting them back. This means that over a long
run in Check=0 mode, we end up treating more and more nodes as offline,
permanently. This trend obfuscates the number of missing pieces that
each segment really has, because we don't check pieces on offline nodes.

This commit changes the onlineNodes set to an "offlineNodes" set, with
an expiration time on the offline-ness quality. So nodes are added to
the offlineNodes set when we see they are offline, and then we only
treat them as offline for the next 30 minutes (configurable). After that
point, we will try connecting to them again.

Change-Id: I14f0332de25cdc6ef655f923739bcb4df71e079e
2023-01-03 23:11:42 +00:00
Andrew Harding
5362dff94b satellite/gc/bloomfilter: implement rangedloop observer
https://github.com/storj/storj/issues/5235

Change-Id: Iffe8f682adfa46e48e47976bf838326e7125ff80
2023-01-03 09:46:02 -07:00
Andrew Harding
590d44301c satellite/audit: implement rangedloop observer
This change implements the ranged loop observer to replace the audit
chore that builds the audit queue.

The strategy employed by this change is to use a collector for each
segment range to  build separate per-node segment reservoirs that are
then merge them during the join step.

In previous observer migrations, there were only a handful of tests so
the strategy was to duplicate them. In this package, there are dozens
of tests that utilize the chore. To reduce code churn and maintenance
burden until the chore is removed, this change introduces a helper that
runs tests under both the chore and observer, providing a pair of
functions that can be used to pause or run the queueing function.

https://github.com/storj/storj/issues/5232

Change-Id: I8bb4b4e55cf98b1aac9f26307e3a9a355cb3f506
2023-01-03 08:52:01 -07:00
Egon Elbre
9544a670d7 storagenode/pieces: fix concurrent empty and restore trash
This ensures that empty trash and restore trash cannot run at the same
time.

Fixes https://github.com/storj/storj/issues/5416

Change-Id: I9d2e3aa3d66e61e5c8a7427a95208bb96089792d
2023-01-03 15:01:54 +00:00
paul cannon
46d99a06d5 cmd/tools/segment-verify: write to pieces csv from WithExists methods
The WithExists methods previously were not writing problematic pieces to
problem-pieces.csv. With this change, they will.

Change-Id: I51eadd3d8f4299e1efa787c9266a7aacfa525eb3
2022-12-28 01:12:46 +00:00
paul cannon
9544936794 cmd/tools/segment-verify: don't double-count notfound
When this branch is followed, `audit.OutcomeFailure` is returned, and
`MarkNotFound()` is immediately called again (in
`(*NodeVerifier).Verify()`). Calling `MarkNotFound()` twice for the same
piece is not correct.

Change-Id: I1a2764bc32ed015628fcd9353ac3307f269b4bbd
2022-12-28 00:37:14 +00:00
paul cannon
aec596bb39 cmd/tools/segment-verify: monkit-ify WithExists methods
It may help to know how much faster these methods are than the
alternative (asking nodes for each piece in turn).

Change-Id: Ieb7c963f62b662f72c84a49de8a09c065c14f782
2022-12-28 00:00:40 +00:00
paul cannon
42e2a14316 cmd/tools/segment-verify: flush after write to pieces csv
It was ok as it was, but since we want to keep a close eye on progress
while the tool is running, it will help to have results written to the
output file immediately instead of after the buffer is full or the
program exits.

Change-Id: Ie027f05771a637afb06969ec775cd32b142b7635
2022-12-27 14:58:54 -06:00
NickolaiYurchenko
96e7c1341f web/satellite: ShareContainer, ShareButton refactored and migrated to use composition api
Change-Id: Ic8b2d2b70cb5a614858b352064f0619b5a518b3d
2022-12-23 21:42:00 +00:00
Wilfred Asomani
ea00213b45 console/userinfo: implement get userinfo
This change implements the get userinfo endpoint stubbed in #5358

Issue: https://github.com/storj/storj/issues/5363

Change-Id: I8d8deb0be1375395bbf3f52e7718990bc77f00b5
2022-12-23 21:03:54 +00:00
Cameron
fb95b5d655 docs/blueprints: Satellite Node Events Design Doc
Add design for identifying and processing node events and sending
notifications to node operators.

github issue: https://github.com/storj/storj/issues/5039

Change-Id: I63341867207c28334bb025c6a104692ce2a16edd
2022-12-23 20:15:00 +00:00
Kaloyan Raev
56896353b6 cmd/uplink: add buffering while reading from stdin
This change is similar to
https://review.dev.storj.io/c/storj/storj/+/7687 but applied when
uploading from stdin with parallelism > 1.

Currently, the paralellism from stdin scales up to 3 or 4, but not
greater than that. If we buffer the content from stdin more aggressively
the parallelism scales to higher levels and reaches the performance of
reading directly from a file.

Change-Id: I1f447686a88074882709992ee6d52dd262e220fb
2022-12-23 16:40:54 +00:00
NickolaiYurchenko
b683f83994 web/satellite: PasswordStrength, NewBillingAddCouponCodeInput components migrated to use composition api
Change-Id: I1b878a7bce574c34ec424e7c8b94077c65fa9529
2022-12-23 15:55:59 +00:00
Vitalii
af238e2ef9 web/satellite: removed old create access grant flow
Removed old flow and feature flag.

Issue:
https://github.com/storj/storj/issues/5407

Change-Id: I9dec18eb7d8c7912ead87188789466db5f59a5ca
2022-12-23 15:14:02 +00:00
paul cannon
b2422caaef cmd/tools/segment-verify: log less retry segments
When Check == 0 (check all pieces), there is nearly always a piece left
in the retry count, so most segments get logged in segments-retry.csv.
This change makes it so we require retry>5 before adding to
segments-retry.csv (only in the check==0 case).

Change-Id: Iaea523c27eb777e3c248c27c7ef5effe77ae54cf
2022-12-23 14:29:25 +00:00
Andrew Harding
6492a777eb satellite/gracefulexit: fix typo in comment
Change-Id: Iadb6a2e550c92cbc29e9e5d50353713035d92eda
2022-12-23 07:11:35 -07:00
NickolaiYurchenko
ad9fbe2953 web/satellite: SelectInput, RegistrationSuccess components migrated to use composition api
Change-Id: Ic2d0e152d48e7b5b57780712905d51ab1139a77d
2022-12-23 12:17:06 +00:00
paul cannon
0b790070a3 cmd/tools/segment-verify: pass over bad segments
Change-Id: I1b4dd9da755c6a2028760723e15219f5821f702f
2022-12-22 18:12:12 -06:00
Kaloyan Raev
bfd189c3b0 cmd/uplink: add --inmemory-erasure-coding flag to cp command
This new advanced flag configures libuplink to store in-memory the
erasure-coded pieces that are temporarily created during upload.

By default, libuplink writes the erasure-coded pieces as temp files on
the disk, but this results in additional IOPS that affect the
performance in hot-rodded scenarios.

If the erasure-coded pieces are kept in-memory and the system has enough
RAM, the upload speed may be boosted with 20-30%.

The flag is added as "advanced" as we don't recommend it by default.

Co-authored-by: Stefan Benten <mail@stefan-benten.de>

Change-Id: Icc54f03b6c0bc27c97126f6f1d22748d21a15959
2022-12-22 19:48:58 +00:00
Malcolm Bouzi
9a59b75159 web/satellite: make date on billing page look unclickable.
ticket: https://github.com/storj/storj/issues/5315
Change-Id: I249bfd3025a278b3a985318f41725ea70fc2135d
2022-12-22 19:06:14 +00:00
Lizzy Thomson
db489125b8 satellite/console: add triggerAttemptPaymentIfFrozen
add triggerAttemptPaymentIfFrozen to check if the account is frozen
and if frozen, will trigger an attempt to pay outstanding invoices

Issue: https://github.com/storj/storj/issues/5398

Change-Id: I0da6a982e2da4204dee219d98ce2d503cbbb6f8e
2022-12-22 18:23:37 +00:00
Andrew Harding
4241e6bf5f satellite/gracefulexit: implement rangedloop observer
The tests are forked from the chore tests with slight adaptations for
being run against the ranged loop. I also moved a benchmark for the
database from chore_test.go to db_test.go.

The pathcollector is reused as a rangedloop.Partial.

https://github.com/storj/storj/issues/5234

Change-Id: I56182031d133812a9f4d4a433c01b9150af39f31
2022-12-22 10:47:10 -07:00
NickolaiYurchenko
013e74f804 web/satellite: route to billing page on frozen status banner changed
redirect path changed from billing overview to billing payment methods;
fixed router path change beeing not tracked;

Change-Id: Ib5b11d4819ec8fe43acbe624a937f9364921d13a
2022-12-22 19:01:18 +02:00
Malcolm Bouzi
d20a2e9a7d web/satellite: update copy on the coupon code application modal.
ticket: https://github.com/storj/storj/issues/5314
Change-Id: I24226f0565670dd87c2c21dbfea1eaf667854bdc
2022-12-22 16:03:53 +00:00
Michal Niewrzal
4851b4e06d cmd/tools/segment-verify: small improvements
* better error handling when Exists method is not avaialble on SN
* more optimal processing of response from Exists method

Change-Id: I6d61c09473e9f5ab76a4601720e8bd520767f4c2
2022-12-22 15:21:33 +00:00
Malcolm Bouzi
5d13a5e0ed web/satellite: update loading screen to include branding.
ticket: https://github.com/storj/storj/issues/5278

Change-Id: I02d5d237ce03224c0b9af5428d6853b32d346f5e
2022-12-21 16:06:43 -05:00
Erik van Velzen
37b4981cc0
satellite/metabase/rangedloop: measure observer duration (#5350)
Track duration of all segment loop observers. Factor out functions to
reduce size.

Still need to send the measurements out via monkit.

Part of https://github.com/storj/storj/issues/5223

Change-Id: Iae0260e250f8ea33affed95c6592a1f42df384eb
2022-12-21 21:58:08 +01:00
Clement Sam
cda1d67465 cmd/tools/segment-verify: adjust to SN Exists endpoint
Change-Id: I409aeae29aa87996f2a6047f976d215a69e9d7f5
2022-12-21 19:24:31 +00:00
Malcolm Bouzi
9535444dcd web/satellite: adds satellite selection through tab control on the forget password page.
ticket: https://github.com/storj/storj/issues/5164
Change-Id: I1e0cefe75883a12c7b26ad42b7c44fddb7e93d10
2022-12-20 16:59:04 +00:00