Commit Graph

7318 Commits

Author SHA1 Message Date
Andrew Harding
5362dff94b satellite/gc/bloomfilter: implement rangedloop observer
https://github.com/storj/storj/issues/5235

Change-Id: Iffe8f682adfa46e48e47976bf838326e7125ff80
2023-01-03 09:46:02 -07:00
Andrew Harding
590d44301c satellite/audit: implement rangedloop observer
This change implements the ranged loop observer to replace the audit
chore that builds the audit queue.

The strategy employed by this change is to use a collector for each
segment range to  build separate per-node segment reservoirs that are
then merge them during the join step.

In previous observer migrations, there were only a handful of tests so
the strategy was to duplicate them. In this package, there are dozens
of tests that utilize the chore. To reduce code churn and maintenance
burden until the chore is removed, this change introduces a helper that
runs tests under both the chore and observer, providing a pair of
functions that can be used to pause or run the queueing function.

https://github.com/storj/storj/issues/5232

Change-Id: I8bb4b4e55cf98b1aac9f26307e3a9a355cb3f506
2023-01-03 08:52:01 -07:00
Egon Elbre
9544a670d7 storagenode/pieces: fix concurrent empty and restore trash
This ensures that empty trash and restore trash cannot run at the same
time.

Fixes https://github.com/storj/storj/issues/5416

Change-Id: I9d2e3aa3d66e61e5c8a7427a95208bb96089792d
2023-01-03 15:01:54 +00:00
paul cannon
46d99a06d5 cmd/tools/segment-verify: write to pieces csv from WithExists methods
The WithExists methods previously were not writing problematic pieces to
problem-pieces.csv. With this change, they will.

Change-Id: I51eadd3d8f4299e1efa787c9266a7aacfa525eb3
2022-12-28 01:12:46 +00:00
paul cannon
9544936794 cmd/tools/segment-verify: don't double-count notfound
When this branch is followed, `audit.OutcomeFailure` is returned, and
`MarkNotFound()` is immediately called again (in
`(*NodeVerifier).Verify()`). Calling `MarkNotFound()` twice for the same
piece is not correct.

Change-Id: I1a2764bc32ed015628fcd9353ac3307f269b4bbd
2022-12-28 00:37:14 +00:00
paul cannon
aec596bb39 cmd/tools/segment-verify: monkit-ify WithExists methods
It may help to know how much faster these methods are than the
alternative (asking nodes for each piece in turn).

Change-Id: Ieb7c963f62b662f72c84a49de8a09c065c14f782
2022-12-28 00:00:40 +00:00
paul cannon
42e2a14316 cmd/tools/segment-verify: flush after write to pieces csv
It was ok as it was, but since we want to keep a close eye on progress
while the tool is running, it will help to have results written to the
output file immediately instead of after the buffer is full or the
program exits.

Change-Id: Ie027f05771a637afb06969ec775cd32b142b7635
2022-12-27 14:58:54 -06:00
NickolaiYurchenko
96e7c1341f web/satellite: ShareContainer, ShareButton refactored and migrated to use composition api
Change-Id: Ic8b2d2b70cb5a614858b352064f0619b5a518b3d
2022-12-23 21:42:00 +00:00
Wilfred Asomani
ea00213b45 console/userinfo: implement get userinfo
This change implements the get userinfo endpoint stubbed in #5358

Issue: https://github.com/storj/storj/issues/5363

Change-Id: I8d8deb0be1375395bbf3f52e7718990bc77f00b5
2022-12-23 21:03:54 +00:00
Cameron
fb95b5d655 docs/blueprints: Satellite Node Events Design Doc
Add design for identifying and processing node events and sending
notifications to node operators.

github issue: https://github.com/storj/storj/issues/5039

Change-Id: I63341867207c28334bb025c6a104692ce2a16edd
2022-12-23 20:15:00 +00:00
Kaloyan Raev
56896353b6 cmd/uplink: add buffering while reading from stdin
This change is similar to
https://review.dev.storj.io/c/storj/storj/+/7687 but applied when
uploading from stdin with parallelism > 1.

Currently, the paralellism from stdin scales up to 3 or 4, but not
greater than that. If we buffer the content from stdin more aggressively
the parallelism scales to higher levels and reaches the performance of
reading directly from a file.

Change-Id: I1f447686a88074882709992ee6d52dd262e220fb
2022-12-23 16:40:54 +00:00
NickolaiYurchenko
b683f83994 web/satellite: PasswordStrength, NewBillingAddCouponCodeInput components migrated to use composition api
Change-Id: I1b878a7bce574c34ec424e7c8b94077c65fa9529
2022-12-23 15:55:59 +00:00
Vitalii
af238e2ef9 web/satellite: removed old create access grant flow
Removed old flow and feature flag.

Issue:
https://github.com/storj/storj/issues/5407

Change-Id: I9dec18eb7d8c7912ead87188789466db5f59a5ca
2022-12-23 15:14:02 +00:00
paul cannon
b2422caaef cmd/tools/segment-verify: log less retry segments
When Check == 0 (check all pieces), there is nearly always a piece left
in the retry count, so most segments get logged in segments-retry.csv.
This change makes it so we require retry>5 before adding to
segments-retry.csv (only in the check==0 case).

Change-Id: Iaea523c27eb777e3c248c27c7ef5effe77ae54cf
2022-12-23 14:29:25 +00:00
Andrew Harding
6492a777eb satellite/gracefulexit: fix typo in comment
Change-Id: Iadb6a2e550c92cbc29e9e5d50353713035d92eda
2022-12-23 07:11:35 -07:00
NickolaiYurchenko
ad9fbe2953 web/satellite: SelectInput, RegistrationSuccess components migrated to use composition api
Change-Id: Ic2d0e152d48e7b5b57780712905d51ab1139a77d
2022-12-23 12:17:06 +00:00
paul cannon
0b790070a3 cmd/tools/segment-verify: pass over bad segments
Change-Id: I1b4dd9da755c6a2028760723e15219f5821f702f
2022-12-22 18:12:12 -06:00
Kaloyan Raev
bfd189c3b0 cmd/uplink: add --inmemory-erasure-coding flag to cp command
This new advanced flag configures libuplink to store in-memory the
erasure-coded pieces that are temporarily created during upload.

By default, libuplink writes the erasure-coded pieces as temp files on
the disk, but this results in additional IOPS that affect the
performance in hot-rodded scenarios.

If the erasure-coded pieces are kept in-memory and the system has enough
RAM, the upload speed may be boosted with 20-30%.

The flag is added as "advanced" as we don't recommend it by default.

Co-authored-by: Stefan Benten <mail@stefan-benten.de>

Change-Id: Icc54f03b6c0bc27c97126f6f1d22748d21a15959
2022-12-22 19:48:58 +00:00
Malcolm Bouzi
9a59b75159 web/satellite: make date on billing page look unclickable.
ticket: https://github.com/storj/storj/issues/5315
Change-Id: I249bfd3025a278b3a985318f41725ea70fc2135d
2022-12-22 19:06:14 +00:00
Lizzy Thomson
db489125b8 satellite/console: add triggerAttemptPaymentIfFrozen
add triggerAttemptPaymentIfFrozen to check if the account is frozen
and if frozen, will trigger an attempt to pay outstanding invoices

Issue: https://github.com/storj/storj/issues/5398

Change-Id: I0da6a982e2da4204dee219d98ce2d503cbbb6f8e
2022-12-22 18:23:37 +00:00
Andrew Harding
4241e6bf5f satellite/gracefulexit: implement rangedloop observer
The tests are forked from the chore tests with slight adaptations for
being run against the ranged loop. I also moved a benchmark for the
database from chore_test.go to db_test.go.

The pathcollector is reused as a rangedloop.Partial.

https://github.com/storj/storj/issues/5234

Change-Id: I56182031d133812a9f4d4a433c01b9150af39f31
2022-12-22 10:47:10 -07:00
NickolaiYurchenko
013e74f804 web/satellite: route to billing page on frozen status banner changed
redirect path changed from billing overview to billing payment methods;
fixed router path change beeing not tracked;

Change-Id: Ib5b11d4819ec8fe43acbe624a937f9364921d13a
2022-12-22 19:01:18 +02:00
Malcolm Bouzi
d20a2e9a7d web/satellite: update copy on the coupon code application modal.
ticket: https://github.com/storj/storj/issues/5314
Change-Id: I24226f0565670dd87c2c21dbfea1eaf667854bdc
2022-12-22 16:03:53 +00:00
Michal Niewrzal
4851b4e06d cmd/tools/segment-verify: small improvements
* better error handling when Exists method is not avaialble on SN
* more optimal processing of response from Exists method

Change-Id: I6d61c09473e9f5ab76a4601720e8bd520767f4c2
2022-12-22 15:21:33 +00:00
Malcolm Bouzi
5d13a5e0ed web/satellite: update loading screen to include branding.
ticket: https://github.com/storj/storj/issues/5278

Change-Id: I02d5d237ce03224c0b9af5428d6853b32d346f5e
2022-12-21 16:06:43 -05:00
Erik van Velzen
37b4981cc0
satellite/metabase/rangedloop: measure observer duration (#5350)
Track duration of all segment loop observers. Factor out functions to
reduce size.

Still need to send the measurements out via monkit.

Part of https://github.com/storj/storj/issues/5223

Change-Id: Iae0260e250f8ea33affed95c6592a1f42df384eb
2022-12-21 21:58:08 +01:00
Clement Sam
cda1d67465 cmd/tools/segment-verify: adjust to SN Exists endpoint
Change-Id: I409aeae29aa87996f2a6047f976d215a69e9d7f5
2022-12-21 19:24:31 +00:00
Malcolm Bouzi
9535444dcd web/satellite: adds satellite selection through tab control on the forget password page.
ticket: https://github.com/storj/storj/issues/5164
Change-Id: I1e0cefe75883a12c7b26ad42b7c44fddb7e93d10
2022-12-20 16:59:04 +00:00
NickolaiYurchenko
0e3e588718 web/satellite: frozen status check and banner added
added api call, user isFrozen status;
new banner added;
DashboardArea migrated to use composition api;

Change-Id: Ia3fe68c2239064b2b729c6de14d7fc1dc8f4cf3b
2022-12-20 17:58:49 +02:00
Wilfred Asomani
54559be966 console/userinfo: return appropriate errors
This modifies the userinfo endpoint to return appropriate errors;
PermissionDenied for untrusted peers and Unimplemented because
the endpoint isn't implemented

Change-Id: I5109bb204b5e1ce2e21fe16b003991b6c900a8ce
2022-12-20 10:50:18 +00:00
Vitalii
2ebdc5ff2f web/satellite: unauthorized error (401) interception for http requests
Implemented interception for http requests.
We redirect user to login page on every 401 response.

Issue:
https://github.com/storj/storj/issues/5339

Change-Id: Icba4fc0031cb2b4e682a1be078cdcf95b7fa6bfe
2022-12-20 07:14:17 +00:00
Wilfred Asomani
e598c2b3b1 console/userinfo: stub userinfo endpoint
This change stubs userinfo endpoint from storj/common/pb/userinfo.proto.
It also adds config for allowed peers, and a method for verifying peers.

Issue: https://github.com/storj/storj/issues/5358

Change-Id: I057a0e873a9e9b3b9ad0bba69305f0d708bd9b9e
2022-12-20 06:10:08 +00:00
Vitalii
dcb16d83dd web/satellite: fix for open bucket and bucket search issues
Fixed bucket search not being cleared on component unmount.
Fixed open bucket issue for project dashboard presented by the conflict with vue-click-outside directive.

Change-Id: I3682c6b131dcdb902369cf74b31c1b270cd15b08
2022-12-20 01:12:06 +02:00
Cameron
01932bda42 satellite/admin: add endpoint to freeze/unfreeze user
Allow the admin to manually freeze/unfreeze users.

github issue: https://github.com/storj/storj/issues/5397

Change-Id: I402ad1bf2e13effb0a5a8ff35bb128d1fcf18448
2022-12-19 18:11:05 +00:00
Fadila Khadar
d23e25ce0f cmd/tools/segment-verify: remove unused test code
Accidentally added some code to a test. As it is unused, this PR removes it.

Change-Id: I7adddc78c5ed747225e365989ab58504a9625ad7
2022-12-19 14:33:08 +00:00
Michal Niewrzal
d1d617d654 storagenode/piecestore: small cleanup
* normalize ExistsCheckWorkers flag to avoid setting incorrect values
* wait for limiter to finish if context was canceled

Change-Id: I688a395bd958cd09233605fc264d43f91ec45ed1
2022-12-19 12:37:12 +00:00
Ethan Adams
1c309a0318 cmd/tools/segment-verify: check for unvetted nodes
this also renames the command from `duplicates` to `node-check`

Change-Id: Idd303b17ec03f5b55fbbb1f4039a7761da37abe6
2022-12-19 09:59:13 +00:00
Michal Niewrzal
5110803102 storagenode/piecestore: add Exists endpoint
Adds new method Exists which can be used to verify which
requested piece ids exists on storage node. Will verify only pieces
which belongs to the satellite that used that endpoint.

Minum WASM size was increased a bit.

https://github.com/storj/storj/issues/5415

Change-Id: Ia5f9cadeb526541b2776a8973eb7d50133ad8636
2022-12-17 04:08:26 +00:00
Lizzy Thomson
678bb12d4b satellite/admin: create an endpoint for isAccountFrozen
Endpoint checks if an account is frozen or unfrozen.

relates to https://github.com/storj/storj/issues/5398

Change-Id: I8ff44063870327e05cf729eaaaed1da6c5fa9217
2022-12-16 22:53:44 +00:00
Jeremy Wharton
471f9e4e10 satellite/{console,satellitedb}: add account freeze service
This change adds an account freeze service with methods for checking
if a user is frozen, freezing a user, and unfreezing a user.
Furthermore, methods for altering the usage limits of a user or project
have been implemented for use by the account freeze service.

Change-Id: I77fecfac5c152f134bec90165acfe4f1dea957e7
2022-12-16 22:24:58 +00:00
paul cannon
7b851b42f7 satellite/audit: split out auditor process
This change creates a new independent process, the 'auditor', comparable
to the repairer, gc, and api processes. This will allow auditors to be
scaled independently of the core.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: I8a29eeb0a6e35753dfa0eab5c1246048065d1e91
2022-12-16 12:44:32 -06:00
paul cannon
fc905a15f7 satellite/audit: newContainment->containment
Now that all the reverification changes have been made and the old code
is out of the way, this commit renames the new things back to the old
names. Mostly, this involves renaming "newContainment" to "containment"
or "NewContainment" to "Containment", but there are a few other renames
that have been promised and are carried out here.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I34e2b857ea338acbb8421cdac18b17f2974f233c
2022-12-16 17:59:52 +00:00
Andrew Harding
99206fca04 satellite/payments/stripecoinpayments: skip line item generation for records with no usage
This change updates the stripecoinpayments service to optionally skip
generating line items for payments records that have no egress, storage,
or segments for the billing period.

This results in a reduction from 4 to 1 Stripe API calls for customers
who have no usage. The final API call is the attempt to generate an
invoice on stripe, which expectedly fails because there are no unapplied line
items. Removing that final API call would require some additional
queries and is out of scope for this change.

This functionality is behind the
`payments.stripe-coin-payments.skip-empty-invoices` feature flag.

https://github.com/storj/storj/issues/5381

Change-Id: Id184969a4c79047c40502336d69c51388ab03bf8
2022-12-16 17:02:50 +00:00
Egon Elbre
ee71fbb41d storagenode/piecestore: start restore trash in the background
Starting restore trash in the background allows the satellite to
continue to the next storagenode without needing to wait until
completion.

Of course, this means the satellite doesn't get feedback whether it
succeeds successfully or not. This means that the restore-trash needs to
be executed several times.

Change-Id: I62d43f6f2e4a07854f6d083a65badf897338083b
2022-12-16 18:15:52 +02:00
Andrew Harding
73d5c6944a satellite/audit: merge support for reservoirs
Change-Id: Ibbedd2a0043412210159fa2523f9e63d987276c3
2022-12-16 15:27:55 +00:00
paul cannon
0342ca1aa6 satellite/audit: delete now-unused code
Now that we are doing scalable piecewise reverifications, the code for
handling the old way of doing things (containment, pending audits,
reporting, testing) can now be removed.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ief1a75f423eff682e8f3d57804e343b3409a6631
2022-12-16 14:53:39 +00:00
paul cannon
a66503b444 satellite/audit: Begin using piecewise reverifications
This commit pulls the big switch! We have been setting up piecewise
reverifications (the workers for which can be scaled independently of
the core) for several commits now, and this commit actually begins
making use of them.

The core of this commit is fairly small, but it requires changing the
semantics in all the tests that relate to reverifications, so it ends up
being a large change. The changes to the tests are mostly mechanical and
repetitive, though, so reviewers needn't worry much.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ibb421cc021664fd6e0096ffdf5b402a69b2d6f18
2022-12-16 14:21:13 +00:00
nadimhq
9190a549ad
docs/testplan: Testplan for Automatic Account Freeze-Unfreeze (#5336)
* docs/testplan: Testplan for Automatic Account Freeze-Unfreeze

This testplan is going to cover the new account Freeze/Unfreeze. It will go over the automatic-account-freeze design doc found under docs/blueprints.

* Update automatic-account-freeze-unfreeze.md

Co-authored-by: Antonio Franco (He/Him) <antonio@storj.io>
2022-12-16 15:10:57 +01:00
Egon Elbre
04f16f8768 cmd/tools/segment-verify: tool for checking duplicate net
Change-Id: Ie47c1282e580ffc418bf3b1f3c8820a48973aefc
2022-12-15 22:58:36 +00:00
Vitalii
92a757cf3f web/satellite: clear vuex state on session timeout
We have to clear vuex app state on session timeout because user's session gets invalidated and user is redirected to login screen.

Issue:
https://github.com/storj/storj/issues/5370

Change-Id: Id654056331c81fac0b46ed90eccea0a044e4e1c9
2022-12-15 21:57:32 +00:00