This change is similar to
https://review.dev.storj.io/c/storj/storj/+/7687 but applied when
uploading from stdin with parallelism > 1.
Currently, the paralellism from stdin scales up to 3 or 4, but not
greater than that. If we buffer the content from stdin more aggressively
the parallelism scales to higher levels and reaches the performance of
reading directly from a file.
Change-Id: I1f447686a88074882709992ee6d52dd262e220fb
When Check == 0 (check all pieces), there is nearly always a piece left
in the retry count, so most segments get logged in segments-retry.csv.
This change makes it so we require retry>5 before adding to
segments-retry.csv (only in the check==0 case).
Change-Id: Iaea523c27eb777e3c248c27c7ef5effe77ae54cf
This new advanced flag configures libuplink to store in-memory the
erasure-coded pieces that are temporarily created during upload.
By default, libuplink writes the erasure-coded pieces as temp files on
the disk, but this results in additional IOPS that affect the
performance in hot-rodded scenarios.
If the erasure-coded pieces are kept in-memory and the system has enough
RAM, the upload speed may be boosted with 20-30%.
The flag is added as "advanced" as we don't recommend it by default.
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
Change-Id: Icc54f03b6c0bc27c97126f6f1d22748d21a15959
add triggerAttemptPaymentIfFrozen to check if the account is frozen
and if frozen, will trigger an attempt to pay outstanding invoices
Issue: https://github.com/storj/storj/issues/5398
Change-Id: I0da6a982e2da4204dee219d98ce2d503cbbb6f8e
The tests are forked from the chore tests with slight adaptations for
being run against the ranged loop. I also moved a benchmark for the
database from chore_test.go to db_test.go.
The pathcollector is reused as a rangedloop.Partial.
https://github.com/storj/storj/issues/5234
Change-Id: I56182031d133812a9f4d4a433c01b9150af39f31
* better error handling when Exists method is not avaialble on SN
* more optimal processing of response from Exists method
Change-Id: I6d61c09473e9f5ab76a4601720e8bd520767f4c2
Track duration of all segment loop observers. Factor out functions to
reduce size.
Still need to send the measurements out via monkit.
Part of https://github.com/storj/storj/issues/5223
Change-Id: Iae0260e250f8ea33affed95c6592a1f42df384eb
added api call, user isFrozen status;
new banner added;
DashboardArea migrated to use composition api;
Change-Id: Ia3fe68c2239064b2b729c6de14d7fc1dc8f4cf3b
This modifies the userinfo endpoint to return appropriate errors;
PermissionDenied for untrusted peers and Unimplemented because
the endpoint isn't implemented
Change-Id: I5109bb204b5e1ce2e21fe16b003991b6c900a8ce
Implemented interception for http requests.
We redirect user to login page on every 401 response.
Issue:
https://github.com/storj/storj/issues/5339
Change-Id: Icba4fc0031cb2b4e682a1be078cdcf95b7fa6bfe
This change stubs userinfo endpoint from storj/common/pb/userinfo.proto.
It also adds config for allowed peers, and a method for verifying peers.
Issue: https://github.com/storj/storj/issues/5358
Change-Id: I057a0e873a9e9b3b9ad0bba69305f0d708bd9b9e
Fixed bucket search not being cleared on component unmount.
Fixed open bucket issue for project dashboard presented by the conflict with vue-click-outside directive.
Change-Id: I3682c6b131dcdb902369cf74b31c1b270cd15b08
* normalize ExistsCheckWorkers flag to avoid setting incorrect values
* wait for limiter to finish if context was canceled
Change-Id: I688a395bd958cd09233605fc264d43f91ec45ed1
Adds new method Exists which can be used to verify which
requested piece ids exists on storage node. Will verify only pieces
which belongs to the satellite that used that endpoint.
Minum WASM size was increased a bit.
https://github.com/storj/storj/issues/5415
Change-Id: Ia5f9cadeb526541b2776a8973eb7d50133ad8636
This change adds an account freeze service with methods for checking
if a user is frozen, freezing a user, and unfreezing a user.
Furthermore, methods for altering the usage limits of a user or project
have been implemented for use by the account freeze service.
Change-Id: I77fecfac5c152f134bec90165acfe4f1dea957e7
This change creates a new independent process, the 'auditor', comparable
to the repairer, gc, and api processes. This will allow auditors to be
scaled independently of the core.
Refs: https://github.com/storj/storj/issues/5251
Change-Id: I8a29eeb0a6e35753dfa0eab5c1246048065d1e91
Now that all the reverification changes have been made and the old code
is out of the way, this commit renames the new things back to the old
names. Mostly, this involves renaming "newContainment" to "containment"
or "NewContainment" to "Containment", but there are a few other renames
that have been promised and are carried out here.
Refs: https://github.com/storj/storj/issues/5230
Change-Id: I34e2b857ea338acbb8421cdac18b17f2974f233c
This change updates the stripecoinpayments service to optionally skip
generating line items for payments records that have no egress, storage,
or segments for the billing period.
This results in a reduction from 4 to 1 Stripe API calls for customers
who have no usage. The final API call is the attempt to generate an
invoice on stripe, which expectedly fails because there are no unapplied line
items. Removing that final API call would require some additional
queries and is out of scope for this change.
This functionality is behind the
`payments.stripe-coin-payments.skip-empty-invoices` feature flag.
https://github.com/storj/storj/issues/5381
Change-Id: Id184969a4c79047c40502336d69c51388ab03bf8
Starting restore trash in the background allows the satellite to
continue to the next storagenode without needing to wait until
completion.
Of course, this means the satellite doesn't get feedback whether it
succeeds successfully or not. This means that the restore-trash needs to
be executed several times.
Change-Id: I62d43f6f2e4a07854f6d083a65badf897338083b
Now that we are doing scalable piecewise reverifications, the code for
handling the old way of doing things (containment, pending audits,
reporting, testing) can now be removed.
Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ief1a75f423eff682e8f3d57804e343b3409a6631
This commit pulls the big switch! We have been setting up piecewise
reverifications (the workers for which can be scaled independently of
the core) for several commits now, and this commit actually begins
making use of them.
The core of this commit is fairly small, but it requires changing the
semantics in all the tests that relate to reverifications, so it ends up
being a large change. The changes to the tests are mostly mechanical and
repetitive, though, so reviewers needn't worry much.
Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ibb421cc021664fd6e0096ffdf5b402a69b2d6f18
* docs/testplan: Testplan for Automatic Account Freeze-Unfreeze
This testplan is going to cover the new account Freeze/Unfreeze. It will go over the automatic-account-freeze design doc found under docs/blueprints.
* Update automatic-account-freeze-unfreeze.md
Co-authored-by: Antonio Franco (He/Him) <antonio@storj.io>
We have to clear vuex app state on session timeout because user's session gets invalidated and user is redirected to login screen.
Issue:
https://github.com/storj/storj/issues/5370
Change-Id: Id654056331c81fac0b46ed90eccea0a044e4e1c9
This change implements DB methods for interacting with the
account_freeze_event table and introduces structures related to
account freeze events.
Change-Id: Ib125b31dfb754b2428212c39b780e14cfc7f97bf
Previously because of the use of a LAG to calculate the hour_interval
the first record, which is usually the first day of the month usually,
doesn’t have a previous record and always assumes the at_rest_total is
for 24 hours.
Resolves https://github.com/storj/storj/issues/5390
Change-Id: Id532f8b38fe9df61432e62655318ff119a733d13
This change fixes the access of unset segments and keys on the reservoir
when the reservoir size is less than the max OR the number of sampled
segments is smaller than the reservoir size. It does so by tucking away
the segments and keys behind methods that return properly sized slices
into the segments/keys arrays.
It also fixes a bug in the housekeeping for the internal index variable
that holds onto how many items in the array have been populated. As part
of this fix, it changes the type of index to int8, which reduces the
size of the reservoir struct by 8 bytes.
The tests have been updated to provide better coverage for this case.
Change-Id: I3ceb17b692fe456fc4c1ca5d67d35c96aeb0a169
problem: after clicking on any bucket serch input was prefilled weird by browser and causes table empty state to appear;
solution: added workaround to add readonly status for input and disable it on focus;
Change-Id: I3dd05e709ef3aceb3b8542fe468bdb41b7b6d353
This adds the capability to the segment-verify tool of checking all
pieces of every indicated segment.
Pieces which could not be accessed (i.e. we couldn't get a single
byte from them) are recorded in a csv file.
I haven't been able to test this in any very meaningful way, yet, but I
am comforted by the fact that the worst things it could possibly do are
(a) download pieces too many times, and (b) miss downloading some
pieces.
Change-Id: I3aba30921572c974993363eb36d0fd5b3ae97907
