Now that we are doing scalable piecewise reverifications, the code for
handling the old way of doing things (containment, pending audits,
reporting, testing) can now be removed.
Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ief1a75f423eff682e8f3d57804e343b3409a6631
This commit pulls the big switch! We have been setting up piecewise
reverifications (the workers for which can be scaled independently of
the core) for several commits now, and this commit actually begins
making use of them.
The core of this commit is fairly small, but it requires changing the
semantics in all the tests that relate to reverifications, so it ends up
being a large change. The changes to the tests are mostly mechanical and
repetitive, though, so reviewers needn't worry much.
Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ibb421cc021664fd6e0096ffdf5b402a69b2d6f18
* docs/testplan: Testplan for Automatic Account Freeze-Unfreeze
This testplan is going to cover the new account Freeze/Unfreeze. It will go over the automatic-account-freeze design doc found under docs/blueprints.
* Update automatic-account-freeze-unfreeze.md
Co-authored-by: Antonio Franco (He/Him) <antonio@storj.io>
We have to clear vuex app state on session timeout because user's session gets invalidated and user is redirected to login screen.
Issue:
https://github.com/storj/storj/issues/5370
Change-Id: Id654056331c81fac0b46ed90eccea0a044e4e1c9
This change implements DB methods for interacting with the
account_freeze_event table and introduces structures related to
account freeze events.
Change-Id: Ib125b31dfb754b2428212c39b780e14cfc7f97bf
Previously because of the use of a LAG to calculate the hour_interval
the first record, which is usually the first day of the month usually,
doesn’t have a previous record and always assumes the at_rest_total is
for 24 hours.
Resolves https://github.com/storj/storj/issues/5390
Change-Id: Id532f8b38fe9df61432e62655318ff119a733d13
This change fixes the access of unset segments and keys on the reservoir
when the reservoir size is less than the max OR the number of sampled
segments is smaller than the reservoir size. It does so by tucking away
the segments and keys behind methods that return properly sized slices
into the segments/keys arrays.
It also fixes a bug in the housekeeping for the internal index variable
that holds onto how many items in the array have been populated. As part
of this fix, it changes the type of index to int8, which reduces the
size of the reservoir struct by 8 bytes.
The tests have been updated to provide better coverage for this case.
Change-Id: I3ceb17b692fe456fc4c1ca5d67d35c96aeb0a169
problem: after clicking on any bucket serch input was prefilled weird by browser and causes table empty state to appear;
solution: added workaround to add readonly status for input and disable it on focus;
Change-Id: I3dd05e709ef3aceb3b8542fe468bdb41b7b6d353
This adds the capability to the segment-verify tool of checking all
pieces of every indicated segment.
Pieces which could not be accessed (i.e. we couldn't get a single
byte from them) are recorded in a csv file.
I haven't been able to test this in any very meaningful way, yet, but I
am comforted by the fact that the worst things it could possibly do are
(a) download pieces too many times, and (b) miss downloading some
pieces.
Change-Id: I3aba30921572c974993363eb36d0fd5b3ae97907
composables folder created (same as react hooks) for functionality reusage;
related test moved to ignored folder;
Change-Id: I494aa27079aa5694bee7b18511eeadd56ced59e9
Add document outlining the design and implementation steps for the paid
tier TLS feature.
https: //github.com/storj/storj/issues/5296
Change-Id: I51f68fc7890f816cef7bf2a319762ad701bac445
Adding this entry means that the database accessed as "reverifyqueue"
(`(*satelliteDBCollection).ReverifyQueue()`) can be located on a
different database host from the other databases, and things should
still work. There aren't any queries that do a JOIN on tables from
reverifyQueue and other things in satellitedb, for example.
This should really have been put here earlier, when reverifyqueue was
first added, but it's ok. This won't have any bearing on things until we
need to deploy to prod.
Refs: https://github.com/storj/storj/issues/5230
Change-Id: I76f68de79cd645c869f3dbfbe3b2c9c4f9359e8f
This method on the Verifier allows the caller to find, out of the nodes
holding pieces in a given segment, which ones are contained.
This method is not yet being used. It will be in a future commit.
Refs: https://github.com/storj/storj/issues/5230
Change-Id: I242cd999913ca4dabbe8a62767ed4869b31fca04
Provides the `segment-verify run buckets` command for verifying segments within a list of buckets.
Bucket list is a csv file with `project_id,bucket_name` to be checked.
https://github.com/storj/storj-private/issues/101
Change-Id: I3d25c27b56fcab4a6a1aebb6f87514d6c97de3ff
Implemented UI error tracking.
We use satellite analytics service to track the fact that UI error occurred and send minimal info to Segment (not Hubspot).
We send only the fact that UI error occurred and the place where this error occurred.
Extended notificator plugin error function to include the place where error occurred.
I made the place argument nullable to be always explicitly provided (build fails if place is not provided).
If place is not null then error event is triggered in the background.
Issue:
https://github.com/storj/storj-private/issues/107
Change-Id: I7d129fb29629979f5be6ff5dea37ad19b1a2397e
update the updateProject function to set user specified bandwidth and storage limits
fixes https://github.com/storj/storj/issues/5185
Change-Id: Ib4132487f6b7ea0afa7c57acfc358857b3e852d1
We missed proper handling of object copies for method
GetStreamPieceCountByNodeID which is used by metabase.GetObjectIPs.
That caused some lack of IPs returned when queriyng IPs of copy and
broke things like pices map on linksharing.
Fixes https://github.com/storj/storj/issues/5406
Change-Id: I9574776f34880788c2dc9ff78a6ae20d44fe628f
fixed issue with required props that were not passed to component;
added default props;
related test added to ignored folder till vue 3 version update with vue-cli-service;
Change-Id: Idc95a4c0b9f124797519652004abf53e066605c2
This change removes the go mod download process for the UI tests, removes the commented out code block that once called the UI tests, and the duplicate web/wasm build step. The Build step for the web/wasm is now executed via a shell script and called much earlier in the tests.
Here we add a worker class comparable to audit.Worker, which will be
responsible for pulling items off of the reverification queue and
calling reverifier.ReverifyPiece on them.
Note that piecewise reverification audits (which this will control) are
not yet being done. That is, nothing is being added to the
reverification queue at this point.
Refs: https://github.com/storj/storj/issues/5251
Change-Id: I94e28830e27caa49f2c8bd4a2336533e187ab69c
The Reporter is responsible for processing results from auditing
operations, logging the results, disqualifying nodes that reached
the maximum reverification count, and passing the results on to
the reputation system.
In this commit, we extend the Reporter so that it knows how to process
the results of piecewise reverification audits.
We also change most reporter-related tests so that reverifications
happen as piecewise reverification audits, exercising the new code.
Note that piecewise reverification audits are not yet being done outside
of tests. In a later commit, we will switch from doing segmentwise
reverifications to piecewise reverifications, as part of the
audit-scaling effort.
Refs: https://github.com/storj/storj/issues/5230
Change-Id: I9438164ce1ea4d9a1790d18d0e1046a8eb04d8e9
While researching logs from a large set of audits, I noticed that nearly
all of them had streamIDs starting with 0 or 1. This seemed very odd,
because streamIDs are supposed to be pretty much entirely random, and
every hex digit from 0-f should have been represented with roughly equal
frequency.
It turned out that our A-Chao implementation of reservoir sampling is
flawed. As far as we can tell, so is the Wikipedia implementation. No
one has yet reviewed the original 1982 paper by Dr. Chao in enough
detail to know where the error originated, but we do know that we have
been auditing segments near the beginning of the segment loop (low
streamIDs) far more often than segments near the end of the segment loop
(high streamIDs).
This change uses an algorithm Wikipedia calls "A-Res" instead, and adds
a test to check for that sort of bias creeping back in somehow. A-Res
will be slightly slower than A-Chao, because of a few extra steps that
need to be done, but it does appear to be selecting items uniformly.
Change-Id: I45eba4c522bafc729cebe2aab6f3fe65cd6336be
Some observers assume that they will observe all the segments for a
given stream, and that they will observe those segments in a sequential
stream over one or more iterations.
This change updates the range provider from rangedlooptest to provide
these guarantees.
The change also removes the Mock suffix from the provider/splitter types
since the package name (rangedlooptest) implies that the type is a test
double.
Change-Id: I927c409807e305787abcde57427baac22f663eaa
- account types can be selected
- checkboxes can be interacted with
see: https://github.com/storj/storj/issues/5163
Change-Id: Id97c026b79fb2993bba9138e9d4754e7a1371bbc
We have a bug in our behavior while doing API pods deployment. At this
time its possible to have pods with multiple versions flag set true only
partially for some of pods. Because of that it's possible to start new
object without removing existing/older version on BeginObject
(new behavior) and also don't remove that existing/older object on
CommitObject. That can cause to have two committed objects with
different versions and that's a state we want to avoid.
To fix it we are removing multiple versions flag from CommitObject to
always try delete existing objects. This way even if we don't remove
existing object on BeginObject it will be always removed while
committing.
Fixes https://github.com/storj/storj/issues/5373
Change-Id: Idc334bf5cc785d2f559af96e92c3de6d82ca58ba
The query changes we did while fixing the usage graph led to wrong
payout calculations directly linked to disk space.
This change:
- avoids converting from Bh to B directly in the query
- returns the at_rest_total in the original bytes*hour value
- returns at_rest_total_bytes as the calculated disk spaced used in bytes
- uses the at_rest_total_bytes only for the disk space graph
- return summary_bytes as the average disk space used within the specified date
- updates the disk space graph header to "average disk space used this month"
The total disk used in the month is also displayed in B not B*day
Resolves https://github.com/storj/storj/issues/5355
Change-Id: I2cfefb0fe711f9c59de2adb547c4ab50b05c7cbb
Add an abstraction rangedloop.SegmentProvider to fetch chunks of
segments from the metainfo database in parallel.
Part of https://github.com/storj/storj/issues/5223
Change-Id: Ife26467ea0c3be550bde0b05464ef1db62dd4d2a
Adds DeleteAllSessionsByUserIDExcept which removes all sessions except the specified session from the database and applies this function to enableMFA and disableMFA
addresses https://github.com/storj/storj-private/issues/15
Change-Id: I5d8c620dadbbda4a1b430ccf8a6121e167dd0761
Because --readonly is default true, passing something like
--disallow-deletes=false would not actually update that
value because the readonly flag would override. this makes it
so that the --disallow-* flags override the --readonly and
--writeonly flags.
Also fixes some minor formatting issues with share like an
extra space after the "Public Access:" entry.
Simplifies the handling of the explicit "none" by making the
flags for the dates optional and using nil to signify that
the value was left unset.
Bump the go.mod to go1.18 to enable the use of generics and
add a small generic function. This can easily be backed out
if it causes problems.
Change-Id: I1c5f1321ad17b8ace778ce55561cbbfc24321a68
This change continues the update of the file browser to use the common table component.
It replaces the use of <tr>s with <table-item>.
It also removes bootstrap from the file browser and fixes resulting UI "bugs"
Issue: https://github.com/storj/storj/issues/5090
Change-Id: Idfe415e9338243ad4fcedc8a74621c9e13e1da29
Minimal implementation of the ranged (=threaded) segment loop
service, to improve performance over the existing loop.
Has tests with a an inmemory segment database
and example observer.
Does not have yet: database link, observer duration tracking,
suspicious processed ratio guard, rate limiting, minimum execution
interval per observer, etc.
Part of https://github.com/storj/storj/issues/5223
Change-Id: I08ffb392c3539e380f4e7b4f1afd56c4c394668d
This change shows STORJ token balance on the billing overview page instead of the Stripe balance it shows currently.
It changes the text on the "Available balance" card to reflect the new balance being displayed. Finally, it adds shortcuts to navigate straight to token history or add tokens modal when call to action on "Balance card"
Issue: https://github.com/storj/storj/issues/5204
Change-Id: Ic88e43c602e4949b6c6be4c7644c04f3c7d38585
