Commit Graph

7277 Commits

Author SHA1 Message Date
paul cannon
fc905a15f7 satellite/audit: newContainment->containment
Now that all the reverification changes have been made and the old code
is out of the way, this commit renames the new things back to the old
names. Mostly, this involves renaming "newContainment" to "containment"
or "NewContainment" to "Containment", but there are a few other renames
that have been promised and are carried out here.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I34e2b857ea338acbb8421cdac18b17f2974f233c
2022-12-16 17:59:52 +00:00
Andrew Harding
99206fca04 satellite/payments/stripecoinpayments: skip line item generation for records with no usage
This change updates the stripecoinpayments service to optionally skip
generating line items for payments records that have no egress, storage,
or segments for the billing period.

This results in a reduction from 4 to 1 Stripe API calls for customers
who have no usage. The final API call is the attempt to generate an
invoice on stripe, which expectedly fails because there are no unapplied line
items. Removing that final API call would require some additional
queries and is out of scope for this change.

This functionality is behind the
`payments.stripe-coin-payments.skip-empty-invoices` feature flag.

https://github.com/storj/storj/issues/5381

Change-Id: Id184969a4c79047c40502336d69c51388ab03bf8
2022-12-16 17:02:50 +00:00
Egon Elbre
ee71fbb41d storagenode/piecestore: start restore trash in the background
Starting restore trash in the background allows the satellite to
continue to the next storagenode without needing to wait until
completion.

Of course, this means the satellite doesn't get feedback whether it
succeeds successfully or not. This means that the restore-trash needs to
be executed several times.

Change-Id: I62d43f6f2e4a07854f6d083a65badf897338083b
2022-12-16 18:15:52 +02:00
Andrew Harding
73d5c6944a satellite/audit: merge support for reservoirs
Change-Id: Ibbedd2a0043412210159fa2523f9e63d987276c3
2022-12-16 15:27:55 +00:00
paul cannon
0342ca1aa6 satellite/audit: delete now-unused code
Now that we are doing scalable piecewise reverifications, the code for
handling the old way of doing things (containment, pending audits,
reporting, testing) can now be removed.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ief1a75f423eff682e8f3d57804e343b3409a6631
2022-12-16 14:53:39 +00:00
paul cannon
a66503b444 satellite/audit: Begin using piecewise reverifications
This commit pulls the big switch! We have been setting up piecewise
reverifications (the workers for which can be scaled independently of
the core) for several commits now, and this commit actually begins
making use of them.

The core of this commit is fairly small, but it requires changing the
semantics in all the tests that relate to reverifications, so it ends up
being a large change. The changes to the tests are mostly mechanical and
repetitive, though, so reviewers needn't worry much.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ibb421cc021664fd6e0096ffdf5b402a69b2d6f18
2022-12-16 14:21:13 +00:00
nadimhq
9190a549ad
docs/testplan: Testplan for Automatic Account Freeze-Unfreeze (#5336)
* docs/testplan: Testplan for Automatic Account Freeze-Unfreeze

This testplan is going to cover the new account Freeze/Unfreeze. It will go over the automatic-account-freeze design doc found under docs/blueprints.

* Update automatic-account-freeze-unfreeze.md

Co-authored-by: Antonio Franco (He/Him) <antonio@storj.io>
2022-12-16 15:10:57 +01:00
Egon Elbre
04f16f8768 cmd/tools/segment-verify: tool for checking duplicate net
Change-Id: Ie47c1282e580ffc418bf3b1f3c8820a48973aefc
2022-12-15 22:58:36 +00:00
Vitalii
92a757cf3f web/satellite: clear vuex state on session timeout
We have to clear vuex app state on session timeout because user's session gets invalidated and user is redirected to login screen.

Issue:
https://github.com/storj/storj/issues/5370

Change-Id: Id654056331c81fac0b46ed90eccea0a044e4e1c9
2022-12-15 21:57:32 +00:00
NickolaiYurchenko
bb170a9d57 web/satellite: VCheckbox, VButton migrated to use composition api
related tests moved to ignored folder or fixed and will be changed after;

Change-Id: Idb4b1a93ac176b98184c746222b5cd1e30dad595
2022-12-15 18:10:08 +00:00
NickolaiYurchenko
8d30b58a8c web/satellite: VInfo, VerticalArrows, VDateRangePicket migrated to use composition api
related tests moved to ignored folder;

Change-Id: Ic76fe5f6da8088484ddcd2956bb6bb6c0d5d6223
2022-12-15 17:30:46 +00:00
NickolaiYurchenko
55774bac10 web/satellite: VLoader, VModal, VList migrated to use composition api
VLIst component and tests deleted;
related tests moved to ignore folder;

Change-Id: Id65754a044cdc8adcd551a6f4e3c5df65b11ffa0
2022-12-15 16:59:59 +00:00
NickolaiYurchenko
e54d9b1c6a web/satellite: VAlidationMessage, TableItem and TablePagination migrated to use composition api
Change-Id: Id02dd4f32fdd9924a2528a7363682fb6e1cb7afa
2022-12-15 16:02:13 +00:00
Jeremy Wharton
d069045058 satellite/{console,satellitedb}: implement account freeze event DB
This change implements DB methods for interacting with the
account_freeze_event table and introduces structures related to
account freeze events.

Change-Id: Ib125b31dfb754b2428212c39b780e14cfc7f97bf
2022-12-15 14:16:24 +00:00
Clement Sam
951d5db7f7 storagenode: fix hour_interval for first day defaulted to 24h
Previously because of the use of a LAG to calculate the hour_interval
the first record, which is usually the first day of the month usually,
doesn’t have a previous record and always assumes the at_rest_total is
for 24 hours.

Resolves https://github.com/storj/storj/issues/5390

Change-Id: Id532f8b38fe9df61432e62655318ff119a733d13
2022-12-15 13:30:11 +00:00
Andrew Harding
93fad70e4b satellite/audit: prevent accessing unset reservoir segments
This change fixes the access of unset segments and keys on the reservoir
when the reservoir size is less than the max OR the number of sampled
segments is smaller than the reservoir size. It does so by tucking away
the segments and keys behind methods that return properly sized slices
into the segments/keys arrays.

It also fixes a bug in the housekeeping for the internal index variable
that holds onto how many items in the array have been populated. As part
of this fix, it changes the type of index to int8, which reduces the
size of the reservoir struct by 8 bytes.

The tests have been updated to provide better coverage for this case.

Change-Id: I3ceb17b692fe456fc4c1ca5d67d35c96aeb0a169
2022-12-14 17:43:17 -07:00
Fadila Khadar
5c3a148d6e satellite/overlaycache: fix typo in UpdateCheckIn request
- fix a malformed SQL query
- add test to be sure we don't have this problem again.

Change-Id: I3fde8c59ba01335411e51d964bec95bc26cfc961
2022-12-14 22:21:45 +01:00
NickolaiYurchenko
4fad04e153 web/satellite: autocomplete search bug fixed
problem: after clicking on any bucket serch input was prefilled weird by browser and causes table empty state to appear;
solution: added workaround to add readonly status for input and disable it on focus;

Change-Id: I3dd05e709ef3aceb3b8542fe468bdb41b7b6d353
2022-12-14 19:39:27 +00:00
paul cannon
727136141a satellite/cmd/tools/segment-verify: check all pieces
This adds the capability to the segment-verify tool of checking all
pieces of every indicated segment.

Pieces which could not be accessed (i.e. we couldn't get a single
byte from them) are recorded in a csv file.

I haven't been able to test this in any very meaningful way, yet, but I
am comforted by the fact that the worst things it could possibly do are
(a) download pieces too many times, and (b) miss downloading some
pieces.

Change-Id: I3aba30921572c974993363eb36d0fd5b3ae97907
2022-12-14 19:06:08 +00:00
Qweder93
9d7e94b57b private/testplanet: extended satellite with rangedloop peer
Change-Id: I3c82f5c970c4fef139fc089937e43c5cb1769815
2022-12-14 16:21:50 +02:00
NickolaiYurchenko
235e9be208 web/satellite: VBar, VBanner migrated to use composition api
composables folder created (same as react hooks) for functionality reusage;
related test moved to ignored folder;

Change-Id: I494aa27079aa5694bee7b18511eeadd56ced59e9
2022-12-14 11:21:11 +00:00
Moby von Briesen
870eefbb5d docs/blueprints: Add paid tier tls document
Add document outlining the design and implementation steps for the paid
tier TLS feature.

https: //github.com/storj/storj/issues/5296
Change-Id: I51f68fc7890f816cef7bf2a319762ad701bac445
2022-12-13 23:08:23 +00:00
Lizzy Thomson
eee79ec3cd web/satellite: fix project selection logic
fixes https://github.com/storj/storj/issues/5294

Change-Id: I4cc3110e94700b0f31d20edabfd31482ae891200
2022-12-13 22:08:12 +00:00
paul cannon
ad82e74cb7 satellite/satellitedb: add reverifyqueue to safelyPartitionableDBs
Adding this entry means that the database accessed as "reverifyqueue"
(`(*satelliteDBCollection).ReverifyQueue()`) can be located on a
different database host from the other databases, and things should
still work. There aren't any queries that do a JOIN on tables from
reverifyQueue and other things in satellitedb, for example.

This should really have been put here earlier, when reverifyqueue was
first added, but it's ok. This won't have any bearing on things until we
need to deploy to prod.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I76f68de79cd645c869f3dbfbe3b2c9c4f9359e8f
2022-12-13 21:14:17 +00:00
paul cannon
47b9134f76 satellite/audit: add IdentifyContainedNodes
This method on the Verifier allows the caller to find, out of the nodes
holding pieces in a given segment, which ones are contained.

This method is not yet being used. It will be in a future commit.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I242cd999913ca4dabbe8a62767ed4869b31fca04
2022-12-13 20:46:43 +00:00
Fadila Khadar
995f78d579 satellite/cmd: segment-verify verifies segments in given bucket list
Provides the `segment-verify run buckets` command for verifying segments within a list of buckets.

Bucket list is a csv file with `project_id,bucket_name` to be checked.

https://github.com/storj/storj-private/issues/101

Change-Id: I3d25c27b56fcab4a6a1aebb6f87514d6c97de3ff
2022-12-13 20:10:00 +00:00
Cameron
021c98c17a satellite/satellitedb: create table account_freeze_events
create table for account freeze events

github issue: https://github.com/storj/storj/issues/5396

Change-Id: Ia4f93ce5caacd02b161853e3261d237bc715f23d
2022-12-13 19:09:25 +00:00
Vitalii
5d727becb4 satellite/{analytics, web}: segment ui error tracking implemented
Implemented UI error tracking.
We use satellite analytics service to track the fact that UI error occurred and send minimal info to Segment (not Hubspot).
We send only the fact that UI error occurred and the place where this error occurred.
Extended notificator plugin error function to include the place where error occurred.
I made the place argument nullable to be always explicitly provided (build fails if place is not provided).
If place is not null then error event is triggered in the background.

Issue:
https://github.com/storj/storj-private/issues/107

Change-Id: I7d129fb29629979f5be6ff5dea37ad19b1a2397e
2022-12-13 18:11:53 +00:00
Andrew Harding
7f2dfaa394 gitignore: add go workspace files
Change-Id: Icd382bb8da16c6100ef68340d665e2a29420b6a2
2022-12-13 10:15:53 -07:00
Lizzy Thomson
0afd3938c7 sat/console update the updateProject to set user specified limits
update the updateProject function to set user specified bandwidth and storage limits

fixes https://github.com/storj/storj/issues/5185

Change-Id: Ib4132487f6b7ea0afa7c57acfc358857b3e852d1
2022-12-13 16:44:41 +00:00
Vitalii
17db59e27a web/satellite: update apollo-client dependency to remove CSP error
This is insane but the problem was caused by the old apollo-client dependency which is 3 years old.
The error was coming from ts-invariant code here
https://github.com/apollographql/invariant-packages/blob/master/packages/ts-invariant/src/invariant.ts#L64

By this change I updated our app to use new @apollo/client dependency but there is one pretty big concern.
This lib has react as a peer dependency (what?!) so to avoid adding react to our vue project I had to import needed functionality from @apollo/client/core instead of just @apollo/client.

There is a discussion here https://github.com/apollographql/apollo-client/issues/7318
According to this ^ apollo team are going to remove react from their peer dependecies list when apollo V4 will be launched.
See https://github.com/apollographql/apollo-client/issues/8190

That's another point why we should move away from using graphql entirely or at least stop using apollo.

Issue:
https://github.com/storj/storj/issues/5371

Change-Id: Ifd484efbcd9e99d8c2e21d0dab93670c9358e25c
2022-12-13 16:04:50 +00:00
Michal Niewrzal
0bbbb9c4c1 satellite/metabase: fix log for multiple committed version
Change-Id: I2556c5b523091c11937a01efff07be9e0dd964aa
2022-12-13 13:08:02 +00:00
Márton Elek
ef2c0f4225 mod: use v1.10.0 tagged commit from storj.io/uplink
Change-Id: I475f2cf6447453dd78eb3549fdaacd286e1da494
2022-12-13 12:04:29 +00:00
Michal Niewrzal
0759cbdc7f satellite/metabase: handle copies with GetStreamPieceCountByNodeID
We missed proper handling of object copies for method
GetStreamPieceCountByNodeID which is used by metabase.GetObjectIPs.
That caused some lack of IPs returned when queriyng IPs of copy and
broke things like pices map on linksharing.

Fixes https://github.com/storj/storj/issues/5406

Change-Id: I9574776f34880788c2dc9ff78a6ae20d44fe628f
2022-12-13 12:32:56 +01:00
Andrew Harding
1cb2eb4c3b satellite/rangedloop: wire up metrics observer
Final touches on https://github.com/storj/storj/issues/5236.

Change-Id: I2259ec4e7825d20db9efb36beb42d6309dee55ba
2022-12-12 19:06:23 +00:00
Márton Elek
b9f0a38846 build: use latest storj-up@main in Earthfile
Change-Id: I480b1a73619fdc92d88294ab37b9a529f1ce5589
2022-12-12 15:06:01 +00:00
NickolaiYurchenko
2b78433cf0 web/satellite: VPagination, VHeader, VSearch migration to use composition api
fixed issue with required props that were not passed to component;
added default props;
related test added to ignored folder till vue 3 version update with vue-cli-service;

Change-Id: Idc95a4c0b9f124797519652004abf53e066605c2
2022-12-12 16:13:14 +02:00
Antonio Franco (He/Him)
9671851ef4
ci: removed ui artifacts (#5400)
This change removes the go mod download process for the UI tests, removes the commented out code block that once called the UI tests, and the duplicate web/wasm build step. The Build step for the web/wasm is now executed via a shell script and called much earlier in the tests.
2022-12-12 14:15:16 +01:00
paul cannon
c575a21509 satellite/audit: add audit.ReverifyWorker
Here we add a worker class comparable to audit.Worker, which will be
responsible for pulling items off of the reverification queue and
calling reverifier.ReverifyPiece on them.

Note that piecewise reverification audits (which this will control) are
not yet being done. That is, nothing is being added to the
reverification queue at this point.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: I94e28830e27caa49f2c8bd4a2336533e187ab69c
2022-12-12 11:57:08 +00:00
paul cannon
1854351da6 satellite/audit: teach Reporter about piecewise audits
The Reporter is responsible for processing results from auditing
operations, logging the results, disqualifying nodes that reached
the maximum reverification count, and passing the results on to
the reputation system.

In this commit, we extend the Reporter so that it knows how to process
the results of piecewise reverification audits.

We also change most reporter-related tests so that reverifications
happen as piecewise reverification audits, exercising the new code.

Note that piecewise reverification audits are not yet being done outside
of tests. In a later commit, we will switch from doing segmentwise
reverifications to piecewise reverifications, as part of the
audit-scaling effort.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I9438164ce1ea4d9a1790d18d0e1046a8eb04d8e9
2022-12-12 11:28:02 +00:00
paul cannon
231c783698 satellite/audit: fix reservoir sampling bias
While researching logs from a large set of audits, I noticed that nearly
all of them had streamIDs starting with 0 or 1. This seemed very odd,
because streamIDs are supposed to be pretty much entirely random, and
every hex digit from 0-f should have been represented with roughly equal
frequency.

It turned out that our A-Chao implementation of reservoir sampling is
flawed. As far as we can tell, so is the Wikipedia implementation. No
one has yet reviewed the original 1982 paper by Dr. Chao in enough
detail to know where the error originated, but we do know that we have
been auditing segments near the beginning of the segment loop (low
streamIDs) far more often than segments near the end of the segment loop
(high streamIDs).

This change uses an algorithm Wikipedia calls "A-Res" instead, and adds
a test to check for that sort of bias creeping back in somehow. A-Res
will be slightly slower than A-Chao, because of a few extra steps that
need to be done, but it does appear to be selecting items uniformly.

Change-Id: I45eba4c522bafc729cebe2aab6f3fe65cd6336be
2022-12-09 18:16:58 -06:00
Andrew Harding
b562cbf98f satellite/metrics: provide a rangedloop observer
https://github.com/storj/storj/issues/5236

Change-Id: Ic1ed7a5533dccacd58285b64579dbdd6210de4f9
2022-12-09 12:04:39 -07:00
Andrew Harding
633ab8dcf6 satellite/metadabase/rangedloop: stream affinity for test provider
Some observers assume that they will observe all the segments for a
given stream, and that they will observe those segments in a sequential
stream over one or more iterations.

This change updates the range provider from rangedlooptest to provide
these guarantees.

The change also removes the Mock suffix from the provider/splitter types
since the package name (rangedlooptest) implies that the type is a test
double.

Change-Id: I927c409807e305787abcde57427baac22f663eaa
2022-12-09 16:49:02 +00:00
Jeremy Wharton
ba7d2c2dbe satellite/payments/stripecoinpayments: add config for price overrides
This change adds configuration flags for defining partner-specific
project usage price overrides.

Resolves https://github.com/storj/storj-private/issues/61

Change-Id: Ia535ac22576382211d045f9ff2c9b983a07e86f3
2022-12-09 15:33:27 +00:00
Malcolm Bouzi
99f4b2a26e web/satellite: Ensure that all users can register through keyboard only to improve accessibility:
- account types can be selected
- checkboxes can be interacted with

see: https://github.com/storj/storj/issues/5163

Change-Id: Id97c026b79fb2993bba9138e9d4754e7a1371bbc
2022-12-09 14:47:25 +00:00
Michal Niewrzal
5c2131ed0d satellite/metabase: always try to remove old version on commit
We have a bug in our behavior while doing API pods deployment. At this
time its possible to have pods with multiple versions flag set true only
partially for some of pods. Because of that it's possible to start new
object without removing existing/older version on BeginObject
(new behavior) and also don't remove that existing/older object on
CommitObject. That can cause to have two committed objects with
different versions and that's a state we want to avoid.

To fix it we are removing multiple versions flag from CommitObject to
always try delete existing objects. This way even if we don't remove
existing object on BeginObject it will be always removed while
committing.

Fixes https://github.com/storj/storj/issues/5373

Change-Id: Idc334bf5cc785d2f559af96e92c3de6d82ca58ba
2022-12-09 13:45:03 +00:00
Clement Sam
7461ffe148 {storagenode,web/multinode}: fix storage usage db/cache retrieval queries
The query changes we did while fixing the usage graph led to wrong
payout calculations directly linked to disk space.

This change:

- avoids converting from Bh to B directly in the query
- returns the at_rest_total in the original bytes*hour value
- returns at_rest_total_bytes as the calculated disk spaced used in bytes
- uses the at_rest_total_bytes only for the disk space graph
- return summary_bytes as the average disk space used within the specified date
- updates the disk space graph header to "average disk space used this month"

The total disk used in the month is also displayed in B not B*day

Resolves https://github.com/storj/storj/issues/5355

Change-Id: I2cfefb0fe711f9c59de2adb547c4ab50b05c7cbb
2022-12-09 11:07:33 +00:00
Erik van Velzen
3cf7ebfad0
satellite/metabase/rangedloop: database abstraction (#5337)
Add an abstraction rangedloop.SegmentProvider to fetch chunks of
segments from the metainfo database in parallel.

Part of https://github.com/storj/storj/issues/5223

Change-Id: Ife26467ea0c3be550bde0b05464ef1db62dd4d2a
2022-12-09 03:01:07 +01:00
Lizzy Thomson
9fedc21fea satellite/console account session management for enabling/disabling MFA
Adds DeleteAllSessionsByUserIDExcept which removes all sessions except the specified session from the database and applies this function to enableMFA and disableMFA

addresses https://github.com/storj/storj-private/issues/15

Change-Id: I5d8c620dadbbda4a1b430ccf8a6121e167dd0761
2022-12-08 18:32:35 +00:00
Jeff Wendling
f2fdd6ca33 cmd/uplink: fix some issues with share
Because --readonly is default true, passing something like
--disallow-deletes=false would not actually update that
value because the readonly flag would override. this makes it
so that the --disallow-* flags override the --readonly and
--writeonly flags.

Also fixes some minor formatting issues with share like an
extra space after the "Public Access:" entry.

Simplifies the handling of the explicit "none" by making the
flags for the dates optional and using nil to signify that
the value was left unset.

Bump the go.mod to go1.18 to enable the use of generics and
add a small generic function. This can easily be backed out
if it causes problems.

Change-Id: I1c5f1321ad17b8ace778ce55561cbbfc24321a68
2022-12-08 17:46:02 +00:00