Commit Graph

7443 Commits

Author SHA1 Message Date
Erik van Velzen
37b4981cc0
satellite/metabase/rangedloop: measure observer duration (#5350)
Track duration of all segment loop observers. Factor out functions to
reduce size.

Still need to send the measurements out via monkit.

Part of https://github.com/storj/storj/issues/5223

Change-Id: Iae0260e250f8ea33affed95c6592a1f42df384eb
2022-12-21 21:58:08 +01:00
Clement Sam
cda1d67465 cmd/tools/segment-verify: adjust to SN Exists endpoint
Change-Id: I409aeae29aa87996f2a6047f976d215a69e9d7f5
2022-12-21 19:24:31 +00:00
Malcolm Bouzi
9535444dcd web/satellite: adds satellite selection through tab control on the forget password page.
ticket: https://github.com/storj/storj/issues/5164
Change-Id: I1e0cefe75883a12c7b26ad42b7c44fddb7e93d10
2022-12-20 16:59:04 +00:00
NickolaiYurchenko
0e3e588718 web/satellite: frozen status check and banner added
added api call, user isFrozen status;
new banner added;
DashboardArea migrated to use composition api;

Change-Id: Ia3fe68c2239064b2b729c6de14d7fc1dc8f4cf3b
2022-12-20 17:58:49 +02:00
Wilfred Asomani
54559be966 console/userinfo: return appropriate errors
This modifies the userinfo endpoint to return appropriate errors;
PermissionDenied for untrusted peers and Unimplemented because
the endpoint isn't implemented

Change-Id: I5109bb204b5e1ce2e21fe16b003991b6c900a8ce
2022-12-20 10:50:18 +00:00
Vitalii
2ebdc5ff2f web/satellite: unauthorized error (401) interception for http requests
Implemented interception for http requests.
We redirect user to login page on every 401 response.

Issue:
https://github.com/storj/storj/issues/5339

Change-Id: Icba4fc0031cb2b4e682a1be078cdcf95b7fa6bfe
2022-12-20 07:14:17 +00:00
Wilfred Asomani
e598c2b3b1 console/userinfo: stub userinfo endpoint
This change stubs userinfo endpoint from storj/common/pb/userinfo.proto.
It also adds config for allowed peers, and a method for verifying peers.

Issue: https://github.com/storj/storj/issues/5358

Change-Id: I057a0e873a9e9b3b9ad0bba69305f0d708bd9b9e
2022-12-20 06:10:08 +00:00
Vitalii
dcb16d83dd web/satellite: fix for open bucket and bucket search issues
Fixed bucket search not being cleared on component unmount.
Fixed open bucket issue for project dashboard presented by the conflict with vue-click-outside directive.

Change-Id: I3682c6b131dcdb902369cf74b31c1b270cd15b08
2022-12-20 01:12:06 +02:00
Cameron
01932bda42 satellite/admin: add endpoint to freeze/unfreeze user
Allow the admin to manually freeze/unfreeze users.

github issue: https://github.com/storj/storj/issues/5397

Change-Id: I402ad1bf2e13effb0a5a8ff35bb128d1fcf18448
2022-12-19 18:11:05 +00:00
Fadila Khadar
d23e25ce0f cmd/tools/segment-verify: remove unused test code
Accidentally added some code to a test. As it is unused, this PR removes it.

Change-Id: I7adddc78c5ed747225e365989ab58504a9625ad7
2022-12-19 14:33:08 +00:00
Michal Niewrzal
d1d617d654 storagenode/piecestore: small cleanup
* normalize ExistsCheckWorkers flag to avoid setting incorrect values
* wait for limiter to finish if context was canceled

Change-Id: I688a395bd958cd09233605fc264d43f91ec45ed1
2022-12-19 12:37:12 +00:00
Ethan Adams
1c309a0318 cmd/tools/segment-verify: check for unvetted nodes
this also renames the command from `duplicates` to `node-check`

Change-Id: Idd303b17ec03f5b55fbbb1f4039a7761da37abe6
2022-12-19 09:59:13 +00:00
Michal Niewrzal
5110803102 storagenode/piecestore: add Exists endpoint
Adds new method Exists which can be used to verify which
requested piece ids exists on storage node. Will verify only pieces
which belongs to the satellite that used that endpoint.

Minum WASM size was increased a bit.

https://github.com/storj/storj/issues/5415

Change-Id: Ia5f9cadeb526541b2776a8973eb7d50133ad8636
2022-12-17 04:08:26 +00:00
Lizzy Thomson
678bb12d4b satellite/admin: create an endpoint for isAccountFrozen
Endpoint checks if an account is frozen or unfrozen.

relates to https://github.com/storj/storj/issues/5398

Change-Id: I8ff44063870327e05cf729eaaaed1da6c5fa9217
2022-12-16 22:53:44 +00:00
Jeremy Wharton
471f9e4e10 satellite/{console,satellitedb}: add account freeze service
This change adds an account freeze service with methods for checking
if a user is frozen, freezing a user, and unfreezing a user.
Furthermore, methods for altering the usage limits of a user or project
have been implemented for use by the account freeze service.

Change-Id: I77fecfac5c152f134bec90165acfe4f1dea957e7
2022-12-16 22:24:58 +00:00
paul cannon
7b851b42f7 satellite/audit: split out auditor process
This change creates a new independent process, the 'auditor', comparable
to the repairer, gc, and api processes. This will allow auditors to be
scaled independently of the core.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: I8a29eeb0a6e35753dfa0eab5c1246048065d1e91
2022-12-16 12:44:32 -06:00
paul cannon
fc905a15f7 satellite/audit: newContainment->containment
Now that all the reverification changes have been made and the old code
is out of the way, this commit renames the new things back to the old
names. Mostly, this involves renaming "newContainment" to "containment"
or "NewContainment" to "Containment", but there are a few other renames
that have been promised and are carried out here.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I34e2b857ea338acbb8421cdac18b17f2974f233c
2022-12-16 17:59:52 +00:00
Andrew Harding
99206fca04 satellite/payments/stripecoinpayments: skip line item generation for records with no usage
This change updates the stripecoinpayments service to optionally skip
generating line items for payments records that have no egress, storage,
or segments for the billing period.

This results in a reduction from 4 to 1 Stripe API calls for customers
who have no usage. The final API call is the attempt to generate an
invoice on stripe, which expectedly fails because there are no unapplied line
items. Removing that final API call would require some additional
queries and is out of scope for this change.

This functionality is behind the
`payments.stripe-coin-payments.skip-empty-invoices` feature flag.

https://github.com/storj/storj/issues/5381

Change-Id: Id184969a4c79047c40502336d69c51388ab03bf8
2022-12-16 17:02:50 +00:00
Egon Elbre
ee71fbb41d storagenode/piecestore: start restore trash in the background
Starting restore trash in the background allows the satellite to
continue to the next storagenode without needing to wait until
completion.

Of course, this means the satellite doesn't get feedback whether it
succeeds successfully or not. This means that the restore-trash needs to
be executed several times.

Change-Id: I62d43f6f2e4a07854f6d083a65badf897338083b
2022-12-16 18:15:52 +02:00
Andrew Harding
73d5c6944a satellite/audit: merge support for reservoirs
Change-Id: Ibbedd2a0043412210159fa2523f9e63d987276c3
2022-12-16 15:27:55 +00:00
paul cannon
0342ca1aa6 satellite/audit: delete now-unused code
Now that we are doing scalable piecewise reverifications, the code for
handling the old way of doing things (containment, pending audits,
reporting, testing) can now be removed.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ief1a75f423eff682e8f3d57804e343b3409a6631
2022-12-16 14:53:39 +00:00
paul cannon
a66503b444 satellite/audit: Begin using piecewise reverifications
This commit pulls the big switch! We have been setting up piecewise
reverifications (the workers for which can be scaled independently of
the core) for several commits now, and this commit actually begins
making use of them.

The core of this commit is fairly small, but it requires changing the
semantics in all the tests that relate to reverifications, so it ends up
being a large change. The changes to the tests are mostly mechanical and
repetitive, though, so reviewers needn't worry much.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ibb421cc021664fd6e0096ffdf5b402a69b2d6f18
2022-12-16 14:21:13 +00:00
nadimhq
9190a549ad
docs/testplan: Testplan for Automatic Account Freeze-Unfreeze (#5336)
* docs/testplan: Testplan for Automatic Account Freeze-Unfreeze

This testplan is going to cover the new account Freeze/Unfreeze. It will go over the automatic-account-freeze design doc found under docs/blueprints.

* Update automatic-account-freeze-unfreeze.md

Co-authored-by: Antonio Franco (He/Him) <antonio@storj.io>
2022-12-16 15:10:57 +01:00
Egon Elbre
04f16f8768 cmd/tools/segment-verify: tool for checking duplicate net
Change-Id: Ie47c1282e580ffc418bf3b1f3c8820a48973aefc
2022-12-15 22:58:36 +00:00
Vitalii
92a757cf3f web/satellite: clear vuex state on session timeout
We have to clear vuex app state on session timeout because user's session gets invalidated and user is redirected to login screen.

Issue:
https://github.com/storj/storj/issues/5370

Change-Id: Id654056331c81fac0b46ed90eccea0a044e4e1c9
2022-12-15 21:57:32 +00:00
NickolaiYurchenko
bb170a9d57 web/satellite: VCheckbox, VButton migrated to use composition api
related tests moved to ignored folder or fixed and will be changed after;

Change-Id: Idb4b1a93ac176b98184c746222b5cd1e30dad595
2022-12-15 18:10:08 +00:00
NickolaiYurchenko
8d30b58a8c web/satellite: VInfo, VerticalArrows, VDateRangePicket migrated to use composition api
related tests moved to ignored folder;

Change-Id: Ic76fe5f6da8088484ddcd2956bb6bb6c0d5d6223
2022-12-15 17:30:46 +00:00
NickolaiYurchenko
55774bac10 web/satellite: VLoader, VModal, VList migrated to use composition api
VLIst component and tests deleted;
related tests moved to ignore folder;

Change-Id: Id65754a044cdc8adcd551a6f4e3c5df65b11ffa0
2022-12-15 16:59:59 +00:00
NickolaiYurchenko
e54d9b1c6a web/satellite: VAlidationMessage, TableItem and TablePagination migrated to use composition api
Change-Id: Id02dd4f32fdd9924a2528a7363682fb6e1cb7afa
2022-12-15 16:02:13 +00:00
Jeremy Wharton
d069045058 satellite/{console,satellitedb}: implement account freeze event DB
This change implements DB methods for interacting with the
account_freeze_event table and introduces structures related to
account freeze events.

Change-Id: Ib125b31dfb754b2428212c39b780e14cfc7f97bf
2022-12-15 14:16:24 +00:00
Clement Sam
951d5db7f7 storagenode: fix hour_interval for first day defaulted to 24h
Previously because of the use of a LAG to calculate the hour_interval
the first record, which is usually the first day of the month usually,
doesn’t have a previous record and always assumes the at_rest_total is
for 24 hours.

Resolves https://github.com/storj/storj/issues/5390

Change-Id: Id532f8b38fe9df61432e62655318ff119a733d13
2022-12-15 13:30:11 +00:00
Andrew Harding
93fad70e4b satellite/audit: prevent accessing unset reservoir segments
This change fixes the access of unset segments and keys on the reservoir
when the reservoir size is less than the max OR the number of sampled
segments is smaller than the reservoir size. It does so by tucking away
the segments and keys behind methods that return properly sized slices
into the segments/keys arrays.

It also fixes a bug in the housekeeping for the internal index variable
that holds onto how many items in the array have been populated. As part
of this fix, it changes the type of index to int8, which reduces the
size of the reservoir struct by 8 bytes.

The tests have been updated to provide better coverage for this case.

Change-Id: I3ceb17b692fe456fc4c1ca5d67d35c96aeb0a169
2022-12-14 17:43:17 -07:00
Fadila Khadar
5c3a148d6e satellite/overlaycache: fix typo in UpdateCheckIn request
- fix a malformed SQL query
- add test to be sure we don't have this problem again.

Change-Id: I3fde8c59ba01335411e51d964bec95bc26cfc961
2022-12-14 22:21:45 +01:00
NickolaiYurchenko
4fad04e153 web/satellite: autocomplete search bug fixed
problem: after clicking on any bucket serch input was prefilled weird by browser and causes table empty state to appear;
solution: added workaround to add readonly status for input and disable it on focus;

Change-Id: I3dd05e709ef3aceb3b8542fe468bdb41b7b6d353
2022-12-14 19:39:27 +00:00
paul cannon
727136141a satellite/cmd/tools/segment-verify: check all pieces
This adds the capability to the segment-verify tool of checking all
pieces of every indicated segment.

Pieces which could not be accessed (i.e. we couldn't get a single
byte from them) are recorded in a csv file.

I haven't been able to test this in any very meaningful way, yet, but I
am comforted by the fact that the worst things it could possibly do are
(a) download pieces too many times, and (b) miss downloading some
pieces.

Change-Id: I3aba30921572c974993363eb36d0fd5b3ae97907
2022-12-14 19:06:08 +00:00
Qweder93
9d7e94b57b private/testplanet: extended satellite with rangedloop peer
Change-Id: I3c82f5c970c4fef139fc089937e43c5cb1769815
2022-12-14 16:21:50 +02:00
NickolaiYurchenko
235e9be208 web/satellite: VBar, VBanner migrated to use composition api
composables folder created (same as react hooks) for functionality reusage;
related test moved to ignored folder;

Change-Id: I494aa27079aa5694bee7b18511eeadd56ced59e9
2022-12-14 11:21:11 +00:00
Moby von Briesen
870eefbb5d docs/blueprints: Add paid tier tls document
Add document outlining the design and implementation steps for the paid
tier TLS feature.

https: //github.com/storj/storj/issues/5296
Change-Id: I51f68fc7890f816cef7bf2a319762ad701bac445
2022-12-13 23:08:23 +00:00
Lizzy Thomson
eee79ec3cd web/satellite: fix project selection logic
fixes https://github.com/storj/storj/issues/5294

Change-Id: I4cc3110e94700b0f31d20edabfd31482ae891200
2022-12-13 22:08:12 +00:00
paul cannon
ad82e74cb7 satellite/satellitedb: add reverifyqueue to safelyPartitionableDBs
Adding this entry means that the database accessed as "reverifyqueue"
(`(*satelliteDBCollection).ReverifyQueue()`) can be located on a
different database host from the other databases, and things should
still work. There aren't any queries that do a JOIN on tables from
reverifyQueue and other things in satellitedb, for example.

This should really have been put here earlier, when reverifyqueue was
first added, but it's ok. This won't have any bearing on things until we
need to deploy to prod.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I76f68de79cd645c869f3dbfbe3b2c9c4f9359e8f
2022-12-13 21:14:17 +00:00
paul cannon
47b9134f76 satellite/audit: add IdentifyContainedNodes
This method on the Verifier allows the caller to find, out of the nodes
holding pieces in a given segment, which ones are contained.

This method is not yet being used. It will be in a future commit.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I242cd999913ca4dabbe8a62767ed4869b31fca04
2022-12-13 20:46:43 +00:00
Fadila Khadar
995f78d579 satellite/cmd: segment-verify verifies segments in given bucket list
Provides the `segment-verify run buckets` command for verifying segments within a list of buckets.

Bucket list is a csv file with `project_id,bucket_name` to be checked.

https://github.com/storj/storj-private/issues/101

Change-Id: I3d25c27b56fcab4a6a1aebb6f87514d6c97de3ff
2022-12-13 20:10:00 +00:00
Cameron
021c98c17a satellite/satellitedb: create table account_freeze_events
create table for account freeze events

github issue: https://github.com/storj/storj/issues/5396

Change-Id: Ia4f93ce5caacd02b161853e3261d237bc715f23d
2022-12-13 19:09:25 +00:00
Vitalii
5d727becb4 satellite/{analytics, web}: segment ui error tracking implemented
Implemented UI error tracking.
We use satellite analytics service to track the fact that UI error occurred and send minimal info to Segment (not Hubspot).
We send only the fact that UI error occurred and the place where this error occurred.
Extended notificator plugin error function to include the place where error occurred.
I made the place argument nullable to be always explicitly provided (build fails if place is not provided).
If place is not null then error event is triggered in the background.

Issue:
https://github.com/storj/storj-private/issues/107

Change-Id: I7d129fb29629979f5be6ff5dea37ad19b1a2397e
2022-12-13 18:11:53 +00:00
Andrew Harding
7f2dfaa394 gitignore: add go workspace files
Change-Id: Icd382bb8da16c6100ef68340d665e2a29420b6a2
2022-12-13 10:15:53 -07:00
Lizzy Thomson
0afd3938c7 sat/console update the updateProject to set user specified limits
update the updateProject function to set user specified bandwidth and storage limits

fixes https://github.com/storj/storj/issues/5185

Change-Id: Ib4132487f6b7ea0afa7c57acfc358857b3e852d1
2022-12-13 16:44:41 +00:00
Vitalii
17db59e27a web/satellite: update apollo-client dependency to remove CSP error
This is insane but the problem was caused by the old apollo-client dependency which is 3 years old.
The error was coming from ts-invariant code here
https://github.com/apollographql/invariant-packages/blob/master/packages/ts-invariant/src/invariant.ts#L64

By this change I updated our app to use new @apollo/client dependency but there is one pretty big concern.
This lib has react as a peer dependency (what?!) so to avoid adding react to our vue project I had to import needed functionality from @apollo/client/core instead of just @apollo/client.

There is a discussion here https://github.com/apollographql/apollo-client/issues/7318
According to this ^ apollo team are going to remove react from their peer dependecies list when apollo V4 will be launched.
See https://github.com/apollographql/apollo-client/issues/8190

That's another point why we should move away from using graphql entirely or at least stop using apollo.

Issue:
https://github.com/storj/storj/issues/5371

Change-Id: Ifd484efbcd9e99d8c2e21d0dab93670c9358e25c
2022-12-13 16:04:50 +00:00
Michal Niewrzal
0bbbb9c4c1 satellite/metabase: fix log for multiple committed version
Change-Id: I2556c5b523091c11937a01efff07be9e0dd964aa
2022-12-13 13:08:02 +00:00
Márton Elek
ef2c0f4225 mod: use v1.10.0 tagged commit from storj.io/uplink
Change-Id: I475f2cf6447453dd78eb3549fdaacd286e1da494
2022-12-13 12:04:29 +00:00
Michal Niewrzal
0759cbdc7f satellite/metabase: handle copies with GetStreamPieceCountByNodeID
We missed proper handling of object copies for method
GetStreamPieceCountByNodeID which is used by metabase.GetObjectIPs.
That caused some lack of IPs returned when queriyng IPs of copy and
broke things like pices map on linksharing.

Fixes https://github.com/storj/storj/issues/5406

Change-Id: I9574776f34880788c2dc9ff78a6ae20d44fe628f
2022-12-13 12:32:56 +01:00