Commit Graph

3575 Commits

Author SHA1 Message Date
dlamarmorgan
a014af45eb satellite/metabase: add tests CommitObjectWithSegments.Versioned
Change-Id: Ied838e6bee4e2d8254708d21c90edb68bdf05916
2023-10-23 11:48:33 +00:00
Egon Elbre
d3429fafd0 satellite/metabase: add CommitObjectWithSegments.Versioned
Change-Id: I4e1fc82b45cca94ff0b48d5b8d9deb6e13d0957b
2023-10-23 14:46:56 +03:00
Jeremy Wharton
1d5ea2d35c web/satellite: allow free users to invite if billing features disabled
This change ensures that the user is never prompted to upgrade when
attempting to invite project members if billing features are configured
to be disabled.

Change-Id: I1c49351b00c0e378da24ad080fd1d3b078c97c71
2023-10-20 21:49:36 +00:00
Jeremy Wharton
a6222afdd0 satellite/console,web/satellite: configure whether free user can invite
This change adds a flag to the satellite config indicating whether
free tier users should be able to send project invitations.

Change-Id: I9c030c88dbef136ba4a9bf2d8f027a8dcd77fd33
2023-10-20 21:06:19 +00:00
Moby von Briesen
e5fd061e70 satellite/satellitedb: Drop value_attributions.partner_id
Removes final instance of legacy "partner ID" column.

Resolves https://github.com/storj/storj/issues/5432

Change-Id: I89657c8f9f366c79a5439632f870a47db590ff99
2023-10-20 20:25:13 +00:00
paul cannon
b2c6ec0091 satellite/metabase: change versions column to INT8
This will allow further progress towards S3-compatible object
versioning.

Refs: https://github.com/storj/storj/issues/6352
Change-Id: I0b3aa93fcacd1f9d91a667d619d6cb41fba602a9
2023-10-20 17:42:23 +00:00
Clement Sam
67bd36ae4d satellite/satellitedb: remove nodes.type from indices on nodes table
Since we no longer need the type column and hopefully,
there are no queries using it, we recreate all indices
that reference the nodes.type column.

Updates https://github.com/storj/storj/issues/5426

Change-Id: I610ccaf474a6f4031e166b79a6d649c4b138e338
2023-10-20 14:48:36 +00:00
Egon Elbre
74e4368cf1 satellite/metabase: add tests CommitObject.Versioned
Change-Id: Ie26e8806cfea0d0ba6ce58d290d329e04754f20a
2023-10-20 00:02:00 +00:00
Egon Elbre
a23d9d20aa satellite/metabase: add CommitObject.Versioned
This allows to commit versioned objects.

Change-Id: I7ae100e508a23899392ba40084198617fe3e4e0c
2023-10-19 23:20:48 +00:00
Jeremy Wharton
0c1306780e satellite/payments/accountfreeze: fix race in test
This change fixes data races in tests for the account freeze chore
caused by the chore service being set while in use.

Change-Id: Ibcb94a947c405314deae0d77419e69b768feab40
2023-10-19 19:41:39 +00:00
Clement Sam
ab57cd30b0 satellite: remove all code references to nodes.type column
Once this is merged and deployed, we can go ahead
and change the db schema.

Note that this will stop updating the `type` column in the nodes
table but it shouldn't be a problem because this column has a
default value set.

Updates https://github.com/storj/storj/issues/5426

Change-Id: I2470ebacbcb0e60cf894617eb69b593227357283
2023-10-19 19:00:33 +00:00
Wilfred Asomani
be025a40cc satellite/{db,console}: add days column to account freeze table
This change adds a new column to the account_freeze_events table,
days_till_escalation, which is the number of days till the freeze event
is escalated. E.g.: 30 days from billing warning event to billing
freeze event.

Issue: #6382

Change-Id: I15cfd6f1208e641d8c380bef61717031befdad73
2023-10-19 18:18:44 +00:00
Márton Elek
5c49ba1d85 satellite/durability: ignore information from new nodes
To get better performance, we pre-load all nodealias/node information at the beginning of the segment loop.

It's possible that we receive a new node alias from the segment table what we are not fully aware of (yet).

The easiest solution is just ignoring. New risks/threats can be detected by a new execution cycle.

Change-Id: Ib54f7edc46eedbab6d13b4d651aaac1425994940
2023-10-19 16:38:39 +00:00
Márton Elek
a63a69dfd9 satellite/nodeselection: support OR in placement definition
Change-Id: Icc7fd465b28c0c6f09f50c4ab8bffbcc77631dbd
2023-10-19 15:21:15 +00:00
Márton Elek
188aa3011b satellite/repair/checker: report checker_segment_off_placement_count per placement
Change-Id: Ic1639899f8f0b55c4ef8fe246e7efc0a5d9a2bc1
2023-10-19 11:59:56 +00:00
Vitalii
6ae28e2306 satellite/{web,console}: enable/disable billing features depending on config value
Added client side logic to disable billing features depending on config value.
Disabled billing endpoints if billing is disabled.

Issue:
https://github.com/storj/storj-private/issues/464

Change-Id: I6e70dc5e2372953b613ddab9f19cb94f008935ce
2023-10-18 21:00:43 +00:00
Clement Sam
bce022ea7a satellite/overlay: remove Type field from NodeDossier
The overlay.NodeDossier struct only tracks information about a
storagenode, the field is deprecated and no longer needed.
This is a kademlia left-over.

Updates https://github.com/storj/storj/issues/5426

Change-Id: Ie278ffd88d1b9a9fde6c81eb5f0e287bab8c9ef0
2023-10-18 18:21:26 +00:00
Clement Sam
0d144ee5af satellite/contact: remove dependency on overlay.NodeDossier
`overlay.NodeDossier` contains information that satellite tracks
about a storagenode.
As part of the work to remove the type column from the nodes
table https://github.com/storj/storj/issues/5426, we would have
to remove the `Type` field in the `overlay.NodeDossier` since it
is only about the storagenode.

The `Local` method on the contact service is also removed because
it is unused. When needed we can create a seperate `SatelliteInfo`
struct which the `Local` method will return.

Change-Id: If0c1a25d9df397a9492bbf1d7f33ba5b6a918878
2023-10-18 18:11:36 +00:00
Wilfred Asomani
32b7b80666 satellite/{console,accountfreeze}: reduce payment retries
This change limits payment attempts to
1. Card updates when billing frozen/warned
2. Right before billing freezing a warned account.

Issue: https://github.com/storj/storj-private/issues/457

Change-Id: Ic6d5c649cdac38d5c3b7365e20a4ceb3b6199ee8
2023-10-18 15:54:46 +00:00
Jeremy Wharton
f8b59a50ff satellite/console: configure sending invites to unregistered emails
This change adds a flag to the satellite config indicating whether
unregistered email addresses should receive project invitation emails.

Change-Id: I0396f25574ddae3f9adaea32a6e7cd15b931bf12
2023-10-17 22:42:57 -05:00
Vitalii
4721d2bd4e satellite/console: feature flag for billing features
Change-Id: I27623b2a9cd100ef38dda34bb48e332d616a478c
2023-10-17 22:56:29 +00:00
Moby von Briesen
e0cf6f1e8b satellite/console,web/satellite: Change Vuetify app prefix to "v2"
Change-Id: Ib0e14bd2f5056c6300a640e678413de767e8ee6d
2023-10-17 22:12:46 +00:00
Jeremy Wharton
2cf4784b20 satellite/console: make project invites exclusive to paid tier
This change makes the project member invitation feature exclusive to
the paid tier.

Change-Id: I13c967c8381d49b2d131e15799ad48487b0f6c74
2023-10-17 09:38:29 -05:00
Jeremy Wharton
524e074a8c satellite/console,web/satellite: disallow creating multiple new invites
This change prevents multiple project invitation records from being
created from a single API request.

Change-Id: I01268fcc0e2f7b5f24870b032cb53f03c7ad0800
2023-10-17 09:05:00 -05:00
Egon Elbre
b2d2a8a744 satellite/metabase: add unique unversioned constraint for tests
While the index shouldn't be necessary as long as our implementation is
correct, it still provides some additional checks for mistakes in the
implementation.

Change-Id: I7ed71ac99a979e375d7f94c8898e6f83ac623cb6
2023-10-17 12:08:50 +00:00
Michal Niewrzal
d7af97c919 satellite/satellitedb: move IterateBucketLocations sql to dbx
This is attempt to move query back to dbx.

It also removes one unused method.

Change-Id: I8182dd8ecf794cdf0cb3158c36cc00810fc683df
2023-10-17 10:30:51 +00:00
Egon Elbre
7ba8a627bc satellite/metabase: add deleteObjectUnversionedCommitted
By using a separate function for deleting the latest object and
fetching the latest version we can simplify some of the code.

However, there can be more performant approaches, such as using
ON CONFLICT for updating the existing object or using select and delete
in the same query in databases that support it.

Change-Id: I52bc3f9fa025f44d05ee010723ffb81f5bd2a2d7
2023-10-17 11:44:35 +03:00
Vitalii
e3713fddb8 satellite/{console, web}: added detailed usage report
Allow user to download detailed usage report from Billing -> Overview screen.
Report is a CSV file containing usage data for all the projects user owns.

Issue:
https://github.com/storj/storj/issues/6154

Change-Id: I3109002bf37b1313652a2be3447aaa7bc6204887
2023-10-16 21:54:18 +00:00
Egon Elbre
41e16bc398 satellite/metabase: versioning, add new statuses
There are several different object types in a versioned table,
which will determine the exact behaviour.

The object type states are:
* Pending   - the object is yet to be committed and is being uploaded.
* Committed - the object has been finished and can be read.
* DeleteMarker - indicates that the object should be treated as not
  present when is at the top of the version stack.

There are also versioning states:
* Unversioned - only one unversioned object is allowed per object key.
* Versioned - multiple objects with the same key are allowed.

Change-Id: I65dfa781e8da253a4e5d572b799d53c351196eee
2023-10-16 17:50:18 +00:00
paul cannon
58f75502f3 satellite/metabase: exclude expired segments from ListVerifySegments results
I believe that this change in semantics won't break anything, because
ListVerifySegments is only used by cmd/tools/segment-verify (which only
needs to operate on non-expired segments) and various tests, none of
which expect ListVerifySegments to include expired segments.

Change-Id: I037f43b16bc5750ed914bc32949418e001df1a8c
2023-10-16 16:58:01 +00:00
Michal Niewrzal
281edfa585 satellite/satellitedb: add index to bucket_storage_tallies table
Additional index on 'interval_start' to support data analysis.

Change-Id: I373787871da594aa1b4ebd66cb684484221eb4d5
2023-10-16 15:09:24 +00:00
Jeremy Wharton
d200726436 satellite/console: rate limit project invitation endpoint
This change adds user ID rate limiting to the endpoint responsible for
sending project invitations.

Resolves storj-private#462

Change-Id: Icf0be7d7bb7f2765725ba3e152a2195bc02484e2
2023-10-16 14:24:33 +00:00
Michal Niewrzal
0eaf43120b satellite/repair/checker: optimize processing, part 3
ClassifySegmentPieces uses custom set implementation instead map.

Side note, for custom set implementation I also checked int8 bit set but
it didn't give better performance so I used simpler implementation.

Benchmark results (compared against part 2 optimization change):
name                                       old time/op    new time/op    delta
RemoteSegment/healthy_segment-8    21.7µs ± 8%    15.4µs ±16%  -29.38%  (p=0.008 n=5+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/healthy_segment-8    7.41kB ± 0%    1.87kB ± 0%  -74.83%  (p=0.000 n=5+4)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/healthy_segment-8       150 ± 0%       130 ± 0%  -13.33%  (p=0.008 n=5+5)

Change-Id: I21feca9ec6ac0a2558ac5ce8894451c54f69e52d
2023-10-16 12:06:16 +00:00
Márton Elek
c3fbac2e7a satellite/overlay/placement: better error message for placement parsing
Given the placement.txt with the content:

```
9:exclude(placement(10))
10:country("DE")
```

Without patch:

```
placement-test --placement=/tmp/placement.txt countrycode=DE
Error: type mismatch: reflect: Call using zero Value argument
```

With the patch:

```
placement-test --placement=/tmp/placement.txt countrycode=DE
Error: Error in line 'exclude(placement(10))' when placement rule is parsed: Placement 10 is referenced before defined. Please define it first!
```

Change-Id: I9ad81016d4a57fdb32f3ff9031b5591f9a7cd2a6
2023-10-16 09:28:45 +00:00
Vitalii
356eb43a9f satellite/console: make Etherscan URL more configurable
make the block explorer URL (e.g. https://etherscan.io) configurable (default to using https://etherscan.io/)

Issue:
https://github.com/storj/storj/issues/6357

Change-Id: I1c9f14eb4653763318fd96c242130a9c0a7d24c5
2023-10-14 01:58:43 +00:00
paul cannon
ee33cb1289 satellite/repair: protect concurrent access to statsCollector
It would appear that we have been making concurrent accesses to
statsCollector for a long, long time (we expect there to be multiple
calls to `Repair()` at the same time on the same instance of
`SegmentRepairer`, up to `config.MaxRepair`, and before this change
there was no sort of synchronization guarding accesses to the
`statsCollector.stats` map.

Refs: https://github.com/storj/storj/issues/6402
Change-Id: I5bcdd13c88913a8d66f6dd906c9037c588960cc9
2023-10-13 09:12:00 -05:00
Jeremy Wharton
ad13cb2bf5 web/satellite, satellite/analytics: add UI type to analytics properties
This change incorporates the UI type (legacy or Vuetify) in the
properties attached to analytics events originating from the satellite
UI.

Resolves #6363

Change-Id: Ie3627bc24e4349407376e28460a5a830d211b47b
2023-10-13 11:43:54 +00:00
Wilfred Asomani
c9421d11e7 satellite/admin: add endpoints to violation freeze/unfreeze users
This change adds two new admin endpoints to freeze users for ToS
violation and to remove them from that state,

Issue: https://github.com/storj/storj-private/issues/386

Change-Id: I49c922377c9cdb315ce2777fcd35dcad432b0539
2023-10-13 10:43:44 +00:00
Wilfred Asomani
594e63f13a satellite/accountfreeze: mark billing frozen users for deletion
This change modifies the billing freeze chore to set pending deletion
status to users who are still frozen after a grace period.
It also modifies the chore to skip already deleted users.

Issue: https://github.com/storj/storj/issues/6303
https://github.com/storj/storj-private/issues/453

Change-Id: I4d0e7dd904463e99424372dc9ac81b71c0bc6e28
2023-10-12 20:25:23 +00:00
Márton Elek
db3578d9ba satellite: durability rangeloop observer for monitoring risks
Change-Id: I92805fcc6e7c1bbe0f42bbf849d22f9908fedadb
2023-10-12 16:32:30 +00:00
Michal Niewrzal
e3e303754b satellite/repair/checker: optimize processing, part 2
Optimizing collecting monkit metrics:
* initialize metrics once at the begining
* avoid using string in map for getting stats structs per redundancy

Benchmark results (compared against part 1 optimization change):
name                                       old time/op    new time/op    delta
RemoteSegment/Cockroach/healthy_segment-8    31.4µs ± 6%    21.7µs ± 8%  -30.73%  (p=0.008 n=5+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/healthy_segment-8    10.2kB ± 0%     7.4kB ± 0%  -27.03%  (p=0.008 n=5+5)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/healthy_segment-8       250 ± 0%       150 ± 0%  -40.00%  (p=0.008 n=5+5)

Change-Id: Ie09476eb469a4d6c09e52550c8ba92b3b4b34271
2023-10-12 10:02:53 +02:00
Michal Niewrzal
de4559d862 satellite/repair/checker: optimize processing, part 1
Optimization by reusing more slices.

Benchmark result:
name                                       old time/op    new time/op    delta
RemoteSegment/healthy_segment-8    33.2µs ± 1%    31.4µs ± 6%   -5.49%  (p=0.032 n=4+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/healthy_segment-8    15.9kB ± 0%    10.2kB ± 0%  -35.92%  (p=0.008 n=5+5)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/healthy_segment-8       280 ± 0%       250 ± 0%  -10.71%  (p=0.008 n=5+5)

Change-Id: I60462169285462dee6cd16d4f4ce1f30fb6cdfdf
2023-10-11 15:50:29 +00:00
Wilfred Asomani
6308da2cc0 satellite/{payment,console,analytics} extend freeze functionality for violation freeze
This change extends the account freeze functionality account for
violation freezes as well.
Also, debug level logs in the freeze chore have been changed to info.
It adds an analytics event for when an invoice is found that belongs to
a user frozen for violation.
And finally adds whether a user is frozen for violation to the
/account/freezestatus response.

Issue: https://github.com/storj/storj-private/issues/386

Change-Id: Id8e40282dc8fd8f242da52791ab8ddbbef3da2bc
2023-10-10 18:39:29 +00:00
Cameron
0cc04208db satellite/console: add limit increase errors to getUserErrorMessage
Change-Id: Icc84b8d09d3c1edc3ae216b71812442d67e3b5c9
2023-10-09 17:44:48 +00:00
Moby von Briesen
3216674c19 satellite/payments/accountfreeze: Increase exclusion threshold
Account freeze excludes accounts who have invoices which are unpaid and
are above a configured value. This change updates that configured value
from $100 to $1000.

Change-Id: Iff381deeb73ef2d8fccaf7a1612a33e8aeb08698
2023-10-06 14:39:20 +00:00
dlamarmorgan
7f02b73b5d cmd/satellite/main: complete/fail pending payments cmd
Add billing commands to manually complete or fail storjscan invoice
payments that are stuck in a pending state.

Change-Id: Ia19f0a2597201d9d17aad0889eaedff095d706b9
2023-10-06 01:23:34 +00:00
Cameron
e072b37a86 satellite/console: add endpoint to request project limit increase
create endpoint to allow pro users to request project limit increase.

github issue: https://github.com/storj/storj/issues/6298

Change-Id: I96c3dff8bf0906904d199fc2c7ee738f3e6b04a3
2023-10-05 17:21:32 +00:00
Egon Elbre
27a13efb17 satellite/console/consoleweb/consoleapi: fix defer in loop
Change-Id: Ifdedad8a5556be47b5f99dc8d8a8ff67904e2688
2023-10-05 12:55:36 +03:00
Ivan Fraixedes
3193ff9155 private/apigen: Don't force casing for API group name/prefix
The API generators rely on the Name and Prefix fields of the
EndpointGroup type to generate code.

Conventional naming code requires using upper or lower case for types,
functions, etc, however requiring the user to set this fields with the
correct casing seems cumbersome for them because they can be adjusted
depending where those values are used on the generated code.

This commit lifts the restriction for the user and adjust the casing of
them according to where they are used.

Change-Id: I700a879d13b4789b4d6ba0519b4d7508061eac73
2023-10-05 06:48:50 +00:00
Sean Harvey
7b50ece931
satellite/admin: add missing test assert to project geofence tests
the existing test was missing a case where deleting the geofencing
should set the placement back to default.

this also changes EveryCountry to DefaultPlacement, as the former is
deprecated.

Change-Id: I29f4f1c3ae1d05e8f5ecaa03a70ac4f17574a475
2023-10-05 18:21:48 +13:00