Commit Graph

8655 Commits

Author SHA1 Message Date
Vitalii
6ae28e2306 satellite/{web,console}: enable/disable billing features depending on config value
Added client side logic to disable billing features depending on config value.
Disabled billing endpoints if billing is disabled.

Issue:
https://github.com/storj/storj-private/issues/464

Change-Id: I6e70dc5e2372953b613ddab9f19cb94f008935ce
2023-10-18 21:00:43 +00:00
Clement Sam
bce022ea7a satellite/overlay: remove Type field from NodeDossier
The overlay.NodeDossier struct only tracks information about a
storagenode, the field is deprecated and no longer needed.
This is a kademlia left-over.

Updates https://github.com/storj/storj/issues/5426

Change-Id: Ie278ffd88d1b9a9fde6c81eb5f0e287bab8c9ef0
2023-10-18 18:21:26 +00:00
Clement Sam
0d144ee5af satellite/contact: remove dependency on overlay.NodeDossier
`overlay.NodeDossier` contains information that satellite tracks
about a storagenode.
As part of the work to remove the type column from the nodes
table https://github.com/storj/storj/issues/5426, we would have
to remove the `Type` field in the `overlay.NodeDossier` since it
is only about the storagenode.

The `Local` method on the contact service is also removed because
it is unused. When needed we can create a seperate `SatelliteInfo`
struct which the `Local` method will return.

Change-Id: If0c1a25d9df397a9492bbf1d7f33ba5b6a918878
2023-10-18 18:11:36 +00:00
Wilfred Asomani
32b7b80666 satellite/{console,accountfreeze}: reduce payment retries
This change limits payment attempts to
1. Card updates when billing frozen/warned
2. Right before billing freezing a warned account.

Issue: https://github.com/storj/storj-private/issues/457

Change-Id: Ic6d5c649cdac38d5c3b7365e20a4ceb3b6199ee8
2023-10-18 15:54:46 +00:00
Jeremy Wharton
6d03b92ea6 private/apigen: fix TS class generation from collection of named struct
This change fixes an issue where the TypeScript API code generator
would silently fail to generate class definitions for named structs
used as the element type of array or slice struct fields.

This issue caused invalid TypeScript code to be generated from the
frontend config struct because it contained a slice of partnered
satellite structs. The TypeScript class definition of the frontend
config referenced the nonexistent partnered satellite class.
The satellite frontend config uses, so this issue caused the

Change-Id: Idfcb6ec1bbc603a43033ee4afb5b421b7454855c
2023-10-17 22:45:07 -05:00
Jeremy Wharton
f8b59a50ff satellite/console: configure sending invites to unregistered emails
This change adds a flag to the satellite config indicating whether
unregistered email addresses should receive project invitation emails.

Change-Id: I0396f25574ddae3f9adaea32a6e7cd15b931bf12
2023-10-17 22:42:57 -05:00
Jeremy Wharton
24ae79345b web/satellite{/vuetify-poc}: show upgrade dialog when trying to invite
This change displays a dialog prompting free tier users to upgrade when
the button to invite project members is clicked.

Also, the Create New Project dialog in the Vuetify UI now opens the
upgrade dialog when its Upgrade button is clicked.

Change-Id: I6e233bd15fd14a486a3e9008bbc6fba3e669d67e
2023-10-17 22:40:34 -05:00
Vitalii
4721d2bd4e satellite/console: feature flag for billing features
Change-Id: I27623b2a9cd100ef38dda34bb48e332d616a478c
2023-10-17 22:56:29 +00:00
Moby von Briesen
e0cf6f1e8b satellite/console,web/satellite: Change Vuetify app prefix to "v2"
Change-Id: Ib0e14bd2f5056c6300a640e678413de767e8ee6d
2023-10-17 22:12:46 +00:00
Jeremy Wharton
2cf4784b20 satellite/console: make project invites exclusive to paid tier
This change makes the project member invitation feature exclusive to
the paid tier.

Change-Id: I13c967c8381d49b2d131e15799ad48487b0f6c74
2023-10-17 09:38:29 -05:00
Jeremy Wharton
524e074a8c satellite/console,web/satellite: disallow creating multiple new invites
This change prevents multiple project invitation records from being
created from a single API request.

Change-Id: I01268fcc0e2f7b5f24870b032cb53f03c7ad0800
2023-10-17 09:05:00 -05:00
Egon Elbre
b2d2a8a744 satellite/metabase: add unique unversioned constraint for tests
While the index shouldn't be necessary as long as our implementation is
correct, it still provides some additional checks for mistakes in the
implementation.

Change-Id: I7ed71ac99a979e375d7f94c8898e6f83ac623cb6
2023-10-17 12:08:50 +00:00
Vitalii
45fdc64300 web/satellite/vuetify-poc: download total usage report
Added CTA to billing -> overview screen for user to download total usage report

Issue:
https://github.com/storj/storj/issues/6154

Change-Id: I465c955486e6fa9bf922d56798b2338c4bd2d73f
2023-10-17 11:26:36 +00:00
Michal Niewrzal
d7af97c919 satellite/satellitedb: move IterateBucketLocations sql to dbx
This is attempt to move query back to dbx.

It also removes one unused method.

Change-Id: I8182dd8ecf794cdf0cb3158c36cc00810fc683df
2023-10-17 10:30:51 +00:00
Egon Elbre
7ba8a627bc satellite/metabase: add deleteObjectUnversionedCommitted
By using a separate function for deleting the latest object and
fetching the latest version we can simplify some of the code.

However, there can be more performant approaches, such as using
ON CONFLICT for updating the existing object or using select and delete
in the same query in databases that support it.

Change-Id: I52bc3f9fa025f44d05ee010723ffb81f5bd2a2d7
2023-10-17 11:44:35 +03:00
JT Olio
2578580e21 docs/blueprints: fix deletes and server side copy
Change-Id: I13dc3580bbfcba4dd97d1a3d073505f0e5688ab3
2023-10-17 07:52:03 +00:00
Vitalii
e3713fddb8 satellite/{console, web}: added detailed usage report
Allow user to download detailed usage report from Billing -> Overview screen.
Report is a CSV file containing usage data for all the projects user owns.

Issue:
https://github.com/storj/storj/issues/6154

Change-Id: I3109002bf37b1313652a2be3447aaa7bc6204887
2023-10-16 21:54:18 +00:00
Egon Elbre
41e16bc398 satellite/metabase: versioning, add new statuses
There are several different object types in a versioned table,
which will determine the exact behaviour.

The object type states are:
* Pending   - the object is yet to be committed and is being uploaded.
* Committed - the object has been finished and can be read.
* DeleteMarker - indicates that the object should be treated as not
  present when is at the top of the version stack.

There are also versioning states:
* Unversioned - only one unversioned object is allowed per object key.
* Versioned - multiple objects with the same key are allowed.

Change-Id: I65dfa781e8da253a4e5d572b799d53c351196eee
2023-10-16 17:50:18 +00:00
paul cannon
58f75502f3 satellite/metabase: exclude expired segments from ListVerifySegments results
I believe that this change in semantics won't break anything, because
ListVerifySegments is only used by cmd/tools/segment-verify (which only
needs to operate on non-expired segments) and various tests, none of
which expect ListVerifySegments to include expired segments.

Change-Id: I037f43b16bc5750ed914bc32949418e001df1a8c
2023-10-16 16:58:01 +00:00
paul cannon
23c5d6c287 cmd/tools/segment-verify: improve logging around common problems
Change-Id: I4f684745df708627f135baee619d17788bc8d63e
2023-10-16 16:14:23 +00:00
Michal Niewrzal
281edfa585 satellite/satellitedb: add index to bucket_storage_tallies table
Additional index on 'interval_start' to support data analysis.

Change-Id: I373787871da594aa1b4ebd66cb684484221eb4d5
2023-10-16 15:09:24 +00:00
Jeremy Wharton
d200726436 satellite/console: rate limit project invitation endpoint
This change adds user ID rate limiting to the endpoint responsible for
sending project invitations.

Resolves storj-private#462

Change-Id: Icf0be7d7bb7f2765725ba3e152a2195bc02484e2
2023-10-16 14:24:33 +00:00
Vitalii
902cf5898f web/satellite/vuetify-poc: remove required rule on S3 credentials info step
Remove required rule on S3 credentials info step when creating new one.

Issue:
https://github.com/storj/storj/issues/6400

Change-Id: Ie59267bb881e6dfa769ae0cdda83ff369ec920cd
2023-10-16 13:07:07 +00:00
Michal Niewrzal
0eaf43120b satellite/repair/checker: optimize processing, part 3
ClassifySegmentPieces uses custom set implementation instead map.

Side note, for custom set implementation I also checked int8 bit set but
it didn't give better performance so I used simpler implementation.

Benchmark results (compared against part 2 optimization change):
name                                       old time/op    new time/op    delta
RemoteSegment/healthy_segment-8    21.7µs ± 8%    15.4µs ±16%  -29.38%  (p=0.008 n=5+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/healthy_segment-8    7.41kB ± 0%    1.87kB ± 0%  -74.83%  (p=0.000 n=5+4)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/healthy_segment-8       150 ± 0%       130 ± 0%  -13.33%  (p=0.008 n=5+5)

Change-Id: I21feca9ec6ac0a2558ac5ce8894451c54f69e52d
2023-10-16 12:06:16 +00:00
Márton Elek
c3fbac2e7a satellite/overlay/placement: better error message for placement parsing
Given the placement.txt with the content:

```
9:exclude(placement(10))
10:country("DE")
```

Without patch:

```
placement-test --placement=/tmp/placement.txt countrycode=DE
Error: type mismatch: reflect: Call using zero Value argument
```

With the patch:

```
placement-test --placement=/tmp/placement.txt countrycode=DE
Error: Error in line 'exclude(placement(10))' when placement rule is parsed: Placement 10 is referenced before defined. Please define it first!
```

Change-Id: I9ad81016d4a57fdb32f3ff9031b5591f9a7cd2a6
2023-10-16 09:28:45 +00:00
Ivan Fraixedes
d8376a2a24 Earthfile: Use same NodeJS version as Makefile
Makefile defines the NodeJS version to use for building front-end
sources.

Earthfile must use the same version for guaranteeing the same results.

Change-Id: I74d5629a2a4489392d2a3303a03a7ccf012f8b24
2023-10-16 08:04:21 +00:00
Vitalii
356eb43a9f satellite/console: make Etherscan URL more configurable
make the block explorer URL (e.g. https://etherscan.io) configurable (default to using https://etherscan.io/)

Issue:
https://github.com/storj/storj/issues/6357

Change-Id: I1c9f14eb4653763318fd96c242130a9c0a7d24c5
2023-10-14 01:58:43 +00:00
paul cannon
ee33cb1289 satellite/repair: protect concurrent access to statsCollector
It would appear that we have been making concurrent accesses to
statsCollector for a long, long time (we expect there to be multiple
calls to `Repair()` at the same time on the same instance of
`SegmentRepairer`, up to `config.MaxRepair`, and before this change
there was no sort of synchronization guarding accesses to the
`statsCollector.stats` map.

Refs: https://github.com/storj/storj/issues/6402
Change-Id: I5bcdd13c88913a8d66f6dd906c9037c588960cc9
2023-10-13 09:12:00 -05:00
Vitalii
7381b5e508 web/satellite: fix object count calculation inside folders
Moved .file_placeholder decrement logic to be outside continuation tokens loop so that object count is decremented only once.

Change-Id: Ie19657753b501df9344dc84d7be2bfb731933faa
2023-10-13 12:27:58 +00:00
Jeremy Wharton
ad13cb2bf5 web/satellite, satellite/analytics: add UI type to analytics properties
This change incorporates the UI type (legacy or Vuetify) in the
properties attached to analytics events originating from the satellite
UI.

Resolves #6363

Change-Id: Ie3627bc24e4349407376e28460a5a830d211b47b
2023-10-13 11:43:54 +00:00
Wilfred Asomani
c9421d11e7 satellite/admin: add endpoints to violation freeze/unfreeze users
This change adds two new admin endpoints to freeze users for ToS
violation and to remove them from that state,

Issue: https://github.com/storj/storj-private/issues/386

Change-Id: I49c922377c9cdb315ce2777fcd35dcad432b0539
2023-10-13 10:43:44 +00:00
Wilfred Asomani
594e63f13a satellite/accountfreeze: mark billing frozen users for deletion
This change modifies the billing freeze chore to set pending deletion
status to users who are still frozen after a grace period.
It also modifies the chore to skip already deleted users.

Issue: https://github.com/storj/storj/issues/6303
https://github.com/storj/storj-private/issues/453

Change-Id: I4d0e7dd904463e99424372dc9ac81b71c0bc6e28
2023-10-12 20:25:23 +00:00
Wilfred Asomani
ee2b6e66de web/satellite/vuetify-poc: improve team page
This change makes minor improvements to the vuetify team page.

Issue: https://github.com/storj/storj/issues/6376

Change-Id: Iedd0e1f47d8075f13b424da65d719057eade14ed
2023-10-12 17:49:55 +00:00
Márton Elek
db3578d9ba satellite: durability rangeloop observer for monitoring risks
Change-Id: I92805fcc6e7c1bbe0f42bbf849d22f9908fedadb
2023-10-12 16:32:30 +00:00
Tome Boshevski
863c96b771
web/satellite: update signup pages plus logo (#6409)
* web/satellite: update signup pages plus logo

Update designs of the signup, login, forgot password, reset password, confirm email pages. Make Business as default on signup page per marketing request. Plus update the logo to remove DCS.
2023-10-12 18:05:44 +02:00
Vitalii
f41e117918 web/satellite/vuetify-poc: fix object browser pagination inside folders
Reflects the same changes made for main app.
https://review.dev.storj.io/c/storj/storj/+/11313

Change-Id: I8e0a2a64b1adfdf411c3044890e85928d592259d
2023-10-12 14:35:09 +00:00
Clement Sam
db7c6d38e5 storagenode/orders: archive unsent order for untrusted satellite
The order service still tries to settle orders at all instances even
when the satellite is marked as untrusted by the trust service, which
will always fail because the trust cache no longer has record of the
URL of the satellite, and it will keep retrying.

This leaves a lot of "satellite is untrusted" errors in the logs.
There has been several complaints on the forum because this was
happening a lot for the stefanlite and I expect it will be the
same issue for the decommisioned satellites US2 and EUN-1 once
the forget-satellite command is run to clean up the satellite.

This change allows the order service to archive unsent orders available
for any untrusted satellite, and will not attempt to settle the order.

https://github.com/storj/storj/issues/6262

Change-Id: If0f7f1783587cd18fab8917d45948f22df5b1dcf
2023-10-12 13:48:07 +00:00
Jeremy Wharton
4e0f062cb5 web/satellite/vuetify-poc: support PDF previewing
This change allows supported platforms to preview PDFs in the object
browser.

Resolves #6397

Change-Id: I6078914f0ddf5f620514e27e51d24ba8c11a2786
2023-10-12 12:00:59 +00:00
Vitalii
7038ddef7f web/satellite/vuetify-poc: fixed styling of stripe input
Made stripe input look more fancy on Payment Methods tab of Billing screen.

Issue:
https://github.com/storj/storj/issues/6399

Change-Id: I69497191b5c7a93905cdd4660ef733f17b4a2854
2023-10-12 13:44:15 +03:00
Michal Niewrzal
e3e303754b satellite/repair/checker: optimize processing, part 2
Optimizing collecting monkit metrics:
* initialize metrics once at the begining
* avoid using string in map for getting stats structs per redundancy

Benchmark results (compared against part 1 optimization change):
name                                       old time/op    new time/op    delta
RemoteSegment/Cockroach/healthy_segment-8    31.4µs ± 6%    21.7µs ± 8%  -30.73%  (p=0.008 n=5+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/healthy_segment-8    10.2kB ± 0%     7.4kB ± 0%  -27.03%  (p=0.008 n=5+5)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/healthy_segment-8       250 ± 0%       150 ± 0%  -40.00%  (p=0.008 n=5+5)

Change-Id: Ie09476eb469a4d6c09e52550c8ba92b3b4b34271
2023-10-12 10:02:53 +02:00
Michal Niewrzal
de4559d862 satellite/repair/checker: optimize processing, part 1
Optimization by reusing more slices.

Benchmark result:
name                                       old time/op    new time/op    delta
RemoteSegment/healthy_segment-8    33.2µs ± 1%    31.4µs ± 6%   -5.49%  (p=0.032 n=4+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/healthy_segment-8    15.9kB ± 0%    10.2kB ± 0%  -35.92%  (p=0.008 n=5+5)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/healthy_segment-8       280 ± 0%       250 ± 0%  -10.71%  (p=0.008 n=5+5)

Change-Id: I60462169285462dee6cd16d4f4ce1f30fb6cdfdf
2023-10-11 15:50:29 +00:00
Artur M. Wolff
3019471514 {Jenkinsfile, Makefile}: update Go to 1.21.3
Change-Id: Ia014ff4fa622903431789d0dc7e93044015ecaa9
2023-10-11 10:47:04 +02:00
Márton Elek
9186365507 storagenode/pieces: more granular io and hashing statistic
This patch adds two new monkit metric:
 * piece_writer_io: the sum of the time, which is spent with io.Write during a piece upload (excluding the fs sync of the commit)
 * piece_writer_hash: the sum of the time, which is spent with hashing

The second is especially important. My storagenode (hosted on a cloud server) spend ~30 ms on hasing data, piece_write_io time is usually 5ms for me.

These metrics can help us to identify the reason of slownes on storagenode sides.

Both of these depend on the size of the piece. To make it more meaningfull without exploding the cardinality, I created a few size categories and classified the pieces based on these. Measurements shows that it can provide usefull results (>2MB uploads are usually 23-28 ms).

Change-Id: Ifa1c205a490046655bcc34891003e7b43ed9c0bc
2023-10-11 07:33:04 +00:00
Wilfred Asomani
6308da2cc0 satellite/{payment,console,analytics} extend freeze functionality for violation freeze
This change extends the account freeze functionality account for
violation freezes as well.
Also, debug level logs in the freeze chore have been changed to info.
It adds an analytics event for when an invoice is found that belongs to
a user frozen for violation.
And finally adds whether a user is frozen for violation to the
/account/freezestatus response.

Issue: https://github.com/storj/storj-private/issues/386

Change-Id: Id8e40282dc8fd8f242da52791ab8ddbbef3da2bc
2023-10-10 18:39:29 +00:00
Tome Boshevski
211659b9b0
web/satellite: update and optimize error page (#6391)
Use 3x smaller size background image and update styles. Update button.
2023-10-10 18:02:27 +02:00
Ivan Fraixedes
386a978310 private/apigen: Validate param types
The Go generator only supports certain types as query and path
parameters and it panics on any an unsupported type.

The Document and TypeScript generator don't have any validation for
them. TypeScript generator generates code that compiles, however, it
won't work properly with all the types not supported by the Go
generator.

Because it doesn't make sense that some types may work on the TypeScript
generator, while the Go generator doesn't, doing the validation in the
Param constructor is better because it reports the issue without having
to run the Go generator and it gives a more understanding panic message.

TypeScript generator generates code that works properly with all the
types supported by the Go generator, hence, there isn't any change int
he TypeScript generator in this commit.

Change-Id: I03085283942a98341726a1560f511a46540df9f5
2023-10-10 12:37:20 +00:00
Vitalii
0db898b0a8 web/satellite/vuetify-poc: tiny changes for project dashboard
Download card: added 'per month' to Limit value.
Coupon card: replaced 'Limit' label with 'Included free usage'.
Also, fixed chart tooltip positioning.

Issue:
https://github.com/storj/storj/issues/6378

Change-Id: I6a4eb544cecaeabab6856acf6e272784f2a1c802
2023-10-10 11:41:28 +00:00
Jeremy Wharton
b6b9cccb72 web/satellite/vuetify-poc: object browser improvements
- Object preview can be opened through the object row actions menu.
- Previewed objects no longer shift vertically when transitioning.
- Previewed objects no longer display the next object's preview when
  transitioning.
- The download notification text has been changed from "Success" to
  "Download Started".
- Clicking object browser breadcrumbs no longer redirects to the
  all projects dashboard.
- Upload progress snackbar:
  - The expanded and collapsed widths are now the same.
  - Clicking an item opens the object preview for it.
  - The "Uploading" tooltip position has been moved to the left so that
    it doesn't block the cancel button.

Resolves #6379

Change-Id: Ic1f5cc7948ffa62dc0bce488b61f6d5e121c77b9
2023-10-10 08:32:27 +00:00
Jeremy Wharton
99ba88ae2f web/satellite/vuetify-poc: allow adding coupons
This change allows users to apply coupon codes via the billing
page's overview section.

Resolves #6393

Change-Id: I6f973f5293362efa8cd76149dc1b43468b49b2c9
2023-10-09 22:53:10 +00:00
Jeremy Wharton
dd3779a623 web/satellite/vuetify-poc: invite through all projects dashboard
This change allows project owners to invite members through the all
projects dashboard.

Resolves #6394

Change-Id: Id36d21432ab7b18532679e900d3e00c52fa21fc9
2023-10-09 20:32:54 +00:00