Commit Graph

8409 Commits

Author SHA1 Message Date
Jeremy Wharton
b671641a28 satellite/console: update CSP to include storjapi.io
This change updates our content security policy to include the domain
storjapi.io and all of its subdomains.

References #6188

Change-Id: I6f3073bc32aa99626c54caf00bf07d2253ccbb8f
2023-08-31 22:02:45 +00:00
Márton Elek
c202929413
satellite/nodeselection: rename (NodeFilter).MatchInclude to Match
As I learned, the `Include` supposed to communicate that some internal change also "included" to the filters during the check -> filters might be stateful.

But it's not the case any more after 552242387, where we removed the only one stateful filter.

Change-Id: I7c36ddadb2defbfa3b6b67bcc115e4427ba9e083
2023-08-31 16:17:52 +02:00
Wilfred Asomani
dcc4bd0d10 satellite/{console,payments}: freeze/warn storjscan users
This change enables the freezing/warning of users who use storjscan.

Issue: https://github.com/storj/storj/issues/6164

Change-Id: I7b00ee09d6527b3818b72326e9065c82ef5a2ac8
2023-08-31 13:22:21 +00:00
Márton Elek
ca0ea50cba satellite/overlay: remove/deprecate NodeSelectionCache.Disabled
Once uppon a time, at the dawn of the implementation of Storj, when all the nodes are read from the database directly, every time.

After a while --  due to performance reasons -- it has been changed for upload and download: where all the nodes are read for a short period of time, and used from memory.

This is the version which was improved recently to support advanced node selections using placement.

But stil we have an old configuration value `service.config.NodeSelectionCache.Disabled`, and the db based implementation: `service.FindStorageNodesWithPreferences(ctx, req, &service.config.Node)`.

For safety, we need to remove this option, to make sure that we use the cache, which has the advanced features.

This patch was supposed to be a very small one (just removing a method and a config: https://review.dev.storj.io/c/storj/storj/+/11074/1/satellite/overlay/service.go), but it turned out that we need to update a lot of unit tests.

These unit tests used the old implementation (which is not used in production any more).

The tests which used both implementation are just updated to use only the new one
The tests which used only the old implementation are refactored (but keeping the test cases).
Using real unit tests (without DB, working on OSX, fast)

Closes https://github.com/storj/storj/issues/6217

Change-Id: I023f92c7e34235665cf8474513e67b2fcc4763eb
2023-08-31 09:46:29 +00:00
Jeremy Wharton
00194f54a2 web/satellite: show limit notifications to paid tier users
This change causes paid tier users to see notifications in the project
dashboard when their usage is approaching or has reached their maximum
or custom usage limits.

Change-Id: I7b68fcdd7d62797b6b26869e109cfb0b193fdddb
2023-08-31 07:31:09 +00:00
Wilfred Asomani
31ec421299 satellite/db: optimize project usage query
This change addresses an issue where the /charges endpoint will take a
while to respond due to a project having a large number of buckets.
The method and queries involved have been optimized and benchmarks show
a performance improvement.

test name                            old ms/op  new ms/op
Postgres/sum_all_partner_usages          3.659      1.101
Postgres/individual_partner_usages        3.74      1.299
Cockroach/sum_all_partner_usages         7.201      2.872
Cockroach/individual_partner_usages      7.247      2.852

Issue: https://github.com/storj/storj-private/issues/277

Change-Id: Ia5082a2e1c3e91120a9db7b01c18847fe04574fe
2023-08-30 22:08:11 +00:00
Jeremy Wharton
0c9c37875d web/satellite/vuetify-poc: unmock file browser table entries
This change shows real file entries in the file browser table,
replacing the mock data. Sorting, searching, and folder navigation have
been implemented.

Resolves #6199

Change-Id: I7360879d2e26605489c20f9d094c3f231fee49cd
2023-08-30 21:12:55 +00:00
Jeremy Wharton
ebfbbca1be web/satellite/vuetify-poc: pull latest changes from internal repo
The changes as of storj/vuetify-storj@2312936 have been pulled into the
Vuetify project.

Change-Id: Ia45d3d0832727d3ca7b9ee9b3dce5949a79a5d31
2023-08-30 20:34:13 +00:00
Wilfred Asomani
fecaa6a71a web/satellite/vuetify-poc: enable claiming wallets
This change adds the ability to claim STORJ wallets and display token
balance.

Issue: https://github.com/storj/storj/issues/6096

Change-Id: Ifc89b586c0e3ed876905ff0a5b270e718cbb689c
2023-08-30 14:33:35 +00:00
Michal Niewrzal
67371c43bd satellite/metainfo: enable UsePendingObjectsTable by project
This small feature will give us ability to test pending_objects table
without enabling it globally.

Change-Id: I802f45987ad329f94adfc0f02957c802b21d8251
2023-08-30 15:44:29 +02:00
Michal Niewrzal
780c0e0b35 satellite/metainfo: adjust ListPendingObjectStreams to pending_objects
table

New method IteratePendingObjectsByKeyNew is used to provide results for
metainfo.ListPendingObjectStreams. This endpoint is used to list
pending objects with the same object key. In this case to support
both tables (objects, pending_objects) we need to do one query per table
and merge results.

Because existing metainfo protobuf API is missing some fields to have
proper listing cursor we are not able to make ListPendingObjectStreams
correct for returning more than single page. We need to fix it
separately.

With this change also turns out that approach to merge results from
listing objects for ListObjects method was wrong and this change is also
fixing this problem.

Handling both tables will be removed at some point and only
pending_objects will be used to look for results.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I8a88a6f885ad529704e6c032f1d97926123c2909
2023-08-30 13:35:54 +00:00
Wilfred Asomani
6e3da022e0 web/satellite: add pagination to billing history
This change adds pagination to the billing history table. It uses the
new invoice-history endpoint since we only list invoices in this table.

Issue: https://github.com/storj/storj/issues/5479

Change-Id: I192d58503434203808a23a7c18e8d1feb6afc73f
2023-08-30 03:19:31 +00:00
Vitalii
614d213432 web/satellite: fix duplicate satellite selection on login and forgot password pages
Remove duplicate satellite in selection dropdown on login and forgot password pages.

Issue:
https://github.com/storj/storj/issues/6121

Change-Id: I81bd0c2c29c695ed9ed7f0d648311448cdc9bf26
2023-08-30 01:17:14 +00:00
Cameron
a5b1c0432f satellite/console: send unverified user activation link in project invite
When an unverified user is sent a project invitation it contains a
registration link currently. Instead, send an activation link.

github issue: https://github.com/storj/storj/issues/6033

Change-Id: I54b88de8347a2532f7a85372c0c5e4df4bf4eb38
2023-08-29 12:54:09 -04:00
Vitalii
d6e0987dd9 web/satellite: added info message to sharing modals
Add notification explaining that the share will be public to anyone with the link.
Remove the cancel button as it may be interpreted as cancelling or revoking the shared access.
Make the copy link full width block button.

Issue:
https://github.com/storj/storj/issues/6213

Change-Id: I7b9580b3d8135802c36af8e68f46630b499ab110
2023-08-29 12:55:26 +03:00
Wilfred Asomani
6219aba40c satellite/{console,consoleweb,consoleapi}: add new endpoint for paged invoices
This change adds a new endpoint for listing invoices for billing history
This endpoint will replace the billing-history endpoint used on the
front end since were only interested in listing invoices.

Issue: https://github.com/storj/storj/issues/5479

Change-Id: I4730f5dc497245c6730e60b7f9986554479d1d3b
2023-08-28 23:24:12 +00:00
Vitalii
f0829d5961 web/satellite: use stripe as ES module
Start using @stripe/stripe-js lib (ES module) instead of regular stripe dependency.
Use strict typing for stripe commands/events.
This lib makes us able to modify stripe input styling in the future.

Change-Id: Iaba4f32a42e87edc85a4fbad82e5107c21bf19b6
2023-08-28 22:45:52 +00:00
Wilfred Asomani
f1e8cdfe3e web/satellite/vuetify-poc: add project usages to billing
This change adds project usage and cost per project to the vuetify app.

Issue: https://github.com/storj/storj/issues/6117

Change-Id: I8921aacb6bb24b41794008100ea6e52deed76b60
2023-08-28 22:05:00 +00:00
Vitalii
4964ca5ffb web/satellite: don't show CTAs on limit cards if user is not project owner
Hide CTAs on limit cards if user is not the owner of selected project.
We do this because user can't update limits if they are not the owner of the project.

Issue:
https://github.com/storj/storj/issues/6214

Change-Id: Ib6b564cd6afc1b4bed08ee9b26108f803f0ffb89
2023-08-28 21:16:42 +00:00
Michal Niewrzal
c010e37374 satellite/metainfo: adjust ListObjects to use pending_objects table
Adjust metainfo.ListObjects method to use IteratePendingObjects to
support new pending_objects table. New method will be used only when
we are listing pending objects.

Because until objects table will be free from pending objects we can
have results in both tables we are merging listing results. This also
means that in some (rare?) cases we may return more results than
specified listing limit. This situation is temporary.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I06389145e5d916c532dfdbd3dcc9ef68ef70e515
2023-08-28 16:22:54 +00:00
Egon Elbre
273ebd61d7 private/apigen/example: make it nicer
Change-Id: I1bd779090a902ed2b99b3993dc7cf61fc250f10f
2023-08-28 15:13:46 +00:00
Ivan Fraixedes
2d8f396eeb private/apigen: Make API base path configurable
Previously the base path for the API was hardcoded to `/api` and the
specified version.

This was not obvious that the generated code was setting that base path
and it was not flexible for serving the API under a different path than
`/api`.

We will likely need to set a different base path if we pretend to serve
the new back office API that we are going to implement alongside the
current admin API until the new back office is fully implemented and
verified that works properly.

This commit also fix add the base path of the endpoints to the
documentation because it was even more confusing for somebody that wants
to use the API having to find out them through looking to the generated
code.

Change-Id: I6efab6b6f3d295129d6f42f7fbba8c2dc19725f4
2023-08-28 14:35:01 +00:00
Cameron
ec42fdae6d web/satellite: add bucket details dialog
issue https://github.com/storj/storj/issues/6115

Change-Id: I4dd3b93bd43f4871cd28ab595bace9e8ce59b7c2
2023-08-28 13:09:03 +00:00
Michal Niewrzal
b26df035f9 satellite/metainfo: tests for new GetBucketLocation method
Change-Id: I809ebab51606aa9e55dff3c40ef2e865caf06924
2023-08-28 11:49:23 +00:00
Artur M. Wolff
37d6df23fa satellite: implement metainfo.GetBucketLocation endpoint
Updates storj/storj-private#408
Updates storj/storj-private#409

Change-Id: Idaaca74b4a5c9c7907d095e0a3a5f29e52843ce6
2023-08-28 13:48:07 +02:00
Márton Elek
5c12a3406d satellite/nodeselection: improve annotation composability
We would like to make it easier to accept multiple annotations.

Examples:
```
country("GB") && annotation(...)
annotated(annotated(X,...),...)
```

Change-Id: I92e622e8b985b314dadddf83b17976c245eb2069
2023-08-28 09:27:04 +00:00
Jeremy Wharton
2cdc1a973f satellite/console: fix project creation race condition
This change fixes an issue where the project limit could be exceeded if
multiple project creation requests were sent sufficiently close to one
another. This could also be used to bypass project name duplication
checking.

Change-Id: I61cde7abaf25dedc5601c6870275de9938d7b949
2023-08-25 17:18:34 +00:00
Ivan Fraixedes
d34c1b6825 private/apigen: Generate docs & typescript for example
Generate the documentation and the typescript code for the example of
that we have to generate an API through the API generator.

The JSON Go struct field tags are needed because the typescript
generator require them, otherwise it panics.

The test had to be adjusted according to match the tags so Go consider
the structs the same type, otherwise, it doesn't compile.

Change-Id: I3e4ebff9174885c50ca2058b86b7ec60e025945c
2023-08-25 12:30:21 +00:00
Moby von Briesen
d76f059c55 private/apigen: Update generated doc links
Make the link more human-friendly - to contain the text of the group and
endpoint names.

Also link back to list of endpoints from each endpoint.

Change-Id: Ia3e2ebe20b58b5f60ecefe9d35fb8fd90dd4b4d7
2023-08-25 11:51:57 +00:00
Jeremy Wharton
61fe95c44a web/satellite/vuetify-poc: update access grant creation dialog
This change updates the access grant creation dialog to align with our
new designs. It also fixes an issue that would occur if the web worker
was initialized after the dialog was mounted.

Resolves #6169

Change-Id: Ic557766e6fcf57cc79c72e670a0e83c7eb2834ba
2023-08-25 09:11:58 +00:00
Wilfred Asomani
e44365d265 web/satellite: use fontsource for font loading
This change updates the vuetify app and the production app to load
fonts using the fontsource package.

Issue: https://github.com/storj/storj/issues/6200

Change-Id: I4b91a4d0dfcfc42f9f71ac03b31d1ef74c53e15d
2023-08-24 23:08:41 +00:00
Wilfred Asomani
36f8eeb272 web/satellite: fix broken preview link
This change fixes an issue where the gallery view will use wrong links
display object previews because they were encoded multiple times.

Issue: https://github.com/storj/customer-issues/issues/961

Change-Id: I498878e9beb927b812e40d4c7e5ae812cfa3554c
2023-08-24 22:30:03 +00:00
Jeremy Wharton
84e75d5994 satellite/satellitedb: remove subquery AOST in console db cleanup funcs
This change fixes an issue where the console DB cleanup chore was never
able to run when using a Cockroach database implementation because of
an inappropriate AS OF SYSTEM TIME clause in the relevant methods.

Resolves #6197

Change-Id: I8456b6df2128678e0eebeb416eb1a955cc9bd706
2023-08-24 21:51:55 +00:00
Wilfred Asomani
fe9f69a757 satellite/{consoleweb,consoleapi}: add cross user api tests
This change adds tests to ensure critical endpoints are not able to be
called by users for other users. It asserts that if cases like that
do happen, a 401 response will be sent.

Issue: https://github.com/storj/storj-private/issues/407

Change-Id: I70097a80f691a7d0fcb0bc5dbce8291144177720
2023-08-24 20:16:20 +00:00
Moby von Briesen
5c155752d2 satellite/analytics: Update lifecyclestage for personal users
Personal users, like business users, should now be classified with
a lifecycle stage of PQL ("product qualified lead") instead of "other"

Change-Id: Iff5139043da1c8e75559302320ff9ca43ea956e5
2023-08-24 09:47:26 -04:00
Jeremy Wharton
0070cd322a web/satellite/vuetify-poc: fix bucket deletion dialog
This change fixes an issue where the bucket deletion dialog wasn't
functional due to an invalid property value. This change also fixes an
issue where access grant worker errors were reported as being caused by
the bucket deletion dialog even when it wasn't open.

Change-Id: If2c2713857c4cc5c3c7ae60e431f5034e78c4c5f
2023-08-24 09:53:03 +00:00
Moby von Briesen
7fbabbcf16 private/apigen: Add list of endpoints with links
This change makes it easier for someone reading the documentation to see
a full list of supported endpoints, and have direct links to the
details.

Change-Id: I46e2f809cfa2760845898eaa3d99db9066d435ef
2023-08-24 02:45:46 +00:00
Sean Harvey
a0c69568b5 Makefile: add sha256sums checksum file with release binaries
Change-Id: I518e075445288f238104b801f89837e2d8d0d781
2023-08-23 20:43:12 +00:00
Moby von Briesen
5340a351b7 satellite/.../consoleapi/gen: Update README
Remove outdated information from the generated API readme, and add a
link to the generated documentation.

Change-Id: Icc098c81f235464344895d2195444044831aac63
2023-08-23 19:49:35 +00:00
Ivan Fraixedes
b0d072270b private/apigen: Add comment about panics
Add a comment in 2 methods about they panic as it's specified for the
`MustWriteDocs`.

Change-Id: I15d5fa420af37318d828bc633bed62988f8b207b
2023-08-23 18:50:37 +00:00
Michal Niewrzal
95a5cfe647 satellite/buckets: handle bucket exists better
In some rare cases when two entities are trying to create the same
bucket at the same time it's possible that we will return internal
error instead of `bucket already exists`. It's because we are not
handling correctly DB error about constraint error. This change checks
if while inserting bucket into DB we got constraint error and propagate
correct error to metainfo API.

Change-Id: Ie6fd2c943b864b4ea7d71e4a162e74dc3510e386
2023-08-23 14:40:31 +00:00
Márton Elek
84ea80c1fd satellite/repair/checker: respect autoExcludeSubnet anntation in checker rangedloop
This patch is a oneliner: rangedloop checker should check the subnets only if it's not turned off with placement annotation.
(see in satellite/repair/checker/observer.go).

But I didn't find any unit test to cover that part, so I had to write one, and I prefered to write it as a unit test not an integration test, which requires a mock repair queue (observer_unit_test.go mock.go).

Because it's small change, I also included a small change: creating a elper method to check if AutoExcludeSubnet annotation is defined

Change-Id: I2666b937074ab57f603b356408ef108cd55bd6fd
2023-08-23 13:45:09 +00:00
Márton Elek
4ccce11893
satellite/overlay: improve realistic placement rule test
10 --> node tag inclusion in raw format
11 --> same, but using same subnet is enabled
12 --> same as 11 but with US restrictions

Change-Id: I20792689e0caf5fe190f566a770d70c3b3824793
2023-08-23 13:56:35 +02:00
Jeremy Wharton
973b54365e web/satellite/vuetify-poc: add project creation dialog
This change implements a dialog box through which users can create
projects.

Resolves #6172

Change-Id: I99311f43ef49dc04bb852a6736771b42f26a3276
2023-08-22 19:45:35 +00:00
Moby von Briesen
b66fc6dcdb web/satellite: Update build script to build vuetify app
Also copy the necessary vuetify dist directory in the `Earthfile` so
that this app can be easily deployed to staging environments.

Change-Id: I8d91c52bb8f7c31fc3764efe60a071e21b399b46
2023-08-22 13:55:10 +00:00
Wilfred Asomani
516241e406 cmd,satellite: remove Graphql code and dependencies
This change removes unused GraphQL code. It also updates storj sim code
to use the GraphQL replacement HTTP endpoints and removes the GraphQL
dependency.

Issue: https://github.com/storj/storj/issues/6142

Change-Id: Ie502553706c4b1282cd883a9275ea7332b8fc92d
2023-08-22 12:23:14 +00:00
Wilfred Asomani
d10ce19f50 web/satellite: remove Graphql dependency
This change removes unused GraphQL code and dependencies.

Issue: https://github.com/storj/storj/issues/6142

Change-Id: Ib2346c7a034f26b46af1bc6113e007b6adcbdd51
2023-08-21 22:17:41 +00:00
Paul Willoughby
fdd4be80bf satellite/metainfo: increase default MaxEncryptedObjectKeyLength
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt.  Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.

Updates https://github.com/storj/gateway-mt/issues/335

Change-Id: I38a0fbb0843fd782aeadca85f9a202821421b5a2
2023-08-21 14:03:16 -06:00
Wilfred Asomani
9cf9721abe web/satellite: use create project http endpoint
This change uses the new POST projects endpoint in place of the GraphQL
createProject query.

Issue: https://github.com/storj/storj/issues/6195

Change-Id: I1776b8b0e8656d2f5fa4219df020615bcc0f2543
2023-08-21 17:30:38 +00:00
Jeremy Wharton
7f9317aa48 satellite/console: fix account creation race condition
This change fixes an issue where multiple unverified users with the
same email address could be created if registration requests were
sent sufficiently close to one another.

Resolves #6156

Change-Id: If8b1a145bcab842ace718119183de59947430463
2023-08-21 16:52:11 +00:00