Commit Graph

8670 Commits

Author SHA1 Message Date
Márton Elek
f40baf8629
go.mod: bump dependencies (private,uplink,common)
Change-Id: I6c55735b45cadaf36697eff53e78b5b09afe9dea
2023-09-06 13:28:22 +02:00
Clement Sam
9ab934e2ae storagenode/piecestore: implement trash recovery for download requests
This change allows a node to look for a piece in the trash when
serving a download request.

If the piece is found in the trash, it restores it to the blobs
directory and continue to serve the request as expected.

Resolves https://github.com/storj/storj/issues/6145

Change-Id: Ibfa3c0b4954875fa977bc995fc4dd2705ca3ce42
2023-09-05 23:04:21 +00:00
Wilfred Asomani
3f1ea4a0b9 web/satellite/vuetify-poc: list invoices
This change lists invoices on the vuetifypoc billing history tab.
It also removes invoice filtering on the main app.

Issue: https://github.com/storj/storj/issues/6099

Change-Id: Id4cc2db003a0208775ddaefc87abf26f4b05106c
2023-09-05 15:01:16 +00:00
Wilfred Asomani
775db4aa3c satellite/payments: filter out draft invoices
This change filters the list of invoices sent to the frontend to only
contain open or paid invoices.

Change-Id: I9ac2640a7587a76b0baf46d941f799e742aa2b3b
2023-09-05 14:22:30 +00:00
Jeremy Wharton
4e3e31a425 satellite/console/consoleweb/consoleapi: refactor api requests in tests
This change refactors the way requests are sent in console API tests,
placing identical logic in a dedicated function to reduce code
duplication.

Change-Id: I7a5ac42d8d68a3fd9a9f8b9d61775659234e883f
2023-09-05 11:17:02 +00:00
Wilfred Asomani
2d18f43b6a web/satellite/vuetify-poc: show file previews
This change enables files to be previewed.

Issue: https://github.com/storj/storj/issues/6108

Change-Id: I3045950281822620de96b86e0180eb0d24e2d629
2023-09-05 10:35:31 +00:00
Márton Elek
e2006d821c satellite/overlay: change Reliable and KnownReliable
as GetParticipatingNodes and GetNodes, respectively.

We now want these functions to include offline and suspended nodes as
well, so that we can force immediate repair when pieces are out of
placement or in excluded countries. With that change, the old names no
longer made sense.

Change-Id: Icbcbad43dbde0ca8cbc80a4d17a896bb89b078b7
2023-09-02 23:34:50 +00:00
Cameron
6896241933 satellite/console: fix unverified invite email
When checking if invited user is unverified, initialize oldest
with unverified row rather than empty User, because empty User
CreatedAt is zero, so no real user could be created earlier.

Change-Id: I74dd8f7fc82951cbb61071632a74b1a9443b41fe
2023-09-01 15:11:58 -04:00
Michal Niewrzal
c9591e9754 satellite/metainfo: better metabase errors handling
Some errors were returned as metabase errors, not pure drpc
errors because of how rpcstatus.Code method is working. Status
code was returned for errors like metabase context canceled but
we would like to not leak our internals to the client.

Change-Id: I3f0194755f8d7359b1e3d342fa3be3d984019ecb
2023-09-01 11:32:10 +00:00
Moby von Briesen
b4c95a3b28 private/apigen/tsgen: Set query params better
The old way did not properly handle escaping, e.g. if the value of a
query param contained `&` or `=` inside it. By using
url.searchParams.set, we can safely add these types of arguments to the
path.

Change-Id: I62d3883b14f9d5a517e4a3d58f019014b46fd1b4
2023-08-31 20:09:29 -04:00
Jeremy Wharton
28d498f91d web/satellite/vuetify-poc: allow file browser downloads
This change allows files to be downloaded from within the file browser
of the Vuetify project.

Resolves #6107

Change-Id: I0ac0384711baccb99c0a6d382fe96f318290789b
2023-08-31 22:41:52 +00:00
Jeremy Wharton
b671641a28 satellite/console: update CSP to include storjapi.io
This change updates our content security policy to include the domain
storjapi.io and all of its subdomains.

References #6188

Change-Id: I6f3073bc32aa99626c54caf00bf07d2253ccbb8f
2023-08-31 22:02:45 +00:00
Márton Elek
c202929413
satellite/nodeselection: rename (NodeFilter).MatchInclude to Match
As I learned, the `Include` supposed to communicate that some internal change also "included" to the filters during the check -> filters might be stateful.

But it's not the case any more after 552242387, where we removed the only one stateful filter.

Change-Id: I7c36ddadb2defbfa3b6b67bcc115e4427ba9e083
2023-08-31 16:17:52 +02:00
Wilfred Asomani
dcc4bd0d10 satellite/{console,payments}: freeze/warn storjscan users
This change enables the freezing/warning of users who use storjscan.

Issue: https://github.com/storj/storj/issues/6164

Change-Id: I7b00ee09d6527b3818b72326e9065c82ef5a2ac8
2023-08-31 13:22:21 +00:00
Márton Elek
ca0ea50cba satellite/overlay: remove/deprecate NodeSelectionCache.Disabled
Once uppon a time, at the dawn of the implementation of Storj, when all the nodes are read from the database directly, every time.

After a while --  due to performance reasons -- it has been changed for upload and download: where all the nodes are read for a short period of time, and used from memory.

This is the version which was improved recently to support advanced node selections using placement.

But stil we have an old configuration value `service.config.NodeSelectionCache.Disabled`, and the db based implementation: `service.FindStorageNodesWithPreferences(ctx, req, &service.config.Node)`.

For safety, we need to remove this option, to make sure that we use the cache, which has the advanced features.

This patch was supposed to be a very small one (just removing a method and a config: https://review.dev.storj.io/c/storj/storj/+/11074/1/satellite/overlay/service.go), but it turned out that we need to update a lot of unit tests.

These unit tests used the old implementation (which is not used in production any more).

The tests which used both implementation are just updated to use only the new one
The tests which used only the old implementation are refactored (but keeping the test cases).
Using real unit tests (without DB, working on OSX, fast)

Closes https://github.com/storj/storj/issues/6217

Change-Id: I023f92c7e34235665cf8474513e67b2fcc4763eb
2023-08-31 09:46:29 +00:00
Jeremy Wharton
00194f54a2 web/satellite: show limit notifications to paid tier users
This change causes paid tier users to see notifications in the project
dashboard when their usage is approaching or has reached their maximum
or custom usage limits.

Change-Id: I7b68fcdd7d62797b6b26869e109cfb0b193fdddb
2023-08-31 07:31:09 +00:00
Wilfred Asomani
31ec421299 satellite/db: optimize project usage query
This change addresses an issue where the /charges endpoint will take a
while to respond due to a project having a large number of buckets.
The method and queries involved have been optimized and benchmarks show
a performance improvement.

test name                            old ms/op  new ms/op
Postgres/sum_all_partner_usages          3.659      1.101
Postgres/individual_partner_usages        3.74      1.299
Cockroach/sum_all_partner_usages         7.201      2.872
Cockroach/individual_partner_usages      7.247      2.852

Issue: https://github.com/storj/storj-private/issues/277

Change-Id: Ia5082a2e1c3e91120a9db7b01c18847fe04574fe
2023-08-30 22:08:11 +00:00
Jeremy Wharton
0c9c37875d web/satellite/vuetify-poc: unmock file browser table entries
This change shows real file entries in the file browser table,
replacing the mock data. Sorting, searching, and folder navigation have
been implemented.

Resolves #6199

Change-Id: I7360879d2e26605489c20f9d094c3f231fee49cd
2023-08-30 21:12:55 +00:00
Jeremy Wharton
ebfbbca1be web/satellite/vuetify-poc: pull latest changes from internal repo
The changes as of storj/vuetify-storj@2312936 have been pulled into the
Vuetify project.

Change-Id: Ia45d3d0832727d3ca7b9ee9b3dce5949a79a5d31
2023-08-30 20:34:13 +00:00
Wilfred Asomani
fecaa6a71a web/satellite/vuetify-poc: enable claiming wallets
This change adds the ability to claim STORJ wallets and display token
balance.

Issue: https://github.com/storj/storj/issues/6096

Change-Id: Ifc89b586c0e3ed876905ff0a5b270e718cbb689c
2023-08-30 14:33:35 +00:00
Michal Niewrzal
67371c43bd satellite/metainfo: enable UsePendingObjectsTable by project
This small feature will give us ability to test pending_objects table
without enabling it globally.

Change-Id: I802f45987ad329f94adfc0f02957c802b21d8251
2023-08-30 15:44:29 +02:00
Michal Niewrzal
780c0e0b35 satellite/metainfo: adjust ListPendingObjectStreams to pending_objects
table

New method IteratePendingObjectsByKeyNew is used to provide results for
metainfo.ListPendingObjectStreams. This endpoint is used to list
pending objects with the same object key. In this case to support
both tables (objects, pending_objects) we need to do one query per table
and merge results.

Because existing metainfo protobuf API is missing some fields to have
proper listing cursor we are not able to make ListPendingObjectStreams
correct for returning more than single page. We need to fix it
separately.

With this change also turns out that approach to merge results from
listing objects for ListObjects method was wrong and this change is also
fixing this problem.

Handling both tables will be removed at some point and only
pending_objects will be used to look for results.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I8a88a6f885ad529704e6c032f1d97926123c2909
2023-08-30 13:35:54 +00:00
Wilfred Asomani
6e3da022e0 web/satellite: add pagination to billing history
This change adds pagination to the billing history table. It uses the
new invoice-history endpoint since we only list invoices in this table.

Issue: https://github.com/storj/storj/issues/5479

Change-Id: I192d58503434203808a23a7c18e8d1feb6afc73f
2023-08-30 03:19:31 +00:00
Vitalii
614d213432 web/satellite: fix duplicate satellite selection on login and forgot password pages
Remove duplicate satellite in selection dropdown on login and forgot password pages.

Issue:
https://github.com/storj/storj/issues/6121

Change-Id: I81bd0c2c29c695ed9ed7f0d648311448cdc9bf26
2023-08-30 01:17:14 +00:00
Cameron
a5b1c0432f satellite/console: send unverified user activation link in project invite
When an unverified user is sent a project invitation it contains a
registration link currently. Instead, send an activation link.

github issue: https://github.com/storj/storj/issues/6033

Change-Id: I54b88de8347a2532f7a85372c0c5e4df4bf4eb38
2023-08-29 12:54:09 -04:00
Vitalii
d6e0987dd9 web/satellite: added info message to sharing modals
Add notification explaining that the share will be public to anyone with the link.
Remove the cancel button as it may be interpreted as cancelling or revoking the shared access.
Make the copy link full width block button.

Issue:
https://github.com/storj/storj/issues/6213

Change-Id: I7b9580b3d8135802c36af8e68f46630b499ab110
2023-08-29 12:55:26 +03:00
Wilfred Asomani
6219aba40c satellite/{console,consoleweb,consoleapi}: add new endpoint for paged invoices
This change adds a new endpoint for listing invoices for billing history
This endpoint will replace the billing-history endpoint used on the
front end since were only interested in listing invoices.

Issue: https://github.com/storj/storj/issues/5479

Change-Id: I4730f5dc497245c6730e60b7f9986554479d1d3b
2023-08-28 23:24:12 +00:00
Vitalii
f0829d5961 web/satellite: use stripe as ES module
Start using @stripe/stripe-js lib (ES module) instead of regular stripe dependency.
Use strict typing for stripe commands/events.
This lib makes us able to modify stripe input styling in the future.

Change-Id: Iaba4f32a42e87edc85a4fbad82e5107c21bf19b6
2023-08-28 22:45:52 +00:00
Wilfred Asomani
f1e8cdfe3e web/satellite/vuetify-poc: add project usages to billing
This change adds project usage and cost per project to the vuetify app.

Issue: https://github.com/storj/storj/issues/6117

Change-Id: I8921aacb6bb24b41794008100ea6e52deed76b60
2023-08-28 22:05:00 +00:00
Vitalii
4964ca5ffb web/satellite: don't show CTAs on limit cards if user is not project owner
Hide CTAs on limit cards if user is not the owner of selected project.
We do this because user can't update limits if they are not the owner of the project.

Issue:
https://github.com/storj/storj/issues/6214

Change-Id: Ib6b564cd6afc1b4bed08ee9b26108f803f0ffb89
2023-08-28 21:16:42 +00:00
Michal Niewrzal
c010e37374 satellite/metainfo: adjust ListObjects to use pending_objects table
Adjust metainfo.ListObjects method to use IteratePendingObjects to
support new pending_objects table. New method will be used only when
we are listing pending objects.

Because until objects table will be free from pending objects we can
have results in both tables we are merging listing results. This also
means that in some (rare?) cases we may return more results than
specified listing limit. This situation is temporary.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I06389145e5d916c532dfdbd3dcc9ef68ef70e515
2023-08-28 16:22:54 +00:00
Egon Elbre
273ebd61d7 private/apigen/example: make it nicer
Change-Id: I1bd779090a902ed2b99b3993dc7cf61fc250f10f
2023-08-28 15:13:46 +00:00
Ivan Fraixedes
2d8f396eeb private/apigen: Make API base path configurable
Previously the base path for the API was hardcoded to `/api` and the
specified version.

This was not obvious that the generated code was setting that base path
and it was not flexible for serving the API under a different path than
`/api`.

We will likely need to set a different base path if we pretend to serve
the new back office API that we are going to implement alongside the
current admin API until the new back office is fully implemented and
verified that works properly.

This commit also fix add the base path of the endpoints to the
documentation because it was even more confusing for somebody that wants
to use the API having to find out them through looking to the generated
code.

Change-Id: I6efab6b6f3d295129d6f42f7fbba8c2dc19725f4
2023-08-28 14:35:01 +00:00
Cameron
ec42fdae6d web/satellite: add bucket details dialog
issue https://github.com/storj/storj/issues/6115

Change-Id: I4dd3b93bd43f4871cd28ab595bace9e8ce59b7c2
2023-08-28 13:09:03 +00:00
Michal Niewrzal
b26df035f9 satellite/metainfo: tests for new GetBucketLocation method
Change-Id: I809ebab51606aa9e55dff3c40ef2e865caf06924
2023-08-28 11:49:23 +00:00
Artur M. Wolff
37d6df23fa satellite: implement metainfo.GetBucketLocation endpoint
Updates storj/storj-private#408
Updates storj/storj-private#409

Change-Id: Idaaca74b4a5c9c7907d095e0a3a5f29e52843ce6
2023-08-28 13:48:07 +02:00
Márton Elek
5c12a3406d satellite/nodeselection: improve annotation composability
We would like to make it easier to accept multiple annotations.

Examples:
```
country("GB") && annotation(...)
annotated(annotated(X,...),...)
```

Change-Id: I92e622e8b985b314dadddf83b17976c245eb2069
2023-08-28 09:27:04 +00:00
Jeremy Wharton
2cdc1a973f satellite/console: fix project creation race condition
This change fixes an issue where the project limit could be exceeded if
multiple project creation requests were sent sufficiently close to one
another. This could also be used to bypass project name duplication
checking.

Change-Id: I61cde7abaf25dedc5601c6870275de9938d7b949
2023-08-25 17:18:34 +00:00
Ivan Fraixedes
d34c1b6825 private/apigen: Generate docs & typescript for example
Generate the documentation and the typescript code for the example of
that we have to generate an API through the API generator.

The JSON Go struct field tags are needed because the typescript
generator require them, otherwise it panics.

The test had to be adjusted according to match the tags so Go consider
the structs the same type, otherwise, it doesn't compile.

Change-Id: I3e4ebff9174885c50ca2058b86b7ec60e025945c
2023-08-25 12:30:21 +00:00
Moby von Briesen
d76f059c55 private/apigen: Update generated doc links
Make the link more human-friendly - to contain the text of the group and
endpoint names.

Also link back to list of endpoints from each endpoint.

Change-Id: Ia3e2ebe20b58b5f60ecefe9d35fb8fd90dd4b4d7
2023-08-25 11:51:57 +00:00
Jeremy Wharton
61fe95c44a web/satellite/vuetify-poc: update access grant creation dialog
This change updates the access grant creation dialog to align with our
new designs. It also fixes an issue that would occur if the web worker
was initialized after the dialog was mounted.

Resolves #6169

Change-Id: Ic557766e6fcf57cc79c72e670a0e83c7eb2834ba
2023-08-25 09:11:58 +00:00
Wilfred Asomani
e44365d265 web/satellite: use fontsource for font loading
This change updates the vuetify app and the production app to load
fonts using the fontsource package.

Issue: https://github.com/storj/storj/issues/6200

Change-Id: I4b91a4d0dfcfc42f9f71ac03b31d1ef74c53e15d
2023-08-24 23:08:41 +00:00
Wilfred Asomani
36f8eeb272 web/satellite: fix broken preview link
This change fixes an issue where the gallery view will use wrong links
display object previews because they were encoded multiple times.

Issue: https://github.com/storj/customer-issues/issues/961

Change-Id: I498878e9beb927b812e40d4c7e5ae812cfa3554c
2023-08-24 22:30:03 +00:00
Jeremy Wharton
84e75d5994 satellite/satellitedb: remove subquery AOST in console db cleanup funcs
This change fixes an issue where the console DB cleanup chore was never
able to run when using a Cockroach database implementation because of
an inappropriate AS OF SYSTEM TIME clause in the relevant methods.

Resolves #6197

Change-Id: I8456b6df2128678e0eebeb416eb1a955cc9bd706
2023-08-24 21:51:55 +00:00
Wilfred Asomani
fe9f69a757 satellite/{consoleweb,consoleapi}: add cross user api tests
This change adds tests to ensure critical endpoints are not able to be
called by users for other users. It asserts that if cases like that
do happen, a 401 response will be sent.

Issue: https://github.com/storj/storj-private/issues/407

Change-Id: I70097a80f691a7d0fcb0bc5dbce8291144177720
2023-08-24 20:16:20 +00:00
Moby von Briesen
5c155752d2 satellite/analytics: Update lifecyclestage for personal users
Personal users, like business users, should now be classified with
a lifecycle stage of PQL ("product qualified lead") instead of "other"

Change-Id: Iff5139043da1c8e75559302320ff9ca43ea956e5
2023-08-24 09:47:26 -04:00
Jeremy Wharton
0070cd322a web/satellite/vuetify-poc: fix bucket deletion dialog
This change fixes an issue where the bucket deletion dialog wasn't
functional due to an invalid property value. This change also fixes an
issue where access grant worker errors were reported as being caused by
the bucket deletion dialog even when it wasn't open.

Change-Id: If2c2713857c4cc5c3c7ae60e431f5034e78c4c5f
2023-08-24 09:53:03 +00:00
Moby von Briesen
7fbabbcf16 private/apigen: Add list of endpoints with links
This change makes it easier for someone reading the documentation to see
a full list of supported endpoints, and have direct links to the
details.

Change-Id: I46e2f809cfa2760845898eaa3d99db9066d435ef
2023-08-24 02:45:46 +00:00
Sean Harvey
a0c69568b5 Makefile: add sha256sums checksum file with release binaries
Change-Id: I518e075445288f238104b801f89837e2d8d0d781
2023-08-23 20:43:12 +00:00
Moby von Briesen
5340a351b7 satellite/.../consoleapi/gen: Update README
Remove outdated information from the generated API readme, and add a
link to the generated documentation.

Change-Id: Icc098c81f235464344895d2195444044831aac63
2023-08-23 19:49:35 +00:00