Commit Graph

8445 Commits

Author SHA1 Message Date
Vitalii
2cf233ac23 web/satellite/vuetify-poc: hide nav sidebar by default for screens less than 1280px
On screen sizes where the sidebar overlaps with the content of the page, the sidebar should be collapsed by default.

Issue:
https://github.com/storj/storj/issues/6229

Change-Id: Ia0a91acd95519de27f9ff8f1ee90c6b8e7932266
2023-09-12 13:48:53 +03:00
Cameron
f9ab2f0de7 satellite/console: prefix oidc paths with /api/v0/
add /api/v0/ prefix to oidc paths to route requests to the api
pod with split UI infrastructure.

Change-Id: I1e7e691487ab1d4e84434d204d10b7f944ae8873
2023-09-11 18:12:39 +00:00
Egon Elbre
487f64e164 satellite/satellitedb,multinode/multinodedb: update to latest dbx
Change-Id: I500df6d0541706c3960d4560721c3783d0d049ff
2023-09-11 17:21:02 +00:00
Egon Elbre
8edb9c5f98 multinode/multinodedb: don't generate rx
Change-Id: Ica1a2bb1ab42e881daa951ab61b1dd99e877aff5
2023-09-11 16:40:52 +00:00
Egon Elbre
87bfb3b02b satellite/satellitedb: don't generate rx
I'm not sure what Rx is, however, we aren't using it --
so let's remove it for now.

Change-Id: I9caacbc150479f93945477101528a4fd60ea865f
2023-09-11 16:00:15 +00:00
Egon Elbre
3e73d414d1 satellite/console/consoleweb: initialize mime lazily
Change-Id: I80b78edcf057acef9b5a599cb77308baddc07692
2023-09-11 15:19:56 +00:00
Michal Niewrzal
0a3ee6ff8a satellite/metabase: remove old object segments on overwrite
While adding support for pending_objects table one case was missed.
When we are uploading object to location where old objects exists
we are not removing old object segments at all. Old object is
overwritten with new object metadata but segments remains without
corresponding object. This fix removes all existing committed objects
(with it's segments) before committing new object.

https://github.com/storj/storj/issues/6255

Change-Id: Id657840edf763fd6aec8191788d819191b074fb7
2023-09-11 14:16:47 +00:00
Vitalii
c31fb9c1cf satellite/payments: restrict addition of duplicate credit cards
By this change we don't allow users to add credit cards that are already bind to their account.
We still allow the same CC number but with a different expiration date.

Issue:
https://github.com/storj/storj/issues/5597

Change-Id: Ifeb0cc5ae0c2f0f7596af4dead70ae7d20d30613
2023-09-11 13:24:43 +03:00
Kaloyan Raev
3119b614ae cmd/uplink: --max-object-ttl flag for share and access restrict commands
Context: https://github.com/storj/storj/issues/6249

Change-Id: Ic65a1d8aef61f1a88752a7b12a23fb854dac8f6d
2023-09-11 08:14:55 +00:00
Vitalii
9254dd2208 web/satellite/vuetify-poc: enable object browser pagination
Enabled object browser pagination for vuetify app.
Also fixed some small bug when returning to first page.

Issue:
https://github.com/storj/storj/issues/5595

Change-Id: I8b5e90a4cd7d7a79a8beeb292b7374db3f93d700
2023-09-09 00:36:55 +00:00
Wilfred Asomani
2bf4113821 web/satellite/veutify-poc: create folder
This change adds the ability to create folders in the file browser.

Issue: https://github.com/storj/storj/issues/6105

Change-Id: I0dae0f9874b571cfd0ae79b2b994b58149d70aa3
2023-09-08 23:57:13 +00:00
Wilfred Asomani
54379fc0ee satellite/{console,analytics}: allow limit increase request
This change adds a  new endpoint that submits limit increase requests
to segment.

Issue: https://github.com/storj/storj/issues/6233

Change-Id: Ie4f70aef31079acbe2f24771b3ea359d5769eb95
2023-09-08 23:17:38 +00:00
Wilfred Asomani
cd7d9cf079 satellite/{console,payments}: unfreeze user on token payment
This change unfreezes/unwarns users in either state after successful
payment with tokens.

Change-Id: I7494d6a33cf9383576f42af695df522d8f409e03
2023-09-08 12:51:01 +00:00
Cameron
c52554a2b9 web/satellite: vuetify upload file and folder
add functionality to upload files and folders in object browser

issue: https://github.com/storj/storj/issues/6101
issue: https://github.com/storj/storj/issues/6102

Change-Id: I4aa86b89adc051b91b0d7fde69dd7375b2a3f370
2023-09-08 11:47:01 +00:00
Kaloyan Raev
4e499fb9bf satellite/metainfo: respect MaxObjectTTL in BeginObject
If MaxObjectTTL is set in the API key, BeginObject will use it for the
object expiration time, unless an explicit ExpireAt is available in the
request.

Context: https://github.com/storj/storj/issues/6249

Change-Id: I2adf57d979a9c68eec3a787f3739d2f1dbec1f7e
2023-09-08 09:30:48 +00:00
Moby von Briesen
8d1a765fd6 satellite/console: Partially revert change to remove graphql
This partially reverts commit 516241e406.

Endpoints are added to the backend, as there are some customers who may
use these endpoints, even though they are no longer necessary for the
satellite UI.

Change-Id: I52a99912d9eacf269fbb2ddca603e53c4af6d6bf
2023-09-07 20:50:24 +00:00
Wilfred Asomani
754bf5f8af Revert "satellite/db: optimize project usage query"
This reverts commit 31ec421299.

This change made the usages endpoint slower for accounts with large
number of projects.

Change-Id: I95870e95c2bf3bc3050087532fd0d20cbb50748b
2023-09-07 19:27:08 +00:00
Michal Niewrzal
df037564d7 satellite/zombiedeletion: remove inactive uploads from pending_objects
With zombie deletion chore we are removing inactive pending objects from
objects table but new we need also to do this for pending_objects table.

https://github.com/storj/storj/issues/6050

Change-Id: Ia29116c103673a1d9e10c2f16654022572210a8a
2023-09-07 18:47:29 +00:00
Márton Elek
ad87d1de74
satellite/satellitedb/overlaycache: fill node tags with join for limited number of nodes
The easiest way to get node information WITH node tags is executing two queries:

 1. select all nodes
 2. select all tags

And we can pair them with a loop, using the in-memory data structures.

But this approach does work only, if we select all nodes, which is true when we use cache (upload, download, repair checker).

But repair process selects only the required nodes, where this approach is suboptimal. (full table scan for all tags, even if we need only tags for a few dozens nodes).

Possible solutions:

 1. We can introduce a cache for repair (similar to upload cache)
 2. Or we can select both node and tag information with one query (join).

This patch implements the second approach.

Note: repair itself is quite slow (10-20 seconds per segements to repair). With 15 seconds execution time and 3 minutes cache staleness, we would use the cache only 12 times per worker. Probably we don't need cache for now.

https://github.com/storj/storj/issues/6198

Change-Id: I0364d94306e9815a1c280b71e843b8f504e3d870
2023-09-07 19:27:53 +02:00
Kaloyan Raev
82b108de69 satellite/console/consolewasm: no direct cast to grant.Permission
A new field is introduced to grant.Permission in storj.io/common. Having
a direct cast here leads to compilation problems when bumping
storj.io/uplink to the latest storj.io/common. Avoiding the direct cast
resolves the issue.

Context: https://github.com/storj/storj/issues/6249

Change-Id: I3b9bc14ebcce8e192e218c621b996300753b8de4
2023-09-07 13:27:53 +03:00
Moby von Briesen
6195b8cd52 satellite/admin: support more options for passing project ID
This change does two things:
* allow using either public ID or private ID to do project-related
  requests in admin UI
* allow passing a UUID string not containing dashes (i.e. a pure hex
  string) in order to do project-related requests in admin UI

Change-Id: I4807a5d7252a48f4a09e3966c406645d55c856e2
2023-09-07 08:53:41 +00:00
Dan Willoughby
091c72319a web/satellite: update welcome email links
Change-Id: Ic918a7ec30db1753d780643492704fc10aafeb99
2023-09-06 15:18:51 -06:00
Wilfred Asomani
7311d08139 web/satellite/vuetify-poc: upload via drag and drop
This change adds the drag-drop upload feature to the vuetify app.

Issue: https://github.com/storj/storj/issues/6104

Change-Id: I177e33a677d94db9ef95a31e32da853a46a7dc51
2023-09-06 19:20:50 +00:00
Vitalii
623b989973 web/satellite/vuetify-poc: unmock STORJ txs table on billing screen
Show real STORJ token transactions on billing screen in vuetify app.

Issue:
https://github.com/storj/storj/issues/6098

Change-Id: I1d7c2a613fefbf68c7ce3b8f62ec7ee992885bc4
2023-09-06 18:31:05 +03:00
Vitalii
6e1fd12930 web/satellite: paginate ListObjects request to show more than 1000 objects
With this change, we are able to fetch all objects to show in the object browser.
AWS SDK V3 provides paginator functionality to automatically make additional requests for every MaxKeys value (we use 500 objects at a time).
By initial request we fetch first 500 objects and save continuation tokens for the rest of the object batches.
Also, we save currently active (fetched) object range.
If user tries to open a page with objects which are out of currently active range then we look for needed continuation token and fetch needed objects batch.
Added a feature flag for this funtionality.

Issue:
https://github.com/storj/storj/issues/5595

Change-Id: If63e3c2ddaac3ea9f2bc1dc63cb49007f897e3e2
2023-09-06 12:47:15 +00:00
Márton Elek
f40baf8629
go.mod: bump dependencies (private,uplink,common)
Change-Id: I6c55735b45cadaf36697eff53e78b5b09afe9dea
2023-09-06 13:28:22 +02:00
Clement Sam
9ab934e2ae storagenode/piecestore: implement trash recovery for download requests
This change allows a node to look for a piece in the trash when
serving a download request.

If the piece is found in the trash, it restores it to the blobs
directory and continue to serve the request as expected.

Resolves https://github.com/storj/storj/issues/6145

Change-Id: Ibfa3c0b4954875fa977bc995fc4dd2705ca3ce42
2023-09-05 23:04:21 +00:00
Wilfred Asomani
3f1ea4a0b9 web/satellite/vuetify-poc: list invoices
This change lists invoices on the vuetifypoc billing history tab.
It also removes invoice filtering on the main app.

Issue: https://github.com/storj/storj/issues/6099

Change-Id: Id4cc2db003a0208775ddaefc87abf26f4b05106c
2023-09-05 15:01:16 +00:00
Wilfred Asomani
775db4aa3c satellite/payments: filter out draft invoices
This change filters the list of invoices sent to the frontend to only
contain open or paid invoices.

Change-Id: I9ac2640a7587a76b0baf46d941f799e742aa2b3b
2023-09-05 14:22:30 +00:00
Jeremy Wharton
4e3e31a425 satellite/console/consoleweb/consoleapi: refactor api requests in tests
This change refactors the way requests are sent in console API tests,
placing identical logic in a dedicated function to reduce code
duplication.

Change-Id: I7a5ac42d8d68a3fd9a9f8b9d61775659234e883f
2023-09-05 11:17:02 +00:00
Wilfred Asomani
2d18f43b6a web/satellite/vuetify-poc: show file previews
This change enables files to be previewed.

Issue: https://github.com/storj/storj/issues/6108

Change-Id: I3045950281822620de96b86e0180eb0d24e2d629
2023-09-05 10:35:31 +00:00
Márton Elek
e2006d821c satellite/overlay: change Reliable and KnownReliable
as GetParticipatingNodes and GetNodes, respectively.

We now want these functions to include offline and suspended nodes as
well, so that we can force immediate repair when pieces are out of
placement or in excluded countries. With that change, the old names no
longer made sense.

Change-Id: Icbcbad43dbde0ca8cbc80a4d17a896bb89b078b7
2023-09-02 23:34:50 +00:00
Cameron
6896241933 satellite/console: fix unverified invite email
When checking if invited user is unverified, initialize oldest
with unverified row rather than empty User, because empty User
CreatedAt is zero, so no real user could be created earlier.

Change-Id: I74dd8f7fc82951cbb61071632a74b1a9443b41fe
2023-09-01 15:11:58 -04:00
Michal Niewrzal
c9591e9754 satellite/metainfo: better metabase errors handling
Some errors were returned as metabase errors, not pure drpc
errors because of how rpcstatus.Code method is working. Status
code was returned for errors like metabase context canceled but
we would like to not leak our internals to the client.

Change-Id: I3f0194755f8d7359b1e3d342fa3be3d984019ecb
2023-09-01 11:32:10 +00:00
Moby von Briesen
b4c95a3b28 private/apigen/tsgen: Set query params better
The old way did not properly handle escaping, e.g. if the value of a
query param contained `&` or `=` inside it. By using
url.searchParams.set, we can safely add these types of arguments to the
path.

Change-Id: I62d3883b14f9d5a517e4a3d58f019014b46fd1b4
2023-08-31 20:09:29 -04:00
Jeremy Wharton
28d498f91d web/satellite/vuetify-poc: allow file browser downloads
This change allows files to be downloaded from within the file browser
of the Vuetify project.

Resolves #6107

Change-Id: I0ac0384711baccb99c0a6d382fe96f318290789b
2023-08-31 22:41:52 +00:00
Jeremy Wharton
b671641a28 satellite/console: update CSP to include storjapi.io
This change updates our content security policy to include the domain
storjapi.io and all of its subdomains.

References #6188

Change-Id: I6f3073bc32aa99626c54caf00bf07d2253ccbb8f
2023-08-31 22:02:45 +00:00
Márton Elek
c202929413
satellite/nodeselection: rename (NodeFilter).MatchInclude to Match
As I learned, the `Include` supposed to communicate that some internal change also "included" to the filters during the check -> filters might be stateful.

But it's not the case any more after 552242387, where we removed the only one stateful filter.

Change-Id: I7c36ddadb2defbfa3b6b67bcc115e4427ba9e083
2023-08-31 16:17:52 +02:00
Wilfred Asomani
dcc4bd0d10 satellite/{console,payments}: freeze/warn storjscan users
This change enables the freezing/warning of users who use storjscan.

Issue: https://github.com/storj/storj/issues/6164

Change-Id: I7b00ee09d6527b3818b72326e9065c82ef5a2ac8
2023-08-31 13:22:21 +00:00
Márton Elek
ca0ea50cba satellite/overlay: remove/deprecate NodeSelectionCache.Disabled
Once uppon a time, at the dawn of the implementation of Storj, when all the nodes are read from the database directly, every time.

After a while --  due to performance reasons -- it has been changed for upload and download: where all the nodes are read for a short period of time, and used from memory.

This is the version which was improved recently to support advanced node selections using placement.

But stil we have an old configuration value `service.config.NodeSelectionCache.Disabled`, and the db based implementation: `service.FindStorageNodesWithPreferences(ctx, req, &service.config.Node)`.

For safety, we need to remove this option, to make sure that we use the cache, which has the advanced features.

This patch was supposed to be a very small one (just removing a method and a config: https://review.dev.storj.io/c/storj/storj/+/11074/1/satellite/overlay/service.go), but it turned out that we need to update a lot of unit tests.

These unit tests used the old implementation (which is not used in production any more).

The tests which used both implementation are just updated to use only the new one
The tests which used only the old implementation are refactored (but keeping the test cases).
Using real unit tests (without DB, working on OSX, fast)

Closes https://github.com/storj/storj/issues/6217

Change-Id: I023f92c7e34235665cf8474513e67b2fcc4763eb
2023-08-31 09:46:29 +00:00
Jeremy Wharton
00194f54a2 web/satellite: show limit notifications to paid tier users
This change causes paid tier users to see notifications in the project
dashboard when their usage is approaching or has reached their maximum
or custom usage limits.

Change-Id: I7b68fcdd7d62797b6b26869e109cfb0b193fdddb
2023-08-31 07:31:09 +00:00
Wilfred Asomani
31ec421299 satellite/db: optimize project usage query
This change addresses an issue where the /charges endpoint will take a
while to respond due to a project having a large number of buckets.
The method and queries involved have been optimized and benchmarks show
a performance improvement.

test name                            old ms/op  new ms/op
Postgres/sum_all_partner_usages          3.659      1.101
Postgres/individual_partner_usages        3.74      1.299
Cockroach/sum_all_partner_usages         7.201      2.872
Cockroach/individual_partner_usages      7.247      2.852

Issue: https://github.com/storj/storj-private/issues/277

Change-Id: Ia5082a2e1c3e91120a9db7b01c18847fe04574fe
2023-08-30 22:08:11 +00:00
Jeremy Wharton
0c9c37875d web/satellite/vuetify-poc: unmock file browser table entries
This change shows real file entries in the file browser table,
replacing the mock data. Sorting, searching, and folder navigation have
been implemented.

Resolves #6199

Change-Id: I7360879d2e26605489c20f9d094c3f231fee49cd
2023-08-30 21:12:55 +00:00
Jeremy Wharton
ebfbbca1be web/satellite/vuetify-poc: pull latest changes from internal repo
The changes as of storj/vuetify-storj@2312936 have been pulled into the
Vuetify project.

Change-Id: Ia45d3d0832727d3ca7b9ee9b3dce5949a79a5d31
2023-08-30 20:34:13 +00:00
Wilfred Asomani
fecaa6a71a web/satellite/vuetify-poc: enable claiming wallets
This change adds the ability to claim STORJ wallets and display token
balance.

Issue: https://github.com/storj/storj/issues/6096

Change-Id: Ifc89b586c0e3ed876905ff0a5b270e718cbb689c
2023-08-30 14:33:35 +00:00
Michal Niewrzal
67371c43bd satellite/metainfo: enable UsePendingObjectsTable by project
This small feature will give us ability to test pending_objects table
without enabling it globally.

Change-Id: I802f45987ad329f94adfc0f02957c802b21d8251
2023-08-30 15:44:29 +02:00
Michal Niewrzal
780c0e0b35 satellite/metainfo: adjust ListPendingObjectStreams to pending_objects
table

New method IteratePendingObjectsByKeyNew is used to provide results for
metainfo.ListPendingObjectStreams. This endpoint is used to list
pending objects with the same object key. In this case to support
both tables (objects, pending_objects) we need to do one query per table
and merge results.

Because existing metainfo protobuf API is missing some fields to have
proper listing cursor we are not able to make ListPendingObjectStreams
correct for returning more than single page. We need to fix it
separately.

With this change also turns out that approach to merge results from
listing objects for ListObjects method was wrong and this change is also
fixing this problem.

Handling both tables will be removed at some point and only
pending_objects will be used to look for results.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I8a88a6f885ad529704e6c032f1d97926123c2909
2023-08-30 13:35:54 +00:00
Wilfred Asomani
6e3da022e0 web/satellite: add pagination to billing history
This change adds pagination to the billing history table. It uses the
new invoice-history endpoint since we only list invoices in this table.

Issue: https://github.com/storj/storj/issues/5479

Change-Id: I192d58503434203808a23a7c18e8d1feb6afc73f
2023-08-30 03:19:31 +00:00
Vitalii
614d213432 web/satellite: fix duplicate satellite selection on login and forgot password pages
Remove duplicate satellite in selection dropdown on login and forgot password pages.

Issue:
https://github.com/storj/storj/issues/6121

Change-Id: I81bd0c2c29c695ed9ed7f0d648311448cdc9bf26
2023-08-30 01:17:14 +00:00
Cameron
a5b1c0432f satellite/console: send unverified user activation link in project invite
When an unverified user is sent a project invitation it contains a
registration link currently. Instead, send an activation link.

github issue: https://github.com/storj/storj/issues/6033

Change-Id: I54b88de8347a2532f7a85372c0c5e4df4bf4eb38
2023-08-29 12:54:09 -04:00