Commit Graph

8508 Commits

Author SHA1 Message Date
Clement Sam
f14fabc90a cmd/storagenode: add forget-satellite subcommand
This change adds a new forget-satellite sub-command to
the storagenode CLI which cleans up untrusted satellite
data.

Issue: https://github.com/storj/storj/issues/6068
Change-Id: Iafa109fdc98afdba7582f568a61c22222da65f02
2023-09-13 19:06:55 +00:00
Wilfred Asomani
dcf3f25f93 satellite/admin: update README
add descriptions for the endpoint that removes a user from the waning
state.

Issue: https://github.com/storj/storj/issues/6118

Change-Id: I211cd3c41c7fefa295d0db1b9f43f53e33b984e6
2023-09-13 17:50:28 +00:00
Jeremy Wharton
7d8b231aaf cmd/satellite: build Vuetify project in Dockerfile
This change updates the satellite Dockerfile to build the Vuetify
project.

References #6251

Change-Id: I699360c0f7eb7a8abdd0bc523ee74910a04fecd3
2023-09-13 12:43:58 +00:00
Michal Niewrzal
fb04a22088 satellite/metainfo: flag to rollout pending_objects table support
To avoid enabling feature for every project at once we would like to
do this partially and control percentage of projects that will have
feature enabled.

https://github.com/storj/storj/issues/6258

Change-Id: Iaac7c42d39da76ed2ecc439847c3b210462befa5
2023-09-13 10:33:23 +00:00
Egon Elbre
d91ee440ba satellite/metabase: use constant for pending version
Currently it wasn't quite clear what was a stub version and an actual
version. Use a PendingVersion constant to make this distinction clear.

Also use PendingVersion = NextVersion = 0, that way it's clearer that
the version hasn't been yet determined. DefaultVersion = 1 might imply
that the object will get that version once commited, however that will
entirely depend on whether use-pending-objects is used or versioning is
enabled or not.

Change-Id: I21398141f97035c48c778f23b542266b834c44f1
2023-09-12 18:01:12 +00:00
Jeremy Wharton
4f8697568d web/satellite: use edge service URL overrides
This change makes the satellite frontend use edge service URL overrides
if they have been configured for a project.

Resolves #6188
Resolves #6190

Change-Id: I4c8fb3f5f00f450fb8cd139383972ab622234fb0
2023-09-12 12:19:20 -05:00
Jeremy Wharton
c8f4f5210d satellite/console: return edge URL overrides in project info responses
API responses containing project information now contain the edge
service URL overrides configured for that project. The overrides are
based on the project's default placement.

References #6188

Change-Id: Ifc3dc74e75c0f5daf0419ac3be184415c65b202e
2023-09-12 12:10:18 -05:00
Clement Sam
89d682f49f satellite/metainfo: prevent internal DB errors in Public API
Resolves https://github.com/storj/storj/issues/6081

Change-Id: I0e530db39947138dcafc1b6bd1710ff1ca96b8c5
2023-09-12 15:12:44 +00:00
Michal Niewrzal
e21978f11a satellite/metainfo: support getting specific object version
Protobuf definition is read to support getting specific version of
object so we just need to wire requested version into metainfo.GetObject
endpoint.

https://github.com/storj/storj/issues/6221

Change-Id: If4568b82119a6c893788a0a86e598b05ff5951cf
2023-09-12 14:27:40 +00:00
Vitalii
4e4da7be6d web/satellite/vuetify-poc: always show details container under validatable inputs
Inputs that have some custom validation messaging attached should not use hide-details=auto to prevent 'jumpy' visual experience.

Issue:
https://github.com/storj/storj/issues/6230

Change-Id: Ia90e122516eb853a3908c0f57634971243fb38b3
2023-09-12 13:47:22 +00:00
Jeremy Wharton
088496efdf web/satellite/vuetify-poc: allow for deleting files and folders
This change allows files and folders to be deleted from within the
Vuetify project's file browser.

Resolves #6106

Change-Id: I0d7b0528b08333aeec29917c4ebef6ea966ac1fa
2023-09-12 13:08:09 +00:00
Egon Elbre
28ee6f024c satellite: don't use fmt.Print in tests
Change-Id: Ia10450240ad075c9d78614adff9164f292fb1fa0
2023-09-12 15:11:02 +03:00
Márton Elek
afa5c54a35 satellite/satellitedb: add placement column to repair_queue
It makes it possible to run dedicated repair worker for different placement definitions.

https://github.com/storj/storj-private/issues/400

Change-Id: I376da867da5dbb4ab392d5f86c766f7543c32ee6
2023-09-12 11:29:18 +00:00
Vitalii
2cf233ac23 web/satellite/vuetify-poc: hide nav sidebar by default for screens less than 1280px
On screen sizes where the sidebar overlaps with the content of the page, the sidebar should be collapsed by default.

Issue:
https://github.com/storj/storj/issues/6229

Change-Id: Ia0a91acd95519de27f9ff8f1ee90c6b8e7932266
2023-09-12 13:48:53 +03:00
Cameron
f9ab2f0de7 satellite/console: prefix oidc paths with /api/v0/
add /api/v0/ prefix to oidc paths to route requests to the api
pod with split UI infrastructure.

Change-Id: I1e7e691487ab1d4e84434d204d10b7f944ae8873
2023-09-11 18:12:39 +00:00
Egon Elbre
487f64e164 satellite/satellitedb,multinode/multinodedb: update to latest dbx
Change-Id: I500df6d0541706c3960d4560721c3783d0d049ff
2023-09-11 17:21:02 +00:00
Egon Elbre
8edb9c5f98 multinode/multinodedb: don't generate rx
Change-Id: Ica1a2bb1ab42e881daa951ab61b1dd99e877aff5
2023-09-11 16:40:52 +00:00
Egon Elbre
87bfb3b02b satellite/satellitedb: don't generate rx
I'm not sure what Rx is, however, we aren't using it --
so let's remove it for now.

Change-Id: I9caacbc150479f93945477101528a4fd60ea865f
2023-09-11 16:00:15 +00:00
Egon Elbre
3e73d414d1 satellite/console/consoleweb: initialize mime lazily
Change-Id: I80b78edcf057acef9b5a599cb77308baddc07692
2023-09-11 15:19:56 +00:00
Michal Niewrzal
0a3ee6ff8a satellite/metabase: remove old object segments on overwrite
While adding support for pending_objects table one case was missed.
When we are uploading object to location where old objects exists
we are not removing old object segments at all. Old object is
overwritten with new object metadata but segments remains without
corresponding object. This fix removes all existing committed objects
(with it's segments) before committing new object.

https://github.com/storj/storj/issues/6255

Change-Id: Id657840edf763fd6aec8191788d819191b074fb7
2023-09-11 14:16:47 +00:00
Vitalii
c31fb9c1cf satellite/payments: restrict addition of duplicate credit cards
By this change we don't allow users to add credit cards that are already bind to their account.
We still allow the same CC number but with a different expiration date.

Issue:
https://github.com/storj/storj/issues/5597

Change-Id: Ifeb0cc5ae0c2f0f7596af4dead70ae7d20d30613
2023-09-11 13:24:43 +03:00
Kaloyan Raev
3119b614ae cmd/uplink: --max-object-ttl flag for share and access restrict commands
Context: https://github.com/storj/storj/issues/6249

Change-Id: Ic65a1d8aef61f1a88752a7b12a23fb854dac8f6d
2023-09-11 08:14:55 +00:00
Vitalii
9254dd2208 web/satellite/vuetify-poc: enable object browser pagination
Enabled object browser pagination for vuetify app.
Also fixed some small bug when returning to first page.

Issue:
https://github.com/storj/storj/issues/5595

Change-Id: I8b5e90a4cd7d7a79a8beeb292b7374db3f93d700
2023-09-09 00:36:55 +00:00
Wilfred Asomani
2bf4113821 web/satellite/veutify-poc: create folder
This change adds the ability to create folders in the file browser.

Issue: https://github.com/storj/storj/issues/6105

Change-Id: I0dae0f9874b571cfd0ae79b2b994b58149d70aa3
2023-09-08 23:57:13 +00:00
Wilfred Asomani
54379fc0ee satellite/{console,analytics}: allow limit increase request
This change adds a  new endpoint that submits limit increase requests
to segment.

Issue: https://github.com/storj/storj/issues/6233

Change-Id: Ie4f70aef31079acbe2f24771b3ea359d5769eb95
2023-09-08 23:17:38 +00:00
Wilfred Asomani
cd7d9cf079 satellite/{console,payments}: unfreeze user on token payment
This change unfreezes/unwarns users in either state after successful
payment with tokens.

Change-Id: I7494d6a33cf9383576f42af695df522d8f409e03
2023-09-08 12:51:01 +00:00
Cameron
c52554a2b9 web/satellite: vuetify upload file and folder
add functionality to upload files and folders in object browser

issue: https://github.com/storj/storj/issues/6101
issue: https://github.com/storj/storj/issues/6102

Change-Id: I4aa86b89adc051b91b0d7fde69dd7375b2a3f370
2023-09-08 11:47:01 +00:00
Kaloyan Raev
4e499fb9bf satellite/metainfo: respect MaxObjectTTL in BeginObject
If MaxObjectTTL is set in the API key, BeginObject will use it for the
object expiration time, unless an explicit ExpireAt is available in the
request.

Context: https://github.com/storj/storj/issues/6249

Change-Id: I2adf57d979a9c68eec3a787f3739d2f1dbec1f7e
2023-09-08 09:30:48 +00:00
Moby von Briesen
8d1a765fd6 satellite/console: Partially revert change to remove graphql
This partially reverts commit 516241e406.

Endpoints are added to the backend, as there are some customers who may
use these endpoints, even though they are no longer necessary for the
satellite UI.

Change-Id: I52a99912d9eacf269fbb2ddca603e53c4af6d6bf
2023-09-07 20:50:24 +00:00
Wilfred Asomani
754bf5f8af Revert "satellite/db: optimize project usage query"
This reverts commit 31ec421299.

This change made the usages endpoint slower for accounts with large
number of projects.

Change-Id: I95870e95c2bf3bc3050087532fd0d20cbb50748b
2023-09-07 19:27:08 +00:00
Michal Niewrzal
df037564d7 satellite/zombiedeletion: remove inactive uploads from pending_objects
With zombie deletion chore we are removing inactive pending objects from
objects table but new we need also to do this for pending_objects table.

https://github.com/storj/storj/issues/6050

Change-Id: Ia29116c103673a1d9e10c2f16654022572210a8a
2023-09-07 18:47:29 +00:00
Márton Elek
ad87d1de74
satellite/satellitedb/overlaycache: fill node tags with join for limited number of nodes
The easiest way to get node information WITH node tags is executing two queries:

 1. select all nodes
 2. select all tags

And we can pair them with a loop, using the in-memory data structures.

But this approach does work only, if we select all nodes, which is true when we use cache (upload, download, repair checker).

But repair process selects only the required nodes, where this approach is suboptimal. (full table scan for all tags, even if we need only tags for a few dozens nodes).

Possible solutions:

 1. We can introduce a cache for repair (similar to upload cache)
 2. Or we can select both node and tag information with one query (join).

This patch implements the second approach.

Note: repair itself is quite slow (10-20 seconds per segements to repair). With 15 seconds execution time and 3 minutes cache staleness, we would use the cache only 12 times per worker. Probably we don't need cache for now.

https://github.com/storj/storj/issues/6198

Change-Id: I0364d94306e9815a1c280b71e843b8f504e3d870
2023-09-07 19:27:53 +02:00
Kaloyan Raev
82b108de69 satellite/console/consolewasm: no direct cast to grant.Permission
A new field is introduced to grant.Permission in storj.io/common. Having
a direct cast here leads to compilation problems when bumping
storj.io/uplink to the latest storj.io/common. Avoiding the direct cast
resolves the issue.

Context: https://github.com/storj/storj/issues/6249

Change-Id: I3b9bc14ebcce8e192e218c621b996300753b8de4
2023-09-07 13:27:53 +03:00
Moby von Briesen
6195b8cd52 satellite/admin: support more options for passing project ID
This change does two things:
* allow using either public ID or private ID to do project-related
  requests in admin UI
* allow passing a UUID string not containing dashes (i.e. a pure hex
  string) in order to do project-related requests in admin UI

Change-Id: I4807a5d7252a48f4a09e3966c406645d55c856e2
2023-09-07 08:53:41 +00:00
Dan Willoughby
091c72319a web/satellite: update welcome email links
Change-Id: Ic918a7ec30db1753d780643492704fc10aafeb99
2023-09-06 15:18:51 -06:00
Wilfred Asomani
7311d08139 web/satellite/vuetify-poc: upload via drag and drop
This change adds the drag-drop upload feature to the vuetify app.

Issue: https://github.com/storj/storj/issues/6104

Change-Id: I177e33a677d94db9ef95a31e32da853a46a7dc51
2023-09-06 19:20:50 +00:00
Vitalii
623b989973 web/satellite/vuetify-poc: unmock STORJ txs table on billing screen
Show real STORJ token transactions on billing screen in vuetify app.

Issue:
https://github.com/storj/storj/issues/6098

Change-Id: I1d7c2a613fefbf68c7ce3b8f62ec7ee992885bc4
2023-09-06 18:31:05 +03:00
Vitalii
6e1fd12930 web/satellite: paginate ListObjects request to show more than 1000 objects
With this change, we are able to fetch all objects to show in the object browser.
AWS SDK V3 provides paginator functionality to automatically make additional requests for every MaxKeys value (we use 500 objects at a time).
By initial request we fetch first 500 objects and save continuation tokens for the rest of the object batches.
Also, we save currently active (fetched) object range.
If user tries to open a page with objects which are out of currently active range then we look for needed continuation token and fetch needed objects batch.
Added a feature flag for this funtionality.

Issue:
https://github.com/storj/storj/issues/5595

Change-Id: If63e3c2ddaac3ea9f2bc1dc63cb49007f897e3e2
2023-09-06 12:47:15 +00:00
Márton Elek
f40baf8629
go.mod: bump dependencies (private,uplink,common)
Change-Id: I6c55735b45cadaf36697eff53e78b5b09afe9dea
2023-09-06 13:28:22 +02:00
Clement Sam
9ab934e2ae storagenode/piecestore: implement trash recovery for download requests
This change allows a node to look for a piece in the trash when
serving a download request.

If the piece is found in the trash, it restores it to the blobs
directory and continue to serve the request as expected.

Resolves https://github.com/storj/storj/issues/6145

Change-Id: Ibfa3c0b4954875fa977bc995fc4dd2705ca3ce42
2023-09-05 23:04:21 +00:00
Wilfred Asomani
3f1ea4a0b9 web/satellite/vuetify-poc: list invoices
This change lists invoices on the vuetifypoc billing history tab.
It also removes invoice filtering on the main app.

Issue: https://github.com/storj/storj/issues/6099

Change-Id: Id4cc2db003a0208775ddaefc87abf26f4b05106c
2023-09-05 15:01:16 +00:00
Wilfred Asomani
775db4aa3c satellite/payments: filter out draft invoices
This change filters the list of invoices sent to the frontend to only
contain open or paid invoices.

Change-Id: I9ac2640a7587a76b0baf46d941f799e742aa2b3b
2023-09-05 14:22:30 +00:00
Jeremy Wharton
4e3e31a425 satellite/console/consoleweb/consoleapi: refactor api requests in tests
This change refactors the way requests are sent in console API tests,
placing identical logic in a dedicated function to reduce code
duplication.

Change-Id: I7a5ac42d8d68a3fd9a9f8b9d61775659234e883f
2023-09-05 11:17:02 +00:00
Wilfred Asomani
2d18f43b6a web/satellite/vuetify-poc: show file previews
This change enables files to be previewed.

Issue: https://github.com/storj/storj/issues/6108

Change-Id: I3045950281822620de96b86e0180eb0d24e2d629
2023-09-05 10:35:31 +00:00
Márton Elek
e2006d821c satellite/overlay: change Reliable and KnownReliable
as GetParticipatingNodes and GetNodes, respectively.

We now want these functions to include offline and suspended nodes as
well, so that we can force immediate repair when pieces are out of
placement or in excluded countries. With that change, the old names no
longer made sense.

Change-Id: Icbcbad43dbde0ca8cbc80a4d17a896bb89b078b7
2023-09-02 23:34:50 +00:00
Cameron
6896241933 satellite/console: fix unverified invite email
When checking if invited user is unverified, initialize oldest
with unverified row rather than empty User, because empty User
CreatedAt is zero, so no real user could be created earlier.

Change-Id: I74dd8f7fc82951cbb61071632a74b1a9443b41fe
2023-09-01 15:11:58 -04:00
Michal Niewrzal
c9591e9754 satellite/metainfo: better metabase errors handling
Some errors were returned as metabase errors, not pure drpc
errors because of how rpcstatus.Code method is working. Status
code was returned for errors like metabase context canceled but
we would like to not leak our internals to the client.

Change-Id: I3f0194755f8d7359b1e3d342fa3be3d984019ecb
2023-09-01 11:32:10 +00:00
Moby von Briesen
b4c95a3b28 private/apigen/tsgen: Set query params better
The old way did not properly handle escaping, e.g. if the value of a
query param contained `&` or `=` inside it. By using
url.searchParams.set, we can safely add these types of arguments to the
path.

Change-Id: I62d3883b14f9d5a517e4a3d58f019014b46fd1b4
2023-08-31 20:09:29 -04:00
Jeremy Wharton
28d498f91d web/satellite/vuetify-poc: allow file browser downloads
This change allows files to be downloaded from within the file browser
of the Vuetify project.

Resolves #6107

Change-Id: I0ac0384711baccb99c0a6d382fe96f318290789b
2023-08-31 22:41:52 +00:00
Jeremy Wharton
b671641a28 satellite/console: update CSP to include storjapi.io
This change updates our content security policy to include the domain
storjapi.io and all of its subdomains.

References #6188

Change-Id: I6f3073bc32aa99626c54caf00bf07d2253ccbb8f
2023-08-31 22:02:45 +00:00