Commit Graph

3540 Commits

Author SHA1 Message Date
Ivan Fraixedes
8381483f79
satellite/admin: Add back-office UI sources
Add the front-end sources of the new back-office.

The front-end doesn't have any business logic, it only has the pages and
the components, so it's purely UI.

The front-end was developed in a separate repository until was
completed.

Change-Id: I382e50789d6b929a67b8a0b887563ef48cb1473d
2023-09-22 18:02:55 +02:00
Márton Elek
98921f9faa satellite/overlay: fix placement selection config parsing
When we do `satellite run api --placement '...'`, the placement rules are not parsed well.

The problem is based on `viper.AllSettings()`, and the main logic is sg. like this (from a new unit test):

```
		r := ConfigurablePlacementRule{}
		err := r.Set(p)
		require.NoError(t, err)
		serialized := r.String()

		r2 := ConfigurablePlacementRule{}
		err = r2.Set(serialized)
		require.NoError(t, err)

		require.Equal(t, p, r2.String())
```

All settings evaluates the placement rules in `ConfigurablePlacementRules` and stores the string representation.

The problem is that we don't have proper `String()` implementation (it prints out the structs instead of the original definition.

There are two main solutions for this problem:

 1. We can fix the `String()`. When we parse a placement rule, the `String()` method should print out the original definition
 2. We can switch to use pure string as configuration parameter, and parse the rules only when required.

I feel that 1 is error prone, we can do it (and in this patch I added a lot of `String()` implementations, but it's hard to be sure that our `String()` logic is inline with the parsing logic.

Therefore I decided to make the configuration value of the placements a string (or a wrapper around string).

That's the main reason why this patch seems to be big, as I updated all the usages.

But the main part is in beginning of the `placement.go` (configuration parsing is not a pflag.Value implementation any more, but a separated step).

And `filter.go`, (a few more String implementation for filters.

https://github.com/storj/storj/issues/6248

Change-Id: I47c762d3514342b76a2e85683b1c891502a0756a
2023-09-21 14:31:41 +00:00
Wilfred Asomani
e599df03a8 satellite/console: add more cross-user api tests
This change adds more endpoints to the cross-user api test.

Issue: https://github.com/storj/storj/issues/6246

Change-Id: I4c3128c3a932c713b10499a8909836a599b79458
2023-09-19 16:50:31 +00:00
Cameron
a2d37bc69a {satellite/console, web/satellite}: send analytics if invitee signs up
send analytics event if project invite link is clicked and if user
signs up.

github issue: https://github.com/storj/storj/issues/5190

Change-Id: I41eee5e679a84b9ec325815655684a98624d5656
2023-09-19 14:50:44 +00:00
Michal Niewrzal
5d0934e4d9 satellite/metabase/rangedloop: disable ranged loop for tests
Currently each testplanet test is running ranged loop no matter if
it's used or not. This is small change with some benefits like:
* saves some cpu cycles
* less log entries
* ranged loop won't interfere with other systems

Change have no big impact on tests execration but I believe it's nice to
have.

Change-Id: I731846bf625cac47ed4f3ca3bc1d1a4659bdcce8
2023-09-18 22:51:10 +00:00
Egon Elbre
c48f58e968 satellite/metabase: simplify BeginExactObject
There's only one value that BeginExactObject should return and there's
no point in restating that.

Also, use a clearer value for the random object version.

Change-Id: I06b26ad87d64e1b04b48458f624edd630f7f2f1d
2023-09-18 19:26:49 +00:00
Wilfred Asomani
8ad0bc5e61 satellite/console: add alt code protected MFA recovery endpoint
This change adds an alternate MFA code recovery endpoint that requires
MFA code to generate codes.

Issue: https://github.com/storj/storj-private/issues/433

Change-Id: I10d922e9ad1ace4300d4bcfea7f48494227f1ff8
2023-09-18 16:07:43 +00:00
Michal Niewrzal
975d953cb8 satellite/metabase: custom error for commit object
We stoped returning lots of errors as is to avoid leaking our internals
but some errors were meanigful for client. Example of such error is
"exceeded maximum number of parts". With this change we are wrapping
some important commit object errors with new ErrFailedPrecondition
error to be able to return it easily to uplink.

Change-Id: Id834b78362ed1920f0c3f6f1c7d9587bfd27e36a
2023-09-15 09:52:42 +00:00
Márton Elek
f4fe983b1e satellite/{placement,nodeselection}: introduce empty() and notEmpty() for tag value selection
It helps to implement rules like `tag("nodeid","select",notEmpty())

Change-Id: If7a4532eacc0e4e670ffe81d504aab9d5b34302f
2023-09-14 19:30:29 +00:00
Michal Niewrzal
881137539c satellite/metabase: commit object is not respecting expiration time
With pending_objects table support enabled we missed passing correctly
expiration time from pending object to committed object. This change
updates commit query to take into account expiration time.

Change-Id: I67146d5b2f7f0bda02925d16275fbc59acb705bd
2023-09-14 16:25:28 +02:00
Vitalii
8f27425284 satellite/{projectaccounting, web}: merge settled and allocated lines for bandwidth graph
Merged bandwidth graph lines to show only allocated-dead for last 3 days and settled for other days.

Issue:
https://github.com/storj/storj/issues/6072

Change-Id: Ic7f03d22ccd82d27ae6e6a85e73e144c9852e33b
2023-09-14 15:39:56 +03:00
Wilfred Asomani
dcf3f25f93 satellite/admin: update README
add descriptions for the endpoint that removes a user from the waning
state.

Issue: https://github.com/storj/storj/issues/6118

Change-Id: I211cd3c41c7fefa295d0db1b9f43f53e33b984e6
2023-09-13 17:50:28 +00:00
Michal Niewrzal
fb04a22088 satellite/metainfo: flag to rollout pending_objects table support
To avoid enabling feature for every project at once we would like to
do this partially and control percentage of projects that will have
feature enabled.

https://github.com/storj/storj/issues/6258

Change-Id: Iaac7c42d39da76ed2ecc439847c3b210462befa5
2023-09-13 10:33:23 +00:00
Egon Elbre
d91ee440ba satellite/metabase: use constant for pending version
Currently it wasn't quite clear what was a stub version and an actual
version. Use a PendingVersion constant to make this distinction clear.

Also use PendingVersion = NextVersion = 0, that way it's clearer that
the version hasn't been yet determined. DefaultVersion = 1 might imply
that the object will get that version once commited, however that will
entirely depend on whether use-pending-objects is used or versioning is
enabled or not.

Change-Id: I21398141f97035c48c778f23b542266b834c44f1
2023-09-12 18:01:12 +00:00
Jeremy Wharton
c8f4f5210d satellite/console: return edge URL overrides in project info responses
API responses containing project information now contain the edge
service URL overrides configured for that project. The overrides are
based on the project's default placement.

References #6188

Change-Id: Ifc3dc74e75c0f5daf0419ac3be184415c65b202e
2023-09-12 12:10:18 -05:00
Clement Sam
89d682f49f satellite/metainfo: prevent internal DB errors in Public API
Resolves https://github.com/storj/storj/issues/6081

Change-Id: I0e530db39947138dcafc1b6bd1710ff1ca96b8c5
2023-09-12 15:12:44 +00:00
Michal Niewrzal
e21978f11a satellite/metainfo: support getting specific object version
Protobuf definition is read to support getting specific version of
object so we just need to wire requested version into metainfo.GetObject
endpoint.

https://github.com/storj/storj/issues/6221

Change-Id: If4568b82119a6c893788a0a86e598b05ff5951cf
2023-09-12 14:27:40 +00:00
Egon Elbre
28ee6f024c satellite: don't use fmt.Print in tests
Change-Id: Ia10450240ad075c9d78614adff9164f292fb1fa0
2023-09-12 15:11:02 +03:00
Márton Elek
afa5c54a35 satellite/satellitedb: add placement column to repair_queue
It makes it possible to run dedicated repair worker for different placement definitions.

https://github.com/storj/storj-private/issues/400

Change-Id: I376da867da5dbb4ab392d5f86c766f7543c32ee6
2023-09-12 11:29:18 +00:00
Cameron
f9ab2f0de7 satellite/console: prefix oidc paths with /api/v0/
add /api/v0/ prefix to oidc paths to route requests to the api
pod with split UI infrastructure.

Change-Id: I1e7e691487ab1d4e84434d204d10b7f944ae8873
2023-09-11 18:12:39 +00:00
Egon Elbre
487f64e164 satellite/satellitedb,multinode/multinodedb: update to latest dbx
Change-Id: I500df6d0541706c3960d4560721c3783d0d049ff
2023-09-11 17:21:02 +00:00
Egon Elbre
87bfb3b02b satellite/satellitedb: don't generate rx
I'm not sure what Rx is, however, we aren't using it --
so let's remove it for now.

Change-Id: I9caacbc150479f93945477101528a4fd60ea865f
2023-09-11 16:00:15 +00:00
Egon Elbre
3e73d414d1 satellite/console/consoleweb: initialize mime lazily
Change-Id: I80b78edcf057acef9b5a599cb77308baddc07692
2023-09-11 15:19:56 +00:00
Michal Niewrzal
0a3ee6ff8a satellite/metabase: remove old object segments on overwrite
While adding support for pending_objects table one case was missed.
When we are uploading object to location where old objects exists
we are not removing old object segments at all. Old object is
overwritten with new object metadata but segments remains without
corresponding object. This fix removes all existing committed objects
(with it's segments) before committing new object.

https://github.com/storj/storj/issues/6255

Change-Id: Id657840edf763fd6aec8191788d819191b074fb7
2023-09-11 14:16:47 +00:00
Vitalii
c31fb9c1cf satellite/payments: restrict addition of duplicate credit cards
By this change we don't allow users to add credit cards that are already bind to their account.
We still allow the same CC number but with a different expiration date.

Issue:
https://github.com/storj/storj/issues/5597

Change-Id: Ifeb0cc5ae0c2f0f7596af4dead70ae7d20d30613
2023-09-11 13:24:43 +03:00
Wilfred Asomani
54379fc0ee satellite/{console,analytics}: allow limit increase request
This change adds a  new endpoint that submits limit increase requests
to segment.

Issue: https://github.com/storj/storj/issues/6233

Change-Id: Ie4f70aef31079acbe2f24771b3ea359d5769eb95
2023-09-08 23:17:38 +00:00
Wilfred Asomani
cd7d9cf079 satellite/{console,payments}: unfreeze user on token payment
This change unfreezes/unwarns users in either state after successful
payment with tokens.

Change-Id: I7494d6a33cf9383576f42af695df522d8f409e03
2023-09-08 12:51:01 +00:00
Kaloyan Raev
4e499fb9bf satellite/metainfo: respect MaxObjectTTL in BeginObject
If MaxObjectTTL is set in the API key, BeginObject will use it for the
object expiration time, unless an explicit ExpireAt is available in the
request.

Context: https://github.com/storj/storj/issues/6249

Change-Id: I2adf57d979a9c68eec3a787f3739d2f1dbec1f7e
2023-09-08 09:30:48 +00:00
Moby von Briesen
8d1a765fd6 satellite/console: Partially revert change to remove graphql
This partially reverts commit 516241e406.

Endpoints are added to the backend, as there are some customers who may
use these endpoints, even though they are no longer necessary for the
satellite UI.

Change-Id: I52a99912d9eacf269fbb2ddca603e53c4af6d6bf
2023-09-07 20:50:24 +00:00
Wilfred Asomani
754bf5f8af Revert "satellite/db: optimize project usage query"
This reverts commit 31ec421299.

This change made the usages endpoint slower for accounts with large
number of projects.

Change-Id: I95870e95c2bf3bc3050087532fd0d20cbb50748b
2023-09-07 19:27:08 +00:00
Michal Niewrzal
df037564d7 satellite/zombiedeletion: remove inactive uploads from pending_objects
With zombie deletion chore we are removing inactive pending objects from
objects table but new we need also to do this for pending_objects table.

https://github.com/storj/storj/issues/6050

Change-Id: Ia29116c103673a1d9e10c2f16654022572210a8a
2023-09-07 18:47:29 +00:00
Márton Elek
ad87d1de74
satellite/satellitedb/overlaycache: fill node tags with join for limited number of nodes
The easiest way to get node information WITH node tags is executing two queries:

 1. select all nodes
 2. select all tags

And we can pair them with a loop, using the in-memory data structures.

But this approach does work only, if we select all nodes, which is true when we use cache (upload, download, repair checker).

But repair process selects only the required nodes, where this approach is suboptimal. (full table scan for all tags, even if we need only tags for a few dozens nodes).

Possible solutions:

 1. We can introduce a cache for repair (similar to upload cache)
 2. Or we can select both node and tag information with one query (join).

This patch implements the second approach.

Note: repair itself is quite slow (10-20 seconds per segements to repair). With 15 seconds execution time and 3 minutes cache staleness, we would use the cache only 12 times per worker. Probably we don't need cache for now.

https://github.com/storj/storj/issues/6198

Change-Id: I0364d94306e9815a1c280b71e843b8f504e3d870
2023-09-07 19:27:53 +02:00
Kaloyan Raev
82b108de69 satellite/console/consolewasm: no direct cast to grant.Permission
A new field is introduced to grant.Permission in storj.io/common. Having
a direct cast here leads to compilation problems when bumping
storj.io/uplink to the latest storj.io/common. Avoiding the direct cast
resolves the issue.

Context: https://github.com/storj/storj/issues/6249

Change-Id: I3b9bc14ebcce8e192e218c621b996300753b8de4
2023-09-07 13:27:53 +03:00
Moby von Briesen
6195b8cd52 satellite/admin: support more options for passing project ID
This change does two things:
* allow using either public ID or private ID to do project-related
  requests in admin UI
* allow passing a UUID string not containing dashes (i.e. a pure hex
  string) in order to do project-related requests in admin UI

Change-Id: I4807a5d7252a48f4a09e3966c406645d55c856e2
2023-09-07 08:53:41 +00:00
Vitalii
6e1fd12930 web/satellite: paginate ListObjects request to show more than 1000 objects
With this change, we are able to fetch all objects to show in the object browser.
AWS SDK V3 provides paginator functionality to automatically make additional requests for every MaxKeys value (we use 500 objects at a time).
By initial request we fetch first 500 objects and save continuation tokens for the rest of the object batches.
Also, we save currently active (fetched) object range.
If user tries to open a page with objects which are out of currently active range then we look for needed continuation token and fetch needed objects batch.
Added a feature flag for this funtionality.

Issue:
https://github.com/storj/storj/issues/5595

Change-Id: If63e3c2ddaac3ea9f2bc1dc63cb49007f897e3e2
2023-09-06 12:47:15 +00:00
Márton Elek
f40baf8629
go.mod: bump dependencies (private,uplink,common)
Change-Id: I6c55735b45cadaf36697eff53e78b5b09afe9dea
2023-09-06 13:28:22 +02:00
Wilfred Asomani
775db4aa3c satellite/payments: filter out draft invoices
This change filters the list of invoices sent to the frontend to only
contain open or paid invoices.

Change-Id: I9ac2640a7587a76b0baf46d941f799e742aa2b3b
2023-09-05 14:22:30 +00:00
Jeremy Wharton
4e3e31a425 satellite/console/consoleweb/consoleapi: refactor api requests in tests
This change refactors the way requests are sent in console API tests,
placing identical logic in a dedicated function to reduce code
duplication.

Change-Id: I7a5ac42d8d68a3fd9a9f8b9d61775659234e883f
2023-09-05 11:17:02 +00:00
Márton Elek
e2006d821c satellite/overlay: change Reliable and KnownReliable
as GetParticipatingNodes and GetNodes, respectively.

We now want these functions to include offline and suspended nodes as
well, so that we can force immediate repair when pieces are out of
placement or in excluded countries. With that change, the old names no
longer made sense.

Change-Id: Icbcbad43dbde0ca8cbc80a4d17a896bb89b078b7
2023-09-02 23:34:50 +00:00
Cameron
6896241933 satellite/console: fix unverified invite email
When checking if invited user is unverified, initialize oldest
with unverified row rather than empty User, because empty User
CreatedAt is zero, so no real user could be created earlier.

Change-Id: I74dd8f7fc82951cbb61071632a74b1a9443b41fe
2023-09-01 15:11:58 -04:00
Michal Niewrzal
c9591e9754 satellite/metainfo: better metabase errors handling
Some errors were returned as metabase errors, not pure drpc
errors because of how rpcstatus.Code method is working. Status
code was returned for errors like metabase context canceled but
we would like to not leak our internals to the client.

Change-Id: I3f0194755f8d7359b1e3d342fa3be3d984019ecb
2023-09-01 11:32:10 +00:00
Jeremy Wharton
b671641a28 satellite/console: update CSP to include storjapi.io
This change updates our content security policy to include the domain
storjapi.io and all of its subdomains.

References #6188

Change-Id: I6f3073bc32aa99626c54caf00bf07d2253ccbb8f
2023-08-31 22:02:45 +00:00
Márton Elek
c202929413
satellite/nodeselection: rename (NodeFilter).MatchInclude to Match
As I learned, the `Include` supposed to communicate that some internal change also "included" to the filters during the check -> filters might be stateful.

But it's not the case any more after 552242387, where we removed the only one stateful filter.

Change-Id: I7c36ddadb2defbfa3b6b67bcc115e4427ba9e083
2023-08-31 16:17:52 +02:00
Wilfred Asomani
dcc4bd0d10 satellite/{console,payments}: freeze/warn storjscan users
This change enables the freezing/warning of users who use storjscan.

Issue: https://github.com/storj/storj/issues/6164

Change-Id: I7b00ee09d6527b3818b72326e9065c82ef5a2ac8
2023-08-31 13:22:21 +00:00
Márton Elek
ca0ea50cba satellite/overlay: remove/deprecate NodeSelectionCache.Disabled
Once uppon a time, at the dawn of the implementation of Storj, when all the nodes are read from the database directly, every time.

After a while --  due to performance reasons -- it has been changed for upload and download: where all the nodes are read for a short period of time, and used from memory.

This is the version which was improved recently to support advanced node selections using placement.

But stil we have an old configuration value `service.config.NodeSelectionCache.Disabled`, and the db based implementation: `service.FindStorageNodesWithPreferences(ctx, req, &service.config.Node)`.

For safety, we need to remove this option, to make sure that we use the cache, which has the advanced features.

This patch was supposed to be a very small one (just removing a method and a config: https://review.dev.storj.io/c/storj/storj/+/11074/1/satellite/overlay/service.go), but it turned out that we need to update a lot of unit tests.

These unit tests used the old implementation (which is not used in production any more).

The tests which used both implementation are just updated to use only the new one
The tests which used only the old implementation are refactored (but keeping the test cases).
Using real unit tests (without DB, working on OSX, fast)

Closes https://github.com/storj/storj/issues/6217

Change-Id: I023f92c7e34235665cf8474513e67b2fcc4763eb
2023-08-31 09:46:29 +00:00
Wilfred Asomani
31ec421299 satellite/db: optimize project usage query
This change addresses an issue where the /charges endpoint will take a
while to respond due to a project having a large number of buckets.
The method and queries involved have been optimized and benchmarks show
a performance improvement.

test name                            old ms/op  new ms/op
Postgres/sum_all_partner_usages          3.659      1.101
Postgres/individual_partner_usages        3.74      1.299
Cockroach/sum_all_partner_usages         7.201      2.872
Cockroach/individual_partner_usages      7.247      2.852

Issue: https://github.com/storj/storj-private/issues/277

Change-Id: Ia5082a2e1c3e91120a9db7b01c18847fe04574fe
2023-08-30 22:08:11 +00:00
Michal Niewrzal
67371c43bd satellite/metainfo: enable UsePendingObjectsTable by project
This small feature will give us ability to test pending_objects table
without enabling it globally.

Change-Id: I802f45987ad329f94adfc0f02957c802b21d8251
2023-08-30 15:44:29 +02:00
Michal Niewrzal
780c0e0b35 satellite/metainfo: adjust ListPendingObjectStreams to pending_objects
table

New method IteratePendingObjectsByKeyNew is used to provide results for
metainfo.ListPendingObjectStreams. This endpoint is used to list
pending objects with the same object key. In this case to support
both tables (objects, pending_objects) we need to do one query per table
and merge results.

Because existing metainfo protobuf API is missing some fields to have
proper listing cursor we are not able to make ListPendingObjectStreams
correct for returning more than single page. We need to fix it
separately.

With this change also turns out that approach to merge results from
listing objects for ListObjects method was wrong and this change is also
fixing this problem.

Handling both tables will be removed at some point and only
pending_objects will be used to look for results.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I8a88a6f885ad529704e6c032f1d97926123c2909
2023-08-30 13:35:54 +00:00
Cameron
a5b1c0432f satellite/console: send unverified user activation link in project invite
When an unverified user is sent a project invitation it contains a
registration link currently. Instead, send an activation link.

github issue: https://github.com/storj/storj/issues/6033

Change-Id: I54b88de8347a2532f7a85372c0c5e4df4bf4eb38
2023-08-29 12:54:09 -04:00
Wilfred Asomani
6219aba40c satellite/{console,consoleweb,consoleapi}: add new endpoint for paged invoices
This change adds a new endpoint for listing invoices for billing history
This endpoint will replace the billing-history endpoint used on the
front end since were only interested in listing invoices.

Issue: https://github.com/storj/storj/issues/5479

Change-Id: I4730f5dc497245c6730e60b7f9986554479d1d3b
2023-08-28 23:24:12 +00:00
Michal Niewrzal
c010e37374 satellite/metainfo: adjust ListObjects to use pending_objects table
Adjust metainfo.ListObjects method to use IteratePendingObjects to
support new pending_objects table. New method will be used only when
we are listing pending objects.

Because until objects table will be free from pending objects we can
have results in both tables we are merging listing results. This also
means that in some (rare?) cases we may return more results than
specified listing limit. This situation is temporary.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I06389145e5d916c532dfdbd3dcc9ef68ef70e515
2023-08-28 16:22:54 +00:00
Ivan Fraixedes
2d8f396eeb private/apigen: Make API base path configurable
Previously the base path for the API was hardcoded to `/api` and the
specified version.

This was not obvious that the generated code was setting that base path
and it was not flexible for serving the API under a different path than
`/api`.

We will likely need to set a different base path if we pretend to serve
the new back office API that we are going to implement alongside the
current admin API until the new back office is fully implemented and
verified that works properly.

This commit also fix add the base path of the endpoints to the
documentation because it was even more confusing for somebody that wants
to use the API having to find out them through looking to the generated
code.

Change-Id: I6efab6b6f3d295129d6f42f7fbba8c2dc19725f4
2023-08-28 14:35:01 +00:00
Michal Niewrzal
b26df035f9 satellite/metainfo: tests for new GetBucketLocation method
Change-Id: I809ebab51606aa9e55dff3c40ef2e865caf06924
2023-08-28 11:49:23 +00:00
Artur M. Wolff
37d6df23fa satellite: implement metainfo.GetBucketLocation endpoint
Updates storj/storj-private#408
Updates storj/storj-private#409

Change-Id: Idaaca74b4a5c9c7907d095e0a3a5f29e52843ce6
2023-08-28 13:48:07 +02:00
Márton Elek
5c12a3406d satellite/nodeselection: improve annotation composability
We would like to make it easier to accept multiple annotations.

Examples:
```
country("GB") && annotation(...)
annotated(annotated(X,...),...)
```

Change-Id: I92e622e8b985b314dadddf83b17976c245eb2069
2023-08-28 09:27:04 +00:00
Jeremy Wharton
2cdc1a973f satellite/console: fix project creation race condition
This change fixes an issue where the project limit could be exceeded if
multiple project creation requests were sent sufficiently close to one
another. This could also be used to bypass project name duplication
checking.

Change-Id: I61cde7abaf25dedc5601c6870275de9938d7b949
2023-08-25 17:18:34 +00:00
Moby von Briesen
d76f059c55 private/apigen: Update generated doc links
Make the link more human-friendly - to contain the text of the group and
endpoint names.

Also link back to list of endpoints from each endpoint.

Change-Id: Ia3e2ebe20b58b5f60ecefe9d35fb8fd90dd4b4d7
2023-08-25 11:51:57 +00:00
Jeremy Wharton
84e75d5994 satellite/satellitedb: remove subquery AOST in console db cleanup funcs
This change fixes an issue where the console DB cleanup chore was never
able to run when using a Cockroach database implementation because of
an inappropriate AS OF SYSTEM TIME clause in the relevant methods.

Resolves #6197

Change-Id: I8456b6df2128678e0eebeb416eb1a955cc9bd706
2023-08-24 21:51:55 +00:00
Wilfred Asomani
fe9f69a757 satellite/{consoleweb,consoleapi}: add cross user api tests
This change adds tests to ensure critical endpoints are not able to be
called by users for other users. It asserts that if cases like that
do happen, a 401 response will be sent.

Issue: https://github.com/storj/storj-private/issues/407

Change-Id: I70097a80f691a7d0fcb0bc5dbce8291144177720
2023-08-24 20:16:20 +00:00
Moby von Briesen
5c155752d2 satellite/analytics: Update lifecyclestage for personal users
Personal users, like business users, should now be classified with
a lifecycle stage of PQL ("product qualified lead") instead of "other"

Change-Id: Iff5139043da1c8e75559302320ff9ca43ea956e5
2023-08-24 09:47:26 -04:00
Moby von Briesen
7fbabbcf16 private/apigen: Add list of endpoints with links
This change makes it easier for someone reading the documentation to see
a full list of supported endpoints, and have direct links to the
details.

Change-Id: I46e2f809cfa2760845898eaa3d99db9066d435ef
2023-08-24 02:45:46 +00:00
Moby von Briesen
5340a351b7 satellite/.../consoleapi/gen: Update README
Remove outdated information from the generated API readme, and add a
link to the generated documentation.

Change-Id: Icc098c81f235464344895d2195444044831aac63
2023-08-23 19:49:35 +00:00
Michal Niewrzal
95a5cfe647 satellite/buckets: handle bucket exists better
In some rare cases when two entities are trying to create the same
bucket at the same time it's possible that we will return internal
error instead of `bucket already exists`. It's because we are not
handling correctly DB error about constraint error. This change checks
if while inserting bucket into DB we got constraint error and propagate
correct error to metainfo API.

Change-Id: Ie6fd2c943b864b4ea7d71e4a162e74dc3510e386
2023-08-23 14:40:31 +00:00
Márton Elek
84ea80c1fd satellite/repair/checker: respect autoExcludeSubnet anntation in checker rangedloop
This patch is a oneliner: rangedloop checker should check the subnets only if it's not turned off with placement annotation.
(see in satellite/repair/checker/observer.go).

But I didn't find any unit test to cover that part, so I had to write one, and I prefered to write it as a unit test not an integration test, which requires a mock repair queue (observer_unit_test.go mock.go).

Because it's small change, I also included a small change: creating a elper method to check if AutoExcludeSubnet annotation is defined

Change-Id: I2666b937074ab57f603b356408ef108cd55bd6fd
2023-08-23 13:45:09 +00:00
Márton Elek
4ccce11893
satellite/overlay: improve realistic placement rule test
10 --> node tag inclusion in raw format
11 --> same, but using same subnet is enabled
12 --> same as 11 but with US restrictions

Change-Id: I20792689e0caf5fe190f566a770d70c3b3824793
2023-08-23 13:56:35 +02:00
Wilfred Asomani
516241e406 cmd,satellite: remove Graphql code and dependencies
This change removes unused GraphQL code. It also updates storj sim code
to use the GraphQL replacement HTTP endpoints and removes the GraphQL
dependency.

Issue: https://github.com/storj/storj/issues/6142

Change-Id: Ie502553706c4b1282cd883a9275ea7332b8fc92d
2023-08-22 12:23:14 +00:00
Paul Willoughby
fdd4be80bf satellite/metainfo: increase default MaxEncryptedObjectKeyLength
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt.  Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.

Updates https://github.com/storj/gateway-mt/issues/335

Change-Id: I38a0fbb0843fd782aeadca85f9a202821421b5a2
2023-08-21 14:03:16 -06:00
Jeremy Wharton
7f9317aa48 satellite/console: fix account creation race condition
This change fixes an issue where multiple unverified users with the
same email address could be created if registration requests were
sent sufficiently close to one another.

Resolves #6156

Change-Id: If8b1a145bcab842ace718119183de59947430463
2023-08-21 16:52:11 +00:00
Márton Elek
b2780b028d satellite/gracefulexit: use placement when gracefulexit pick new nodes
It's quite straightforward change, and AFAIK graceful exit will be decommissioned very soon.

Therefore I didn't create big unit tests, yet. But I can be convinced to invest more time.

Change-Id: Ia588e516d7af5171fa47f9bab100edd3bf2b2cf9
2023-08-21 15:40:05 +00:00
Michal Niewrzal
005fb19a7b satellite/metabase: adjust BucketEmpty to use pending_objects table
Extends metabase.BucketEmpty logic to check also pending_objects
table for any entry.

https://github.com/storj/storj/issues/6057

Change-Id: Ia26c272de24a983b308a0b692e6bd5800487eb98
2023-08-21 15:01:59 +00:00
Michal Niewrzal
16588033fd satellite/metabase: delete bucket deletes also from pending_objects
While deleting bucket we need also to delete pending objects from
pending_objects table.

Part of https://github.com/storj/storj/issues/6048

Change-Id: Icc83eaecf8388704e0b6329c397e8028debcf672
2023-08-21 14:05:13 +00:00
Michal Niewrzal
ae2cba1d23 satellite/metabase: add IteratePendingObjectsByKeyNew method
New metabase method IteratePendingObjectsByKeyNew to iterate
over entries in pending_objects table with the same object key.
Implementation and tests are mostly copy of code for
IteratePendingObjectsByKey. Main difference is that pending_objects
table have StreamID column part of primary key instead Version.

Method will be used to support new table in
metainfo.ListPendingObjectStreams request.

After full transition to pending_objects table we should remove 'New'
suffix from methods names.

Part of https://github.com/storj/storj/issues/6047

Change-Id: Ifc1ecbc534f8510fbd70c4ec676cf2bf8abb94cb
2023-08-21 08:08:03 +00:00
Michal Niewrzal
929dc80091 satellite/metabase: add IteratePendingObjects method
New metabase method IteratePendingObjects to iterate over entries in
pending_objects table. Implementation and tests are mostly copy of
code that we are using to iterate over objects table. Main difference
is that pending_objects table have StreamID column part of primary
key instead Version. Also structure of pending object is smaller
than the one from object table but it's a detail.

Method will be used to support new table in metainfo.ListObjects
request.

Next step will be to port rest of iterator implementation to support
pending_objects table in metainfo.ListPendingObjectStreams.

Part of https://github.com/storj/storj/issues/6047

Change-Id: Ia578182f88840539f3668d4a242953e061eace02
2023-08-21 08:07:16 +00:00
Michal Niewrzal
5a8ef89824 satellite/{metainfo,metabase}: delete from pending_objects table
We are deleting pending objects while aborting multipart upload. We are
using metainfo BeginDeleteObject to do that. This change starts using
DeletePendingObjectNew to delete entry from pending_objects table when
request indicates that object is in this table.

Part of https://github.com/storj/storj/issues/6048

Change-Id: I4478a9c13c8e3db48dc5de3087ef03d1b4c47a5c
2023-08-21 08:06:23 +00:00
Wilfred Asomani
33c0a82fb7 satellite/console: add create project http endpoint
This change adds an HTTP endpoint for creating projects, to be used in
place of the GraphQL version.

Issue: https://github.com/storj/storj/issues/6195

Change-Id: I0377353418df7c152db6a935e99a3ea7ab4ce625
2023-08-21 06:58:03 +00:00
Márton Elek
f0afe0d2ea satelite/repairer: ignore declumping when subnet filtering is turned off with filter annotation
+ restoring the functionality of repairer.doPlacementCheck

Change-Id: I75521f2da280758345face07eeea661765717318
2023-08-18 09:35:38 +00:00
Márton Elek
5522423871 satellite/nodeselection: remove AutoExcludeSubnet filter
It's statefull, therefore it can hit naive users. (NodeFilters couldn't be reused for more than one iterations).

But looks like we don't need it, as `SelectBySubnet` doest the same job.

Change-Id: Ie85b7f9c2bd9a47293f4e3b359f8b619215c7649
2023-08-18 08:31:00 +00:00
Márton Elek
b218002752 satellite/overlay/placement: improve placement configurability with &&, placement, region and ! support in country
This patch makes it easier to configure existing placement rules only with string.

 1. placement(n) rule can be used to reuse earlier definitions
 2 .&& can be used in addition to all(n1,n2)
 3. country(c) accepts exclusions (like '!RU'), regions ('EU','EEA'), all and none

See the 'full example' unit test, which uses all of these, in a realistic example.

https://github.com/storj/storj/issues/6126

Change-Id: Ica76f016ebd002eb7ea8103d4258bacd6a6d77bf
2023-08-17 16:12:53 +00:00
Egon Elbre
b5e1e5a9e2 private/apigen: use typescript in docs markdown
Fixes https://github.com/storj/storj/issues/6187

Change-Id: I34016513f29a9538343f0909951ab1a4605bb585
2023-08-17 15:33:21 +00:00
Márton Elek
9ddc8b4ca3 satellite/repair: piecescheck.OutOfPlacementPiecesSet should not contain offline nodes
When we check the availability of the pieces, we do:

```
result.NumUnhealthyRetrievable = len(result.ClumpedPiecesSet) + len(result.OutOfPlacementPiecesSet)
// + some magic if there are overlaps between them
numHealthy := len(pieces) - len(piecesCheck.MissingPiecesSet) - piecesCheck.NumUnhealthyRetrievable
```

This works only if OutOfPlacementPieceSet doesn't contain the offline nodes (which are already included in MissingPieceSet).

But `result.OutOfPlacementPieces.Set` should include all the nodes (even offline), as in case of lucky conditions, we are able to remove those pieces from DB.

The solution is to remove all offline nodes from `NumUnhealthyRetrievable`.

Change-Id: I90baa0396352dd040e1e1516314b3271f8712034
2023-08-16 17:35:10 +00:00
Márton Elek
de7aabc8c9 satellite/{repair,rangedloop,overlay}: fix node tag placement selection for repair
This patch fixes the node tag based placement of rangedloop/repairchecker + repair process.

The main change is just adding the node tags for Reliable and KnownReliabel database calls + adding new tests to prove, it works.

https://github.com/storj/storj/issues/6126

Change-Id: I245d654a18c1d61b2c72df49afa0718d0de76da1
2023-08-16 15:45:41 +00:00
Márton Elek
da08117fcd satellite/~placement: do not ignore placement check for placement=0
There are cases when we would like to override the default placement=0 rule.

For example when we would like to exclude tagged nodes from the selection (by default).

Therefore we couldn't use a shortcut any more, we should always check the placement rules, even if we use placement=0.

TODO: we need to update common, and rename `EveryCountry` to `DefaultPlacement`, just to avoid confusion.

https://github.com/storj/storj/issues/6126

Change-Id: Iba6c655bd623e04351ea7ff91fd741785dc193e4
2023-08-16 07:06:56 +00:00
Márton Elek
c08792f066 satellite/overlay: implement an exclude filter for placement configuration
https://github.com/storj/storj/issues/6126

Change-Id: I05215b5d46bec958001cc020edf1fa97b00d3299
2023-08-15 17:29:29 +00:00
Márton Elek
0e17b1018c satellite/{nodeselection,overlay}: support annotations on node filters
Change-Id: I844d8a25042750aae189175842113e2f052d5b17
2023-08-15 16:49:57 +00:00
littleskunk
792bb113bc
satellite/console: Enable gallery view and limits area by default (#6177) 2023-08-14 17:05:19 +02:00
Moby von Briesen
b9206b1844 satellite/console: support hosting Vuetify POC on subdomain
This change allows you to host the vuetify app on <x>.example.com where
the main app is hosted on example.com. A configuration is added to
specify an exact subdomain for cookies. For example, if my production
app is hosted on us1.storj.io and my vuetify app is hosted on
vuetify.us1.storj.io, the cookie domain should be set to ".us1.storj.io"
so that any authentication cookie is accessible to lower-level
subdomains.

Since the vuetify app does not currently support login/signup on its
own, it is still required to first login to the main satellite UI, then
navigate to the Vuetify app after the session cookie is set.

If the "vuetifypoc" prefix is not desirable when using subdomain hosting
for vuetify, the VITE_VUETIFY_PREFIX variable can be modified in
web/satellite/.env before running `npm run build-vuetify`. For now, we
should keep this prefix because it makes developing on the vuetify app
significantly easier if subdomains are not being used.

Issue: https://github.com/storj/storj/issues/6144

Change-Id: Iba1a5737892c8ee8f38148a17b94e3222f8798e6
2023-08-14 13:15:41 +00:00
Márton Elek
ceb7b7375c satellite/geoip: exclude nodes with represented_contry from geofencing
Change-Id: I80b8e5d98f46559b158a26c47fff0586b97aff79
2023-08-14 12:36:33 +00:00
Cameron
969599b60b satellite/console: delete project members endpoint
This commit adds a new endpoint on the console api to delete project
members and invitations.

issue: https://github.com/storj/storj/issues/6136

Change-Id: I980bb97afd1ed2ed8f0f27cc2e8dc1d80d7eef05
2023-08-11 13:48:33 -04:00
Egon Elbre
0335697664 satellite/accounting/live: ensure we don't panic when we get nil
Change-Id: I99b6b94d37a9856e8a705679d117b42d16326e81
2023-08-10 19:41:57 +03:00
Egon Elbre
1f261bcc70 go.mod: bump lang to 1.19 and common
Change-Id: I8d91f97d786456da29ebe89a78412c50efbb8ccc
2023-08-10 18:41:15 +03:00
Wilfred Asomani
5bdb7bc3f0 satellite/console: add update project http endpoint
This change adds an endpoint update projects, to be used in place of
the graphql alternative.

Issue: https://github.com/storj/storj/issues/6134

Change-Id: I26c04f4400f71721cbddb7f64405e6c9b78edb4d
2023-08-09 16:07:51 +00:00
Egon Elbre
dc41978743 all: fix golangci failures
Change-Id: I07421388d53c837e35a4727cead26fc21c324d04
2023-08-09 11:44:44 +03:00
Clement Sam
9e3d54fec4 satellite/admin: extend API to allow setting and deleting account level geofence
Issue: https://github.com/storj/storj-private/issues/357
Change-Id: I04589e18214e7090ccd686fd531066d942afa6ed
2023-08-09 03:34:37 +00:00
Jeremy Wharton
a00ec7af40 satellite/console: create http endpoint for getting bucket usage totals
This change introduces an HTTP endpoint for retrieving bucket usage
totals. In the future, this will replace its GraphQL counterpart.

References #6141

Change-Id: Ic6a0069a7e58b90dc2b6c55f164393f036c6acf4
2023-08-08 21:26:17 +00:00
Cameron
683119b835 satellite/console: get project members endpoint
This commit adds a new endpoint on the satellite console api to get
project members and invitations.

issue: https://github.com/storj/storj/issues/6137

Change-Id: I66cb064eeaffb1c34878462b3e6b3be8f3629f4e
2023-08-08 14:10:29 -04:00
Michal Niewrzal
887209bc24 satellite/metabase: fix CommitObject query for postgres
Query was not working for postgres because types were not correctly set.

Change-Id: Ic898aa37b71a4754e4ebe82085f3563fc954616a
2023-08-08 15:31:44 +00:00
Wilfred Asomani
34e1caa55a satellite: add get user paged projects http endpoint
This change adds an endpoint to get a user's projects, similar to
the OwnedProjects GraphQL query.
The console.ProjectInfo struct has been renamed to UpsertProjectInfo
to more accurately reflect its use.

Issue: https://github.com/storj/storj/issues/6135

Change-Id: I802fe4694a5cc75a9df2b565476f6e6f473431d4
2023-08-08 14:02:53 +00:00
Michal Niewrzal
7be844351d satellite/metainfo: remove ServerSideCopyDuplicateMetadata
https://github.com/storj/storj/issues/5891

Change-Id: Ib5440169107acca6e832c2280e1ad12dfd380f28
2023-08-08 12:15:10 +00:00
Michal Niewrzal
9550b5f4a5 satellite/metabase: drop object deletion code for copy with references
With this change we are removing code responsible for deleting objects
and supporting server side copies created with references. In practice
we are restoring delete queries that we had before server side copy
implementation (with small exception, see bellow).

From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.

https://github.com/storj/storj/issues/5891

Change-Id: Iee4e7a9688cff27a60fb95d60dd233d996f41c85
2023-08-08 11:31:12 +00:00
Michal Niewrzal
03f8ad323d satellite/metabase: remove segment_copies support from ListSegments
We don't need to support segment copies with references anymore.
We migrated to copies where all metadata are copied from original
segment to copy.

https://github.com/storj/storj/issues/5891

Change-Id: Ic91dc21b0386ddf5c51aea45530024cd463e8ba9
2023-08-08 11:21:08 +00:00