Commit Graph

3406 Commits

Author SHA1 Message Date
littleskunk
792bb113bc
satellite/console: Enable gallery view and limits area by default (#6177) 2023-08-14 17:05:19 +02:00
Moby von Briesen
b9206b1844 satellite/console: support hosting Vuetify POC on subdomain
This change allows you to host the vuetify app on <x>.example.com where
the main app is hosted on example.com. A configuration is added to
specify an exact subdomain for cookies. For example, if my production
app is hosted on us1.storj.io and my vuetify app is hosted on
vuetify.us1.storj.io, the cookie domain should be set to ".us1.storj.io"
so that any authentication cookie is accessible to lower-level
subdomains.

Since the vuetify app does not currently support login/signup on its
own, it is still required to first login to the main satellite UI, then
navigate to the Vuetify app after the session cookie is set.

If the "vuetifypoc" prefix is not desirable when using subdomain hosting
for vuetify, the VITE_VUETIFY_PREFIX variable can be modified in
web/satellite/.env before running `npm run build-vuetify`. For now, we
should keep this prefix because it makes developing on the vuetify app
significantly easier if subdomains are not being used.

Issue: https://github.com/storj/storj/issues/6144

Change-Id: Iba1a5737892c8ee8f38148a17b94e3222f8798e6
2023-08-14 13:15:41 +00:00
Márton Elek
ceb7b7375c satellite/geoip: exclude nodes with represented_contry from geofencing
Change-Id: I80b8e5d98f46559b158a26c47fff0586b97aff79
2023-08-14 12:36:33 +00:00
Cameron
969599b60b satellite/console: delete project members endpoint
This commit adds a new endpoint on the console api to delete project
members and invitations.

issue: https://github.com/storj/storj/issues/6136

Change-Id: I980bb97afd1ed2ed8f0f27cc2e8dc1d80d7eef05
2023-08-11 13:48:33 -04:00
Egon Elbre
0335697664 satellite/accounting/live: ensure we don't panic when we get nil
Change-Id: I99b6b94d37a9856e8a705679d117b42d16326e81
2023-08-10 19:41:57 +03:00
Egon Elbre
1f261bcc70 go.mod: bump lang to 1.19 and common
Change-Id: I8d91f97d786456da29ebe89a78412c50efbb8ccc
2023-08-10 18:41:15 +03:00
Wilfred Asomani
5bdb7bc3f0 satellite/console: add update project http endpoint
This change adds an endpoint update projects, to be used in place of
the graphql alternative.

Issue: https://github.com/storj/storj/issues/6134

Change-Id: I26c04f4400f71721cbddb7f64405e6c9b78edb4d
2023-08-09 16:07:51 +00:00
Egon Elbre
dc41978743 all: fix golangci failures
Change-Id: I07421388d53c837e35a4727cead26fc21c324d04
2023-08-09 11:44:44 +03:00
Clement Sam
9e3d54fec4 satellite/admin: extend API to allow setting and deleting account level geofence
Issue: https://github.com/storj/storj-private/issues/357
Change-Id: I04589e18214e7090ccd686fd531066d942afa6ed
2023-08-09 03:34:37 +00:00
Jeremy Wharton
a00ec7af40 satellite/console: create http endpoint for getting bucket usage totals
This change introduces an HTTP endpoint for retrieving bucket usage
totals. In the future, this will replace its GraphQL counterpart.

References #6141

Change-Id: Ic6a0069a7e58b90dc2b6c55f164393f036c6acf4
2023-08-08 21:26:17 +00:00
Cameron
683119b835 satellite/console: get project members endpoint
This commit adds a new endpoint on the satellite console api to get
project members and invitations.

issue: https://github.com/storj/storj/issues/6137

Change-Id: I66cb064eeaffb1c34878462b3e6b3be8f3629f4e
2023-08-08 14:10:29 -04:00
Michal Niewrzal
887209bc24 satellite/metabase: fix CommitObject query for postgres
Query was not working for postgres because types were not correctly set.

Change-Id: Ic898aa37b71a4754e4ebe82085f3563fc954616a
2023-08-08 15:31:44 +00:00
Wilfred Asomani
34e1caa55a satellite: add get user paged projects http endpoint
This change adds an endpoint to get a user's projects, similar to
the OwnedProjects GraphQL query.
The console.ProjectInfo struct has been renamed to UpsertProjectInfo
to more accurately reflect its use.

Issue: https://github.com/storj/storj/issues/6135

Change-Id: I802fe4694a5cc75a9df2b565476f6e6f473431d4
2023-08-08 14:02:53 +00:00
Michal Niewrzal
7be844351d satellite/metainfo: remove ServerSideCopyDuplicateMetadata
https://github.com/storj/storj/issues/5891

Change-Id: Ib5440169107acca6e832c2280e1ad12dfd380f28
2023-08-08 12:15:10 +00:00
Michal Niewrzal
9550b5f4a5 satellite/metabase: drop object deletion code for copy with references
With this change we are removing code responsible for deleting objects
and supporting server side copies created with references. In practice
we are restoring delete queries that we had before server side copy
implementation (with small exception, see bellow).

From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.

https://github.com/storj/storj/issues/5891

Change-Id: Iee4e7a9688cff27a60fb95d60dd233d996f41c85
2023-08-08 11:31:12 +00:00
Michal Niewrzal
03f8ad323d satellite/metabase: remove segment_copies support from ListSegments
We don't need to support segment copies with references anymore.
We migrated to copies where all metadata are copied from original
segment to copy.

https://github.com/storj/storj/issues/5891

Change-Id: Ic91dc21b0386ddf5c51aea45530024cd463e8ba9
2023-08-08 11:21:08 +00:00
Vitalii
6c3300c522 satellite/console: add delete api keys http endpoint
This change adds an endpoint to delete API keys, similar to GraphQL mutation.

Issue:
https://github.com/storj/storj/issues/6140

Change-Id: Ia4a808222a057a199d803d8ea6b944c411a4dc8d
2023-08-08 10:46:54 +00:00
Michal Niewrzal
a5cbec7b3b satellite/metabase: drop bucket deletion code for copy with references
With this change we are removing code responsible to handle deleting
bucket and supporting server side copies created with references. In practice we are restoring delete queries that we had before server side copy implementation (with small exception, see bellow).

From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.

https://github.com/storj/storj/issues/5891

Change-Id: If866d9f3a1b01e9ebd9b49c4740a6425ba06dd43
2023-08-08 09:07:44 +00:00
Vitalii
f57bc81ce7 satellite/console: add create api key http endpoint
This change adds an endpoint to create new API key, similar to GraphQL mutation.

Issue:
https://github.com/storj/storj/issues/6139

Change-Id: I2b35d680fa8e019666c811ad3bdf16201e3b8946
2023-08-07 23:58:38 +00:00
Vitalii
256bd18120 satellite/console: add get project api keys http endpoint
This change adds an endpoint to get paged project API keys, similar to GraphQL query.

Issue:
https://github.com/storj/storj/issues/6138

Change-Id: I5dea9e4ac61e798cc8a2e56a2755d722c1b66bfa
2023-08-07 18:26:31 +00:00
Wilfred Asomani
70f6b60d91 satellite/console: add get user projects http endpoint
This change adds an endpoint to get a user's projects, similar to
the MyProjects GraphQL query.

Issue: https://github.com/storj/storj/issues/6132

Change-Id: I91feb5a1ee8c1231a8a5e6de9b8dc5b256f857c5
2023-08-07 16:05:09 +00:00
paul cannon
6e46a926bb satellite/nodeselection: expand SelectedNode
In the repair subsystem, it is necessary to acquire several extra
properties of nodes that are holding pieces of things or may be
selected to hold pieces. We need to know if a node is 'online' (the
definition of "online" may change somewhat depending on the situation),
if a node is in the process of graceful exit, and whether a node is
suspended. We can't just filter out nodes with all of these properties,
because sometimes we need to know properties about nodes even when the
nodes are suspended or gracefully exiting.

I thought the best way to do this was to add fields to SelectedNode,
and (to avoid any confusion) arrange for the added fields to be
populated wherever SelectedNode is returned, whether or not the new
fields are necessarily going to be used.

If people would rather I use a separate type from SelectedNode, I can do
that instead.

Change-Id: I7804a0e0a15cfe34c8ff47a227175ea5862a4ebc
2023-08-07 12:44:49 +00:00
Márton Elek
0b02a48a10
satellite/nodeselection: SelectBySubnet should use placement filters for all nodes
Current node selection logic (in case of using SelectBySubnet):

 1. selects one subnet randomly
 2. selects one node randomly from the subnet
 3. applies the placement NodeFilters to the node and ignore it, if doesn't match

This logic is wrong:

 1. Imagine that we have a subnet with two DE and one GB nodes.
 2. We would like to select DE nodes
 2. In case of GB node is selected (randomly) in step2, step3 will ignore the subnet, even if there are good (DE) nodes in there.

Change-Id: I7673f52c89b46e0cc7b20a9b74137dc689d6c17e
2023-08-04 10:48:15 +02:00
Michal Niewrzal
03c52f184e satellite/metabase: adjust CommitObject to use pending_objects table
Change is adjusting CommitObject to use `pending_objects` table to
commit object.

Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table during commit.

General goal is to support both tables until `objects` table will be
free from pending objects.

Part of https://github.com/storj/storj/issues/6046

Change-Id: I2ebe0cd6b446727c98c8e210d4d00504dd0dacb6
2023-08-03 10:21:22 +00:00
Márton Elek
2ed08922d9 satellite/api: configuration option to set additional node tag authorities
Change-Id: Iba387e37cb586ce1378668c99beb3b4db8a9064d
2023-08-03 08:34:02 +00:00
Michal Niewrzal
f40805763e satellite/metabase: adjust segment commit to use pending_objects table
Change is adjusting CommitSegment to check pending object existence in
`pending_objects` or `objects` table.

Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.

General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.

Part of https://github.com/storj/storj/issues/6046

Change-Id: I954444a53b4733ae6fc909420573242b02746787
2023-08-02 16:56:25 +00:00
Michal Niewrzal
7b2006a883 satellite/metabase: adjust BeginSegment to use pending_objects table
Change is adjusting BeginSegment to check pending object existence in
`pending_objects` or `objects` table.

Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.

General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.

Part of https://github.com/storj/storj/issues/6046

Change-Id: I08aaa605c23d82695fde352fdbd0a7fd11f46bb5
2023-08-02 15:30:23 +00:00
Michal Niewrzal
cebf255d64 satellite/metabase: adjust BeginObjectNextVersion to use pending_objects
Change is adjusting BeginObjectNextVersion to create pending object in
`pending_objects` or `objects` table depends on configuration. This is
first change to move pending objects from objects table.

General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.

To be able to decide if we need to use `pending_objects` table or
`objects` table we extend satellite stream id to keep that information
for later use.

BeginObjectExactVersion will be not adjusted because at the moment it's
used only in tests.

Part of https://github.com/storj/storj/issues/6046

Change-Id: Ibf21965f63cca5e1775469994a29f1fd1261af4e
2023-08-02 14:42:26 +00:00
Márton Elek
63c8cfe4c3 satellite/nodeselection: remove CountryCodeExclude
We don't need it any more, as CountryCode uses location.Set, which supports exclusion (`Without`).

Change-Id: Ie311ae19fefa0bc9a0161496af1233ef4a6607df
2023-08-02 09:48:36 +00:00
Márton Elek
f7b39aaed4 satellite/nodeselection: remove stats/size from nodeselection state
stats/size/count is not used by any production code, and it's not required, as we can assert the state with other checks.

real motivation: next commits will make the Selector of the State configurable, therefore we won't have one single Stat, it depends on the request parameters.

(we plan to support both network and id based randomization)

Change-Id: I631828fc0046d2fef5b7a674fc0268a0446e9655
2023-08-01 18:29:41 +00:00
Wilfred Asomani
7c65c0cea5 satellite/{db,console,payments}: unfreeze user with no failed invoices
This change extends the autofreeze chore to go through users who have
been warned/frozen to check if they have no failed invoices. If they do
not, this extension unwarns/unfreezes them.

Issue: https://github.com/storj/storj/issues/6077

Change-Id: I570b1d4b2e29574bd8b9ae37eb2d4fb41d178336
2023-08-01 17:54:39 +00:00
Michal Niewrzal
65aa9c11bc satellite/gc/sender: adjust some defaults
Change-Id: I94f5a18bc2d4218e834259b2ba22f3578745c975
2023-08-01 14:27:44 +00:00
Márton Elek
6f002f4220
satellite/overlay: NR placement should exclude nodes without geofencing information
https://github.com/storj/storj-private/issues/378

Change-Id: If2af02083496e5a8eefe27beabb406388ee50644
2023-07-31 09:55:54 +02:00
Wilfred Asomani
c934974652 satellite/{console,web}: update error handling
This change updates how API errors are handled and sent.

Change-Id: Ia4c71eeb6f2d009a47b59ce77a23f70b8b10f6dc
2023-07-27 21:01:01 +00:00
Clement Sam
cc12a48c24 satellite/admin: extend admin API to allow setting and deleting geofence for projects
Issue: https://github.com/storj/storj-private/issues/357
Change-Id: Ib59319581641f1f5da71c629143e12f11eb04925
2023-07-27 11:40:26 +00:00
Michal Niewrzal
4cc167a6bd satellite/metabase: use better label for ignoring FTS queries
For some test queries we are using workaround to filter them out from
full table scan detection. To avoid confustion what is this all about
we are changing label to be more descriptive.

Change-Id: I41a744e8faf400e3e8de7e416d8f4242f9093fce
2023-07-26 20:01:38 +00:00
Michal Niewrzal
3d9c217627 satellite/metabase: add pending_objects table
This change adds only schema definition of pending_objects table and
small amount of supporting code which will be useful for testing later.

With this table we would like to achieve two major things:
* simplify `objects` table, before we will start working on object
versioning
* gain performance by removing need to filter `objects` results with `status` column, which is not indexed and we would like to avoid that

https://github.com/storj/storj/issues/6045

Change-Id: I6097ce1c644a8a3dad13185915fe01989ad41d90
2023-07-26 20:00:58 +00:00
Vitalii
6d94d6a681 satellite/{console, accounting}: fix project bandwidth calculation which is shown on project dashboard
Another try to fix calculation of used bandwidth which is displayed on Project Dashboard.
This change sums up allocated-dead traffic for the last 3 days and settled traffic for the period which is earlier than 3 days ago.

Issue:
https://github.com/storj/storj-private/issues/293

Change-Id: I91e652eba69f81bd21e0d053ac170e2b926b3cb4
2023-07-26 06:14:48 +00:00
Michal Niewrzal
73a279235a satellite/accounting/live: get project totals in batches
After infrastructure changes redis instance is not neccessay close
to core instance (where tally is calculated) and round trips to get
data from redis can be very costly. From less than hour calculation can take few hours for larger satellite.

This change combines 'segment' and 'storage' usage requests into
batches to reduce latency impact on tally calculation.

https://github.com/storj/storj/issues/5800

Change-Id: I87e57ec09e88fd167060a4ed51dc8b0274a095c5
2023-07-24 14:13:04 +00:00
Wilfred Asomani
8ed4c573db satellite/admin: add endpoint to unwarn user
This change enables the admin UI to remove the warning status of users.

resolves: storj-private/issues/342

Change-Id: Ib960ffb33fdabc045884ce7fa2c55c3553db0fb0
2023-07-21 17:10:09 +00:00
Vitalii
6e4044b245 satellite/payments: slightly refactor user upgrade observer of billing chore
Resolves post-merge comments from here
https://review.dev.storj.io/c/storj/storj/+/10780

Reworked some definitions. Added checks and comments.

Change-Id: I19c63804ad1f30a1ffd8cb87e96f43deed20a685
2023-07-21 03:43:42 +00:00
Wilfred Asomani
2c934d1cfd satellite/console: trigger invoice payment on card remove
This change triggers payment when a card is removed.

Issue: https://github.com/storj/storj/issues/6056

Change-Id: Ia1f2b49609ec9d111cbffb0757bec0b513abb79d
2023-07-20 15:46:50 +00:00
Egon Elbre
7d1031feda satellite/accounting: fix duplicate code-block
Change-Id: Ic6de7337a5ff9a2fec73aa3d8a74ba5b9076a8c4
2023-07-20 07:58:53 +00:00
Vitalii
5317135416 satellite/payments: fix config value for auto upgrade user tier flow
Fixed config value which indicates how many base units of US micro dollars are needed to auto upgrade user to paid tier.

Change-Id: I22821ac22fc3eaeeea21c6dec4e6912025df63aa
2023-07-19 10:43:07 +00:00
Wilfred Asomani
7cc873a62a satellite/payments: prevent removing other users' cards
This change patches a loophole that allowed accounts to remove cards
that belong to other users.

Closes #storj/storj-private#326

Change-Id: I33e9efe5c9cdb03aa48ad4c6b1d3283c396a7890
2023-07-19 09:44:06 +00:00
Egon Elbre
23631dc8bb satellite/accounting: fix TestProjectSegmentLimit*
Tally ensures that live accounting has the latest information,
however, when there are concurrent updates to live-accounting
it may by off by a few segments. Disable tally for those tests.

Change-Id: I6fa8a1794334bba093e18f29cb76e7b8d1244979
2023-07-18 14:26:05 +00:00
Egon Elbre
b1e7d70a86 satellite/payments/billing: fix test
`time.Now()` can return the same value when called sequentially.

Change-Id: I800c7696b919ad073e4558fb51c8d2eb4a04f05e
2023-07-18 16:25:42 +03:00
Vitalii
583ad54d86 satellite/{payments, console}: added functionality to get wallet's transactions (including pending)
Added new functionality to query storjscan for all wallet transactions (including pending).
Added new endpoint to query all wallet transactions.

Issue:
https://github.com/storj/storj/issues/5978

Change-Id: Id15fddfc9c95efcaa32aa21403cb177f9297e1ab
2023-07-18 11:09:29 +00:00
Vitalii
2ee0195eba satellite/payments: extend billing chore functionality to upgrade user
Added new observer for billing chore to check user's balance and upgrade their account if balance is more than or equal to needed amount for upgrade.
Added new config value which stands for needed amount of base units of US micro dollars needed to upgrade user.

Issue:
https://github.com/storj/storj/issues/5978

Change-Id: Ic3992cd3114397bfdd9e231ca090ff21ca66648b
2023-07-18 13:15:02 +03:00
Michal Niewrzal
0303920da7 satellite/metainfo: remove unused method
Change-Id: I08e307e6909cdc46951c5f3112d77a685e67fe2e
2023-07-18 08:45:29 +00:00