Commit Graph

8372 Commits

Author SHA1 Message Date
paul cannon
6e46a926bb satellite/nodeselection: expand SelectedNode
In the repair subsystem, it is necessary to acquire several extra
properties of nodes that are holding pieces of things or may be
selected to hold pieces. We need to know if a node is 'online' (the
definition of "online" may change somewhat depending on the situation),
if a node is in the process of graceful exit, and whether a node is
suspended. We can't just filter out nodes with all of these properties,
because sometimes we need to know properties about nodes even when the
nodes are suspended or gracefully exiting.

I thought the best way to do this was to add fields to SelectedNode,
and (to avoid any confusion) arrange for the added fields to be
populated wherever SelectedNode is returned, whether or not the new
fields are necessarily going to be used.

If people would rather I use a separate type from SelectedNode, I can do
that instead.

Change-Id: I7804a0e0a15cfe34c8ff47a227175ea5862a4ebc
2023-08-07 12:44:49 +00:00
Márton Elek
0b02a48a10
satellite/nodeselection: SelectBySubnet should use placement filters for all nodes
Current node selection logic (in case of using SelectBySubnet):

 1. selects one subnet randomly
 2. selects one node randomly from the subnet
 3. applies the placement NodeFilters to the node and ignore it, if doesn't match

This logic is wrong:

 1. Imagine that we have a subnet with two DE and one GB nodes.
 2. We would like to select DE nodes
 2. In case of GB node is selected (randomly) in step2, step3 will ignore the subnet, even if there are good (DE) nodes in there.

Change-Id: I7673f52c89b46e0cc7b20a9b74137dc689d6c17e
2023-08-04 10:48:15 +02:00
Michal Niewrzal
03c52f184e satellite/metabase: adjust CommitObject to use pending_objects table
Change is adjusting CommitObject to use `pending_objects` table to
commit object.

Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table during commit.

General goal is to support both tables until `objects` table will be
free from pending objects.

Part of https://github.com/storj/storj/issues/6046

Change-Id: I2ebe0cd6b446727c98c8e210d4d00504dd0dacb6
2023-08-03 10:21:22 +00:00
Márton Elek
2ed08922d9 satellite/api: configuration option to set additional node tag authorities
Change-Id: Iba387e37cb586ce1378668c99beb3b4db8a9064d
2023-08-03 08:34:02 +00:00
Wilfred Asomani
034542db8f web/satellite/vuetify: add enable MFA dialog
Added enable MFA dialog and functionality.

Issue: https://github.com/storj/storj/issues/6091

Change-Id: Idf2f27937549b9bb709ddcc70ffa4611beea7b78
2023-08-02 17:31:02 +00:00
Michal Niewrzal
f40805763e satellite/metabase: adjust segment commit to use pending_objects table
Change is adjusting CommitSegment to check pending object existence in
`pending_objects` or `objects` table.

Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.

General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.

Part of https://github.com/storj/storj/issues/6046

Change-Id: I954444a53b4733ae6fc909420573242b02746787
2023-08-02 16:56:25 +00:00
Cameron
fac522d8dd web/satellite: implement vuetify poc project dashboard
issue: https://github.com/storj/storj/issues/6060

Change-Id: I40497a39afacd4787468e1259225ddd75e78295b
2023-08-02 16:22:22 +00:00
Michal Niewrzal
7b2006a883 satellite/metabase: adjust BeginSegment to use pending_objects table
Change is adjusting BeginSegment to check pending object existence in
`pending_objects` or `objects` table.

Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.

General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.

Part of https://github.com/storj/storj/issues/6046

Change-Id: I08aaa605c23d82695fde352fdbd0a7fd11f46bb5
2023-08-02 15:30:23 +00:00
Michal Niewrzal
cebf255d64 satellite/metabase: adjust BeginObjectNextVersion to use pending_objects
Change is adjusting BeginObjectNextVersion to create pending object in
`pending_objects` or `objects` table depends on configuration. This is
first change to move pending objects from objects table.

General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.

To be able to decide if we need to use `pending_objects` table or
`objects` table we extend satellite stream id to keep that information
for later use.

BeginObjectExactVersion will be not adjusted because at the moment it's
used only in tests.

Part of https://github.com/storj/storj/issues/6046

Change-Id: Ibf21965f63cca5e1775469994a29f1fd1261af4e
2023-08-02 14:42:26 +00:00
Jeremy Wharton
391a63f9fa web/satellite: bump Vuetify to v3.3.10
This change updates the Vuetify dependency version so that we may use
new components.

Change-Id: Ib4ae4db3b50c267179503ff866077379d52b6d07
2023-08-02 13:46:20 +00:00
Vitalii
2d2f1b858e web/satellite: show pending transactions during token upgrade flow
Made UI updates to reflect pending token payments during upgrade account flow.
So pending transactions with confirmations count are displayed on Add STORJ Tokens step of upgrade flow.
While this modal is open we make a request to storjscan once in 20 seconds to get recent confirmations count.
When all transactions become confirmed we show success view where a sum of STORJ tokens received is displayed.

Issue:
https://github.com/storj/storj/issues/5978

Change-Id: Icfdc1e5080ed58cea1822cb7d85551ba8064c636
2023-08-02 15:39:25 +03:00
Márton Elek
63c8cfe4c3 satellite/nodeselection: remove CountryCodeExclude
We don't need it any more, as CountryCode uses location.Set, which supports exclusion (`Without`).

Change-Id: Ie311ae19fefa0bc9a0161496af1233ef4a6607df
2023-08-02 09:48:36 +00:00
Márton Elek
f7b39aaed4 satellite/nodeselection: remove stats/size from nodeselection state
stats/size/count is not used by any production code, and it's not required, as we can assert the state with other checks.

real motivation: next commits will make the Selector of the State configurable, therefore we won't have one single Stat, it depends on the request parameters.

(we plan to support both network and id based randomization)

Change-Id: I631828fc0046d2fef5b7a674fc0268a0446e9655
2023-08-01 18:29:41 +00:00
Wilfred Asomani
7c65c0cea5 satellite/{db,console,payments}: unfreeze user with no failed invoices
This change extends the autofreeze chore to go through users who have
been warned/frozen to check if they have no failed invoices. If they do
not, this extension unwarns/unfreezes them.

Issue: https://github.com/storj/storj/issues/6077

Change-Id: I570b1d4b2e29574bd8b9ae37eb2d4fb41d178336
2023-08-01 17:54:39 +00:00
JT Olio
e78658d174 go.mod: use latest maxmind-golang library
Change-Id: I6f06785a096d7d05b175d468482db658b7436a87
2023-08-01 15:22:35 +00:00
Michal Niewrzal
65aa9c11bc satellite/gc/sender: adjust some defaults
Change-Id: I94f5a18bc2d4218e834259b2ba22f3578745c975
2023-08-01 14:27:44 +00:00
Vitalii
59034ca094 web/satellite: upgraded aws-sdk dependencies to resolve vulnerabilities
Upgraded package-lock version to correspond to node v18+.
Upgraded aws-sdk dependencies to resolve vulnerabilities.
Also, fixed typing errors in object browser pinia module.

Change-Id: I35e6e219e66f98ca167ccb4ac57ad07ac99efff1
2023-08-01 13:41:15 +00:00
Vitalii
d40091a3cb web/satellite/vuetify: change name dialog
Added change name dialog for vuetify app and wired it up with backend.

Issue:
https://github.com/storj/storj/issues/6088

Change-Id: I37af595c9e48034d59ead051918bbb4ec5e6ff31
2023-08-01 14:26:16 +03:00
Wilfred Asomani
af69e20cc8 web/satellite: fix errant pagination
This change fixes an issue where changing the size of a page will also
change the current page. It uses a more accurate calculation to
determine if the new size and current page will result in a page index
error.

Issue: https://github.com/storj/storj/issues/6094

Change-Id: Ie62f79191eb34ccd75ccd42b923b8866fe4e4d7f
2023-07-31 22:43:52 +00:00
Vitalii
ceef4b8362 web/satellite/vuetify: added change password dialog
Added change password dialog for vuetify app and wired it up with backend.

Issue:
https://github.com/storj/storj/issues/6089

Change-Id: Ib1dffa947b65c299b278a48ed1491a623895a0bd
2023-07-31 17:03:26 +03:00
Márton Elek
6f002f4220
satellite/overlay: NR placement should exclude nodes without geofencing information
https://github.com/storj/storj-private/issues/378

Change-Id: If2af02083496e5a8eefe27beabb406388ee50644
2023-07-31 09:55:54 +02:00
Vitalii
ae5e742a12 web/satellite: update wording for upgrade account banner
Updated wording because we don't want to dissuade customers from upgrading and using Storj by implying that there is a ceiling to what they can store and egress with Storj.

Issue:
https://github.com/storj/storj/issues/6040

Change-Id: I8a08bea47b698e66e023d1bfcaa5ef09dae79192
2023-07-28 09:46:49 +00:00
Vitalii
62f52c829a web/satellite/vuetify: add charts to project dashboard
Imported chart components from main project to vuetify app project dashboard with slight updates.

Issue:
https://github.com/storj/storj/issues/6071

Change-Id: I51254ea60aab59c7a841784b5dac7d41446e1df4
2023-07-27 23:16:40 +00:00
Vitalii
fcbb37fb66 web/satellite: fix build-watch
Build-watch is broken again since we don't use NODE_ENV variable anymore.
Adjusted code to work with 'mode' setting.

Change-Id: I08b19ef59b39e6b14ce05d7340f032ee8377c49c
2023-07-27 21:36:18 +00:00
Wilfred Asomani
c934974652 satellite/{console,web}: update error handling
This change updates how API errors are handled and sent.

Change-Id: Ia4c71eeb6f2d009a47b59ce77a23f70b8b10f6dc
2023-07-27 21:01:01 +00:00
Vitalii
2c56599ca0 web/satellite: revert uploads queue size
Revert upload parallelism queue size back to 4 segments at a time.
Increasing this number causes some memory issues for some users.

Issue:
https://github.com/storj/storj/issues/6120

Change-Id: Ic5bcd1f13ee625fdc2613ee23d6945c905f03142
2023-07-27 19:57:11 +03:00
Clement Sam
cc12a48c24 satellite/admin: extend admin API to allow setting and deleting geofence for projects
Issue: https://github.com/storj/storj-private/issues/357
Change-Id: Ib59319581641f1f5da71c629143e12f11eb04925
2023-07-27 11:40:26 +00:00
Michal Niewrzal
4cc167a6bd satellite/metabase: use better label for ignoring FTS queries
For some test queries we are using workaround to filter them out from
full table scan detection. To avoid confustion what is this all about
we are changing label to be more descriptive.

Change-Id: I41a744e8faf400e3e8de7e416d8f4242f9093fce
2023-07-26 20:01:38 +00:00
Michal Niewrzal
3d9c217627 satellite/metabase: add pending_objects table
This change adds only schema definition of pending_objects table and
small amount of supporting code which will be useful for testing later.

With this table we would like to achieve two major things:
* simplify `objects` table, before we will start working on object
versioning
* gain performance by removing need to filter `objects` results with `status` column, which is not indexed and we would like to avoid that

https://github.com/storj/storj/issues/6045

Change-Id: I6097ce1c644a8a3dad13185915fe01989ad41d90
2023-07-26 20:00:58 +00:00
Moby von Briesen
28737f5c62 web/satellite: Implement "sign out" in vuetify poc
Implements the "sign out" button in the vuetify app to clear all app
data and navigate to login.

For now, there is a hard refresh after the router navigates to login,
becuase the vuetify poc does not have its own login page to route to.
The hard refresh allows the user to be presented with the main app's
login page after signing out.

Change-Id: Idb1c1e6af8511ea5db6533e46b96b9c15693379f
2023-07-26 13:49:30 -04:00
Moby von Briesen
7487809476 web/satellite: unmock list buckets in vuetify app
This change implements the table on the buckets view of the vuetify app.
Searching and pagination are implemented, but sorting functionality
needs to be added in a separate change - we need to extend the
bucketsStore/backend functionality in order to support sorting.

Resolves https://github.com/storj/storj/issues/6062

Change-Id: I4e92e6facbd7b21f4ec081e41f7e568ddb3cea29
2023-07-26 13:49:30 -04:00
Michal Niewrzal
16c5b3c340 mod: bump uplink to v1.11.0
Change-Id: I88922421f250a261b78227a409f4879ae5221140
2023-07-26 15:01:18 +00:00
Vitalii
6d94d6a681 satellite/{console, accounting}: fix project bandwidth calculation which is shown on project dashboard
Another try to fix calculation of used bandwidth which is displayed on Project Dashboard.
This change sums up allocated-dead traffic for the last 3 days and settled traffic for the period which is earlier than 3 days ago.

Issue:
https://github.com/storj/storj-private/issues/293

Change-Id: I91e652eba69f81bd21e0d053ac170e2b926b3cb4
2023-07-26 06:14:48 +00:00
Vitalii
5db0fd8846 web/satellite: wire up vuetify access page to backend
Added logic to list, sort, search and paginate access grants table for Vuetify POC

Change-Id: I215a9ad4f894b6ac985cb6a3059fdcf007e3d914
2023-07-24 23:17:19 +00:00
Jeremy Wharton
28711e30ad web/satellite/vuetify-poc: add project ID to project-specific routes
A project ID path parameter has been incorporated into project-specific
routes. Previously, project IDs were stored only in local storage,
which made it impossible to bookmark or share links to pages within a
particular project.

Resolves #6076

Change-Id: I4f46f58a95dfc9b67d95fdb1c6a65b50fcafe02b
2023-07-24 22:25:29 +00:00
Jeremy Wharton
5a03e29fca web/satellite/vuetify-poc: unmock all projects dashboard data
The all projects dashboard of the Vuetify project has been updated to
display real project information rather than mock data. Additionally,
projects may now be selected and project invitations replied to through
the all projects dashboard.

Resolves #6036
Resolves #6038

Change-Id: I8aef0dc07c172e2bc8e2f7eb26a3d205bd56067f
2023-07-24 21:39:28 +00:00
Jeremy Wharton
9086078ac7 web/satellite: remove glob patterns from linting commands
The glob patterns in the linting commands have been removed in favor of
a simpler way of expressing which files should be linted.

Change-Id: I150a01725642b4bc445e2e157ddf86e50e3911a2
2023-07-24 10:33:07 -05:00
Michal Niewrzal
73a279235a satellite/accounting/live: get project totals in batches
After infrastructure changes redis instance is not neccessay close
to core instance (where tally is calculated) and round trips to get
data from redis can be very costly. From less than hour calculation can take few hours for larger satellite.

This change combines 'segment' and 'storage' usage requests into
batches to reduce latency impact on tally calculation.

https://github.com/storj/storj/issues/5800

Change-Id: I87e57ec09e88fd167060a4ed51dc8b0274a095c5
2023-07-24 14:13:04 +00:00
Jeremy Wharton
f30e0986b6 web/satellite/vuetify-poc: fix app store name conflict
The name of the Vuetify Pinia app store has been renamed so that it
does not conflict with the regular app store. Previously, whichever
store was created first would override the other due to them having
identical names.

Change-Id: I11cb3b70b7385320accd30644cb3ad6cfad7d00e
2023-07-22 20:51:50 -05:00
Cameron
b16c8ba2e4 web/satellite: add indeterminate progress bar
Display indeterminate progress bars in upload modal if progress is less
than 1.

Change-Id: Icdad8e59914985f3ed8fd25dd01dba7e9ff88cf0
2023-07-21 18:11:18 -04:00
Wilfred Asomani
8ed4c573db satellite/admin: add endpoint to unwarn user
This change enables the admin UI to remove the warning status of users.

resolves: storj-private/issues/342

Change-Id: Ib960ffb33fdabc045884ce7fa2c55c3553db0fb0
2023-07-21 17:10:09 +00:00
Vitalii
4ac8320f3c web/satellite: update wording for upgrade account token flow
Update wording to reflect latest auto upgrade account flow.

new wording:
Send more than $10 in STORJ Tokens to the following deposit address to upgrade to a Pro account.
Your account will be upgraded after your transaction receives 15 confirmations.
If your account is not automatically upgraded, please fill out this limit increase request form.

Issue:
https://github.com/storj/storj-private/issues/367

Change-Id: I46fcb9243722313b98e5c54e8194d6152f7e1631
2023-07-21 15:27:31 +00:00
Michal Niewrzal
d9525a0f27 cmd/tools/migrate-segment-copies: fix placement scan from DB
Placement can be null in DB and we need adjust scanning this column
from DB.

Additionally this change sets application name for DB connection.

Change-Id: I3c7d6294f4a3e5e441160b2fd4aeafffe705ec76
2023-07-21 13:12:08 +00:00
Vitalii
6e4044b245 satellite/payments: slightly refactor user upgrade observer of billing chore
Resolves post-merge comments from here
https://review.dev.storj.io/c/storj/storj/+/10780

Reworked some definitions. Added checks and comments.

Change-Id: I19c63804ad1f30a1ffd8cb87e96f43deed20a685
2023-07-21 03:43:42 +00:00
Wilfred Asomani
2c934d1cfd satellite/console: trigger invoice payment on card remove
This change triggers payment when a card is removed.

Issue: https://github.com/storj/storj/issues/6056

Change-Id: Ia1f2b49609ec9d111cbffb0757bec0b513abb79d
2023-07-20 15:46:50 +00:00
Egon Elbre
dc1509ee42 Makefile,web: bump to node@18
Change-Id: Ib50e18ecda9520a802a5a565e4dfe034ae41e98a
2023-07-20 16:44:13 +03:00
Artur M. Wolff
d45bc879c3 {Jenkinsfile, Makefile}: update Go to 1.20.6
Change-Id: I0a9b483ddcf2e0322840b1eb61982a21cea88e72
2023-07-20 09:20:30 +00:00
Egon Elbre
a058b7e982 web/satellite: fix npm for windows
`NODE_ENV=development xyz` approach does not work with Windows.

Change-Id: I3f0c4a6a2e9566df4923d5e6bc317c7d09573f18
2023-07-20 07:59:03 +00:00
Egon Elbre
7d1031feda satellite/accounting: fix duplicate code-block
Change-Id: Ic6de7337a5ff9a2fec73aa3d8a74ba5b9076a8c4
2023-07-20 07:58:53 +00:00
Jeremy Wharton
ca263c05bb web/satellite/vuetify-poc: go to all projects dashboard on logo click
This change navigates users to the all projects dashboard when the logo
in the navigation bar has been clicked.

Change-Id: Id63b616d12eea510b52f715f3003838541d2ab63
2023-07-19 12:06:29 -05:00