Commit Graph

8202 Commits

Author SHA1 Message Date
Michal Niewrzal
61933bc6f0 satellite/{metainfo,metabase}: optimize expired/zombie objects deletion
* optimize SQL for zombie objects deletion query by reducing some direct
selects to segments table
* set AOST for expired/zombie object deletion (was 0 since now)

https://github.com/storj/storj/issues/5881

Change-Id: I50482151d056a86fe0e31678a463f413d410759d
2023-06-09 11:22:46 +00:00
Michal Niewrzal
aea3baf6a9 satellite/accounting/tally: calculate pending object count
Change-Id: I4ee6072f4c60fafd809e8184ada9c1abf7edd8aa
2023-06-09 10:15:27 +00:00
Jeff Wendling
c559df6a1b all: bump storj.io/uplink
this fixes a memory leak in the upload codepath

Change-Id: I6bf6214d331cbe4e8e8912942035ef01a285e3b4
2023-06-08 19:52:24 +00:00
Wilfred Asomani
6ca9f6d2c6 web/satellite: add projects table to all projects dashboard
This change adds a tableview to all projects dashboard. It also
addresses some minor UI issues.

Issues: https://github.com/storj/storj/issues/5925
https://github.com/storj/storj/issues/5924
https://github.com/storj/storj/issues/5923

Change-Id: Ic18566b192005bb1720e0388352bc82a3739a723
2023-06-08 17:14:05 +00:00
Wilfred Asomani
7422fe393b satellite/db: add memberCount to project by userID
This change modifies the projects.GetByUserID function to also fetch
each project's member count.

Change-Id: I5ef4aa3726566920ee9360f5a4f09a8cb490c67e
2023-06-08 17:05:28 +00:00
Márton Elek
3e3ec2358d web/storagenode: zksync era compatible links wallet/transactions
Change-Id: I9079ca20e4d95abedbcf04c0d8529a0f2c6ac938
2023-06-08 13:47:02 +00:00
Egon Elbre
df53914faa satellite/satellitedb: use utilities for conversions
This avoids some potential typos.

Change-Id: Icc5262e1f96fe220dd07212c00acacf6960ee909
2023-06-08 16:13:47 +03:00
Márton Elek
974b4f938a go.mod: bump storj.io/private
Change-Id: I2d130cf2e794c8e5e0a978433cb3d504e6747af0
2023-06-08 12:01:55 +00:00
Márton Elek
e4b8febda7 satellite/metainfo: use default_placement of project during bucket creation
https://github.com/storj/storj/issues/5879

Change-Id: I100e834b3b625b3e0ff00cdcb7675ec6bb570992
2023-06-08 11:21:42 +00:00
Márton Elek
2ea6ca9c18
satellite/console: fill default placement of project based on user info
https://github.com/storj/storj/issues/5879

Change-Id: I5aacbe57a7aab65e11915dd8bf0c68f89da32fb4
2023-06-08 12:47:49 +02:00
Márton Elek
5c26661cd3 dbx/satellitedb: introduce new default_placement fields for users/projects
https://github.com/storj/storj/issues/5879

Change-Id: Ie6d50fd94c09f43c12f895444ed2a86f0820e01c
2023-06-08 10:02:10 +00:00
Wilfred Asomani
3980fcc39b web/satellite: add project details link to dashboard nav area
This change adds a link to the project dashboard nav area to navigate
to the selected project's detail page.

Issue: https://github.com/storj/storj/issues/5930

Change-Id: Ief26505feeb62e88b8e6096d4be70004dc068b64
2023-06-08 09:29:36 +00:00
Michal Niewrzal
f7c7851519 satellite/metainfo: filter metainfo.GetObjectIPs by bucket/object placement
For now we will use bucket placement to determine if we should exclude
some node IPs from metainfo.GetObjectIPs results. Bucket placement is
retrieved directly from DB in parallel to metabase
GetStreamPieceCountByNodeID request.

GetObjectIPs is not heavily used so additional request to DB shouldn't
be a problem for now.

https://github.com/storj/storj/issues/5950

Change-Id: Idf58b1cfbcd1afff5f23868ba2f71ce239f42439
2023-06-07 16:52:02 +00:00
Márton Elek
a463dd0c71 build: use go 1.19 for nighlty containers
Change-Id: I56d23cf797a893685df9014a12affa84687330a9
2023-06-07 16:08:05 +00:00
JT Olio
8c8340f1bc go.mod: bump eventkit
Change-Id: I2ab54bf640fc8a2aeafab77418c63bcd9cabd1aa
2023-06-07 15:35:24 +00:00
Vitalii
4aafa7f043 web/satellite: fix access grant worker load for production builds
Change-Id: Ifdcf3b490c10f491ff861b0e7887ba0edfcbfd7f
2023-06-07 16:42:59 +03:00
Wilfred Asomani
58836834da satellite/{consoleweb,emails}: send new email for project invite
This change adds new email templates for project invites, one for
existing users, one for new users. It changes the project invite code
to use the new template for existing users.

Issue: https://github.com/storj/storj/issues/5860

Change-Id: Ic7b14a677277ea6c25ee527d03f709474fc05f83
2023-06-07 11:11:30 +00:00
Vitalii
5ee991ac64 web/satellite: delete and share modals for gallery view
Added share and delete modals for object browser gallery view.

Issue:
https://github.com/storj/storj/issues/5824

Change-Id: Icdfb14d20155e3de2ce0cb885621435fd16ad855
2023-06-07 13:38:15 +03:00
Jeremy Wharton
7d039364b9 satellite/console/.../consoleapi: keep special chars in registration info
We no longer use registration information in ways that could be
exploited by malicious agents, so filtering special characters is not
necessary and has been removed.

Resolves storj-private#133

Change-Id: I3eb4803c71ccb307b38f0288fe2af5eec70f8309
2023-06-07 01:52:50 +00:00
JT Olio
1437257dbf satellite: save and return which node features are enabled
current feature is if tcp fastopen was successfully enabled

Change-Id: Ide251863a9790b0fbebdf2e82dfd2afa8f25c408
2023-06-06 21:13:29 +00:00
paul cannon
25a5df9752 satellite/repair: don't reuse allNodeIDs
We were reusing a slice to save on allocations, but it turns out the
function using it was being called in multiple goroutines at the same
time.

This is definitely a problem with repairer/segments.go. I'm not 100%
sure if it also is a problem with checker/observer.go, but I'm making
the change there as well to be on the safe side for now.

Repair workers only ran with this bug on testing satellites, and it
looks like the worst that could have happened was that we repaired
pieces off of well-behaved, non-clumped, in-placement nodes by mistake.

Change-Id: I33c112b05941b63d066caab6a34a543840c6b85d
2023-06-06 10:28:04 -05:00
Jeremy Wharton
c858479ef0 web/satellite: show project invitations in dashboard area
This change allows users to view and interact with their project member
invitations through a banner in the dashboard area. The banner only
appears if the All Projects Dashboard is disabled, as it provides the
same functionality in a different way.

References #5855

Change-Id: Ia0771e2af52c40a72f1cacf72bc9098cc68f0dcd
2023-06-06 08:35:50 -05:00
Jeremy Wharton
cafa6db971 web/satellite: show project invitations in all projects dashboard
This change allows users to view and interact with their project member
invitations from within the All Projects Dashboard.

References #5855

Change-Id: If5b4af46924530c91f8a5c16bfb5134de313dc90
2023-06-06 08:34:01 -05:00
Jeremy Wharton
6359c537c0 satellite/console: add API methods for responding to project invites
API endpoints and associated methods have been implemented to allow
users to accept or decline their pending project member invitations
through the satellite frontend.

References #5855

Change-Id: Ic23721c64a65e741dc1015838e617fd1af5c8ca4
2023-06-06 10:38:00 +00:00
Vitalii
b64be2338d web/satellite: added distribution modal for gallery view
Added Object Geographic Distribution modal for gallery view.

Issue:
https://github.com/storj/storj/issues/5824

Change-Id: I379bb933d4f9e759e59882aceb0e74cce79cc5e5
2023-06-06 13:02:40 +03:00
Vitalii
accf695aa3 web/satellite: options dropdown and details modal for gallery view
Added options dropdown and object details modal.
Also added closing gallery on ESC click.

Issue:
https://github.com/storj/storj/issues/5824

Change-Id: I721a2d0272049cc38c1219098d17c170ce6c862e
2023-06-06 13:02:40 +03:00
Jeff Wendling
32f683fe9d satellite/orders: filter nodes based on segment placement
this change adds code to CreateGetOrderLimits to filter
out any nodes that are not in the placement specified
by the segment. notably, it does not change the audit
or repair order limits. the list segments code had to be
changed to include getting the placement field from the
database.

Change-Id: Ice3e42a327811bb20928c619a72ed94e0c1464ac
2023-06-05 13:56:22 -04:00
Michal Niewrzal
337eb9be6a satellite/repair/checker: put into queue segment off placement
Checker when qualifying segment for repair is now looking at pieces
location and if they are outisde segment placement puts them into
repair queue.

Fixes https://github.com/storj/storj/issues/5895

Change-Id: If0d941b30ad94c5ef02fb1a03c7f3d04a2df25c7
2023-06-05 15:53:49 +00:00
JT Olio
1966340b2a storagenode: report fastopen support
Change-Id: Ic707bce32729540391d5139e7f3a82c693bcb834
2023-06-05 15:20:13 +00:00
Michal Niewrzal
128b0a86e3 satellite/repair/repairer: repair pieces out of placement
Segment repairer should take into account segment 'placement' field
and remove or repair pieces from nodes that are outside this placement.

In case when after considering pieces out of placement we are still above
repair threshold we are only updating segment pieces to remove
problematic pieces. Otherwise we are doing regular repair.

https://github.com/storj/storj/issues/5896

Change-Id: I72b652aff2e6b20be3ac6dbfb1d32c2840ce3d59
2023-06-05 14:48:36 +00:00
Egon Elbre
52cefb816c satellite/satellitedb: add utility for converting slices
Change-Id: I2654a9ef7c58016bd5af923c66f5f31819ab9b9d
2023-06-05 14:12:02 +00:00
Márton Elek
52598d1545 satellite/admin: introduce new placement rule
Change-Id: I582d38febd6187e06a46f14fd0f37531a2139c99
2023-06-05 13:34:26 +00:00
Stefan Benten
11324111e3 satellite/admin: secure accidental user email change
It currently is possible to create a violation with regards to
the uniqueness of the user account emails that is used for the
login.
When an update via the admin API is made, it currently is possible
to set the accounts email to an already occupied email address.
This will result in very flacky login behaviour, as well as creating
a lot of other related issues.

This small change adds a check to ensure the email is not attached to
any account.

Change-Id: I167be673082d59ef32cafe41047fce9f5ae534d0
2023-06-05 12:17:31 +00:00
Wilfred Asomani
2e3ed973de satellite/{consoleapi,web}: limit user input
This change limits the length of user input fields like search, email,
username. It also limits the receivable size of request payloads.
This is to prevent potential DDoS attacks resulting from receiving
large payloads.
Improvements are also made to the accounts page and register success
pages to display long names/emails better.

Issue: https://github.com/storj/storj-private/issues/201

Change-Id: I5d36eb83609b3605335a8d150b275c89deaf3b43
2023-06-05 11:43:56 +00:00
Márton Elek
61ad5751f6
cmd/uplink: exclude unnecessary spans from distributed tracing
We have a special method to exclude methods (which are called to frequently) from distributed traces.

https://github.com/storj/common/blob/main/tracing/excluded.go

But this works only, if we define the exclusion function during the initialization.

Let's do it in this patch.

Change-Id: Icf12202bd7213b5c0009332ce2755b267f2bdbae
2023-06-05 12:33:19 +02:00
JT Olio
08a63fa5d1 go.mod: bump storj.io/common and storj.io/uplink
Change-Id: Ifc33429ab3e6f5b52b685c800b7c96dff6b904f2
2023-06-02 18:01:16 -04:00
JT Olio
3566bbf968 versioncontrol: implement partial version rollout automation
see https://github.com/storj/storj/blob/main/docs/blueprints/rollout-automation.md
for context.

there are new config values, but the default values make the
system behave as if there were no changes. existing rollout
configs should work just fine.

Change-Id: I8477adf8b05b5affa5bca1c1a106810c934e5845
2023-06-02 16:01:33 +00:00
JT Olio
36038af3d1 go.mod: bump storj.io/common
Change-Id: I18d4d491cdca99f6694a20f3ead5fba3f75bb658
2023-06-02 11:23:02 -04:00
Vitalii
89457b3472 web/satellite: initial implementation of gallery view
Added new gallery view for object browser.
It is behind new feature flag.

TODO: add options dropdown and modals

Issue:
https://github.com/storj/storj/issues/5824

Change-Id: I21829c599cd904b833eaf429690c66c3da306a0f
2023-06-02 14:03:49 +03:00
Clement Sam
b64179c82a {storagenode/pieces,cmd/storagenode}: refactor lazyfilewalker commands and tests
With this change we are directly testing how the command
is executed when the args are passed

Change-Id: Ibb33926014c9d71c928e0fd374bf4edc5a8a1232
2023-06-02 00:11:53 +00:00
Vitalii
8c4a9f9277 web/satellite: improve onboarding CLI flow small screen adaptation
Improved onboarding CLI flow view on smaller screens.
Also improved notification boxes adaptation.

Issue:
https://github.com/storj/storj/issues/5894

Change-Id: I83dd8968df937d0d50535707e02914bd232fed21
2023-06-01 21:08:00 +00:00
Wilfred Asomani
eb4fd2180b satellite/{console,web}: prevent unneeded redirect to all projects dashboard
This change prevents the redirect to all projects dashboard when no
project is selected (if all projects dash is enabled).
Since a previously selected project id is saved in local storage, it is
used to store it's associated project in memory.
This change also makes a small change to a test that ignores potential
failures.

Issue: https://github.com/storj/storj/issues/5920

Change-Id: Ie758893dfb655893520c642fb47b934cd59f177e
2023-06-01 14:23:20 +00:00
Antonio Franco
aa64162b44
Ordersfiles common unittests (#5516)
* storagenode/orders/ordersfiles: unit test coverage
This change implements unit testing on common.go from the ordersfile package.

* storagenode/orders/ordersfiles: unit test coverage
This change implements the zeebo assert library instead of gotools as to not introduce a new dependency.

* storagenode/orders/ordersfiles: unit test coverage
This change implements the zeebo assert library instead of gotools as to not introduce a new dependency.
2023-06-01 12:27:14 +00:00
Moby von Briesen
563239d555 satellite/satellitedb: Add value_attributions.partner_id back
There were some dbx-generated queries on the old satellite version that
still reference this column, so we need to add it back until the next
version.

Change-Id: I78b19336d9ca0384936d6cc11f5c50e579b4f2ab
2023-05-31 17:31:33 -04:00
Lizzy Thomson
87f2da92b1 web/satellite: replace instances of bandwidth and download to egress
Improve consistency of UI to use the term egress instead of bandwidth
or download

Issue https://github.com/storj/storj/issues/5887

Change-Id: I898ab5d55e5ad8a2d2dd0420c8496d57bc2ecb61
2023-05-31 09:18:58 -06:00
Wilfred Asomani
09c87a1a0d web/satellite: update transactions table design
This change slightly modifies the token transactions table to match the
figma and account for bonuses.

Issue: https://github.com/storj/storj/issues/5755

Change-Id: I2bb27d5e331223a1e5e807ace09bbd6c3bc2c3b0
2023-05-31 14:25:14 +00:00
Moby von Briesen
058dbd4cb7 web/satellite: Disable limits area of dashboard
Add a config flag (default false) to hide the new limit cards (e.g.
segment, storage, bandwidth limits) from the UI. We need to investigate
some queries the egress card is using before enabling these everywhere.

Change-Id: I762e7d9e6a0a4315f1520e688b2bad32b100e5a0
2023-05-31 13:40:54 +03:00
Moby von Briesen
037b07712d satellite/console: Disable all projects dashboard by default
Waiting for basic table view support before enabling

https://github.com/storj/storj/issues/5927

Change-Id: I48388f733138b5aa784c7b2fd3c4d3c813b9e40f
2023-05-30 15:07:43 -04:00
Michal Niewrzal
fe21fd42f7 satellite/overlay: add GetNodesOutOfPlacement method
We would like to verify if nodes matches specific placement e.g. to
validate segment pieces are correctly geofenced.

https://github.com/storj/storj/issues/5896

Change-Id: I842767dccc121a3c60224f677ab55e5dc150c76e
2023-05-30 14:57:20 +02:00
Vitalii
e6959004c9 web/satellite: migrate webpack to vite
Migrated webpack to vite.
Replaced jest with vitest (because I couldn't resolve 'import.meta' issue)
Replaced legacy vue-svg-loader with vite-svg-loader
Replaced bip39 dep with bip39-english (it includes only english wordlist)
Replaced aws-sdk v2 with aws-sdk V3 (because v2 was throwing some weird error in console. Ref: https://stackoverflow.com/questions/75107933/aws-sdk-contributes-to-build-error-uncaught-typeerror-e-is-not-a-constructor)
Renamed VUE_APP_ENDPOINT_URL env variable to VITE_ENDPOINT_URL
Removed a ton of dependencies (like babel and jest-related stuff).

Tested in Chrome, Safari, Brave, Firefox and Opera browsers.

Additionally fixed logout errors from buckets and object browser routes.

TODO: try to remove util and stream-browserify dependencies and see if it works

Change-Id: I4562649a59eb0ba80c1a672d55c59fceb8c80b23
2023-05-30 09:25:30 +00:00