Commit Graph

3267 Commits

Author SHA1 Message Date
Moby von Briesen
771ec3237a satellite/console: Update CSP to include storjsatelliteshare.io
Change-Id: I8238f6e2644a0f4ee727d0b8da56e59733dd3252
2023-06-12 13:31:00 -04:00
Jeremy Wharton
acdc9f3608 satellite/analytics: add events for project member invitation responses
This change adds analytics events for accepting and rejecting project
member invitations.

References #5855

Change-Id: I2101eb711312a683ecdb4b7d0c6d7cc3ae09c440
2023-06-12 16:07:32 +00:00
Jeremy Wharton
bb620e746b satellite/satellitedb: add index for project_members project_id column
An index has been added on the project_id column of the project_members
satellite database table so that we can retrieve members of a project
without performing a full table scan.

References #5855

Change-Id: I1cc30686f836c8fd1aa319247ce857a2392e7a52
2023-06-11 18:12:33 -05:00
Michal Niewrzal
7c33521ace satellite/repair/repairer: use placement to select nodes for repair upload
We missed to set placement as a part of selection request. It can case
uploading repaired data out of specified placement.

I will provide test as a separate change.

Change-Id: I4efe67f2d5f545a1d70e831e5d297f0977a4eed1
2023-06-10 20:55:39 +02:00
JT Olio
3fff61f04a payments: don't redefine compensation rates twice
Change-Id: Ic00abe3795a000d4f0284c99f270180123a2f663
2023-06-10 12:40:43 +00:00
Vitalii
e83e98e46a web/satellite: vuetify POC
Built side Vuetify subproject inside web/satellite with limited functinality.
For now it has navigation side bar, simple project dashboard and team page (where you can list/add team members).

Issue:
https://github.com/storj/storj/issues/5854

Change-Id: I9ff3e80b8ace1dc31de6a788174c5ffc19f050f8
2023-06-10 00:59:40 +00:00
Michal Niewrzal
61933bc6f0 satellite/{metainfo,metabase}: optimize expired/zombie objects deletion
* optimize SQL for zombie objects deletion query by reducing some direct
selects to segments table
* set AOST for expired/zombie object deletion (was 0 since now)

https://github.com/storj/storj/issues/5881

Change-Id: I50482151d056a86fe0e31678a463f413d410759d
2023-06-09 11:22:46 +00:00
Michal Niewrzal
aea3baf6a9 satellite/accounting/tally: calculate pending object count
Change-Id: I4ee6072f4c60fafd809e8184ada9c1abf7edd8aa
2023-06-09 10:15:27 +00:00
Wilfred Asomani
7422fe393b satellite/db: add memberCount to project by userID
This change modifies the projects.GetByUserID function to also fetch
each project's member count.

Change-Id: I5ef4aa3726566920ee9360f5a4f09a8cb490c67e
2023-06-08 17:05:28 +00:00
Egon Elbre
df53914faa satellite/satellitedb: use utilities for conversions
This avoids some potential typos.

Change-Id: Icc5262e1f96fe220dd07212c00acacf6960ee909
2023-06-08 16:13:47 +03:00
Márton Elek
e4b8febda7 satellite/metainfo: use default_placement of project during bucket creation
https://github.com/storj/storj/issues/5879

Change-Id: I100e834b3b625b3e0ff00cdcb7675ec6bb570992
2023-06-08 11:21:42 +00:00
Márton Elek
2ea6ca9c18
satellite/console: fill default placement of project based on user info
https://github.com/storj/storj/issues/5879

Change-Id: I5aacbe57a7aab65e11915dd8bf0c68f89da32fb4
2023-06-08 12:47:49 +02:00
Márton Elek
5c26661cd3 dbx/satellitedb: introduce new default_placement fields for users/projects
https://github.com/storj/storj/issues/5879

Change-Id: Ie6d50fd94c09f43c12f895444ed2a86f0820e01c
2023-06-08 10:02:10 +00:00
Michal Niewrzal
f7c7851519 satellite/metainfo: filter metainfo.GetObjectIPs by bucket/object placement
For now we will use bucket placement to determine if we should exclude
some node IPs from metainfo.GetObjectIPs results. Bucket placement is
retrieved directly from DB in parallel to metabase
GetStreamPieceCountByNodeID request.

GetObjectIPs is not heavily used so additional request to DB shouldn't
be a problem for now.

https://github.com/storj/storj/issues/5950

Change-Id: Idf58b1cfbcd1afff5f23868ba2f71ce239f42439
2023-06-07 16:52:02 +00:00
Wilfred Asomani
58836834da satellite/{consoleweb,emails}: send new email for project invite
This change adds new email templates for project invites, one for
existing users, one for new users. It changes the project invite code
to use the new template for existing users.

Issue: https://github.com/storj/storj/issues/5860

Change-Id: Ic7b14a677277ea6c25ee527d03f709474fc05f83
2023-06-07 11:11:30 +00:00
Jeremy Wharton
7d039364b9 satellite/console/.../consoleapi: keep special chars in registration info
We no longer use registration information in ways that could be
exploited by malicious agents, so filtering special characters is not
necessary and has been removed.

Resolves storj-private#133

Change-Id: I3eb4803c71ccb307b38f0288fe2af5eec70f8309
2023-06-07 01:52:50 +00:00
JT Olio
1437257dbf satellite: save and return which node features are enabled
current feature is if tcp fastopen was successfully enabled

Change-Id: Ide251863a9790b0fbebdf2e82dfd2afa8f25c408
2023-06-06 21:13:29 +00:00
paul cannon
25a5df9752 satellite/repair: don't reuse allNodeIDs
We were reusing a slice to save on allocations, but it turns out the
function using it was being called in multiple goroutines at the same
time.

This is definitely a problem with repairer/segments.go. I'm not 100%
sure if it also is a problem with checker/observer.go, but I'm making
the change there as well to be on the safe side for now.

Repair workers only ran with this bug on testing satellites, and it
looks like the worst that could have happened was that we repaired
pieces off of well-behaved, non-clumped, in-placement nodes by mistake.

Change-Id: I33c112b05941b63d066caab6a34a543840c6b85d
2023-06-06 10:28:04 -05:00
Jeremy Wharton
6359c537c0 satellite/console: add API methods for responding to project invites
API endpoints and associated methods have been implemented to allow
users to accept or decline their pending project member invitations
through the satellite frontend.

References #5855

Change-Id: Ic23721c64a65e741dc1015838e617fd1af5c8ca4
2023-06-06 10:38:00 +00:00
Jeff Wendling
32f683fe9d satellite/orders: filter nodes based on segment placement
this change adds code to CreateGetOrderLimits to filter
out any nodes that are not in the placement specified
by the segment. notably, it does not change the audit
or repair order limits. the list segments code had to be
changed to include getting the placement field from the
database.

Change-Id: Ice3e42a327811bb20928c619a72ed94e0c1464ac
2023-06-05 13:56:22 -04:00
Michal Niewrzal
337eb9be6a satellite/repair/checker: put into queue segment off placement
Checker when qualifying segment for repair is now looking at pieces
location and if they are outisde segment placement puts them into
repair queue.

Fixes https://github.com/storj/storj/issues/5895

Change-Id: If0d941b30ad94c5ef02fb1a03c7f3d04a2df25c7
2023-06-05 15:53:49 +00:00
Michal Niewrzal
128b0a86e3 satellite/repair/repairer: repair pieces out of placement
Segment repairer should take into account segment 'placement' field
and remove or repair pieces from nodes that are outside this placement.

In case when after considering pieces out of placement we are still above
repair threshold we are only updating segment pieces to remove
problematic pieces. Otherwise we are doing regular repair.

https://github.com/storj/storj/issues/5896

Change-Id: I72b652aff2e6b20be3ac6dbfb1d32c2840ce3d59
2023-06-05 14:48:36 +00:00
Egon Elbre
52cefb816c satellite/satellitedb: add utility for converting slices
Change-Id: I2654a9ef7c58016bd5af923c66f5f31819ab9b9d
2023-06-05 14:12:02 +00:00
Márton Elek
52598d1545 satellite/admin: introduce new placement rule
Change-Id: I582d38febd6187e06a46f14fd0f37531a2139c99
2023-06-05 13:34:26 +00:00
Stefan Benten
11324111e3 satellite/admin: secure accidental user email change
It currently is possible to create a violation with regards to
the uniqueness of the user account emails that is used for the
login.
When an update via the admin API is made, it currently is possible
to set the accounts email to an already occupied email address.
This will result in very flacky login behaviour, as well as creating
a lot of other related issues.

This small change adds a check to ensure the email is not attached to
any account.

Change-Id: I167be673082d59ef32cafe41047fce9f5ae534d0
2023-06-05 12:17:31 +00:00
Wilfred Asomani
2e3ed973de satellite/{consoleapi,web}: limit user input
This change limits the length of user input fields like search, email,
username. It also limits the receivable size of request payloads.
This is to prevent potential DDoS attacks resulting from receiving
large payloads.
Improvements are also made to the accounts page and register success
pages to display long names/emails better.

Issue: https://github.com/storj/storj-private/issues/201

Change-Id: I5d36eb83609b3605335a8d150b275c89deaf3b43
2023-06-05 11:43:56 +00:00
JT Olio
36038af3d1 go.mod: bump storj.io/common
Change-Id: I18d4d491cdca99f6694a20f3ead5fba3f75bb658
2023-06-02 11:23:02 -04:00
Vitalii
89457b3472 web/satellite: initial implementation of gallery view
Added new gallery view for object browser.
It is behind new feature flag.

TODO: add options dropdown and modals

Issue:
https://github.com/storj/storj/issues/5824

Change-Id: I21829c599cd904b833eaf429690c66c3da306a0f
2023-06-02 14:03:49 +03:00
Wilfred Asomani
eb4fd2180b satellite/{console,web}: prevent unneeded redirect to all projects dashboard
This change prevents the redirect to all projects dashboard when no
project is selected (if all projects dash is enabled).
Since a previously selected project id is saved in local storage, it is
used to store it's associated project in memory.
This change also makes a small change to a test that ignores potential
failures.

Issue: https://github.com/storj/storj/issues/5920

Change-Id: Ie758893dfb655893520c642fb47b934cd59f177e
2023-06-01 14:23:20 +00:00
Moby von Briesen
563239d555 satellite/satellitedb: Add value_attributions.partner_id back
There were some dbx-generated queries on the old satellite version that
still reference this column, so we need to add it back until the next
version.

Change-Id: I78b19336d9ca0384936d6cc11f5c50e579b4f2ab
2023-05-31 17:31:33 -04:00
Moby von Briesen
058dbd4cb7 web/satellite: Disable limits area of dashboard
Add a config flag (default false) to hide the new limit cards (e.g.
segment, storage, bandwidth limits) from the UI. We need to investigate
some queries the egress card is using before enabling these everywhere.

Change-Id: I762e7d9e6a0a4315f1520e688b2bad32b100e5a0
2023-05-31 13:40:54 +03:00
Moby von Briesen
037b07712d satellite/console: Disable all projects dashboard by default
Waiting for basic table view support before enabling

https://github.com/storj/storj/issues/5927

Change-Id: I48388f733138b5aa784c7b2fd3c4d3c813b9e40f
2023-05-30 15:07:43 -04:00
Michal Niewrzal
fe21fd42f7 satellite/overlay: add GetNodesOutOfPlacement method
We would like to verify if nodes matches specific placement e.g. to
validate segment pieces are correctly geofenced.

https://github.com/storj/storj/issues/5896

Change-Id: I842767dccc121a3c60224f677ab55e5dc150c76e
2023-05-30 14:57:20 +02:00
Wilfred Asomani
4791a6422d satellite/console: add STORJ bonus to transactions
This change includes STORJ bonuses to the list of transactions returned
by the /wallet/payments endpoint.

Issue: https://github.com/storj/storj/issues/5755

Change-Id: Icc95c2cb9dd9fc5ee7a373e68c1cf8a991e1aa58
2023-05-26 20:36:30 +00:00
Michal Niewrzal
c48bd81e5f satellite/satellitedb: update SelectAllStorageNodes* to set country code
Methods SelectAllStorageNodesUpload and SelectAllStorageNodesDownload
are not returning full info with overlay.SelectedNode because its
missing CountryCode.

Change-Id: Ie3cb396bf28d7ec4c6ab8927e5bb560236036aa6
2023-05-26 11:02:29 +00:00
Sean Harvey
403f5eff81
satellite/admin: add /api/apikeys/{apikey} GET endpoint
This allows scripted automation to get more details of the
API key such as project ID, and paid tier status.

Updates https://github.com/storj/gateway-mt/issues/321

Change-Id: I8a835752d4fd67382aca804b8c93e63de6c9a846
2023-05-26 17:18:05 +12:00
Jeremy Wharton
c3d72a269e satellite/{accounting,satellitedb}: create tallies for empty buckets
Tallies are now created for buckets with no objects. Previously, the
bucket tally collector skipped empty buckets since it created tallies
only using information from the objects table. Methods that used
bucket tallies when calculating usage costs would return incorrect
results because of this.

Change-Id: I0b37fe7159a11cc02a51562000dad9258555d9f9
2023-05-25 19:48:59 +00:00
Moby von Briesen
919278dbf9 satellite/satellitedb: drop partnerID columns
The last code referencing these columns was removed as of satellite
release v1.79, so it is safe to remove them now.

Resolves https://github.com/storj/storj/issues/5432

Change-Id: I2e9d641b2511a61e0b9482ef0f4955a73c290709
2023-05-24 14:38:57 +00:00
Moby von Briesen
714c9f40c1 satellite/console: Turn all projects dashboard on by default
Related issue: https://github.com/storj/storj/issues/5872

Change-Id: Idd78f78a63534e15f99ebf3015431bbe52ba18ef
2023-05-24 13:45:22 +00:00
Vitalii
98591f34f9 satellite/{web, console}: implemented new objects uploading hover component
Added new feature flag.
Implemented new objects uploading hover component.

Issue:
https://github.com/storj/storj/issues/5825

Change-Id: If43ade5db33aecba6c089ce727503518c59f6ffc
2023-05-23 15:52:41 +00:00
Jeremy Wharton
d8f64326f5 satellite/accounting/tally: remove objects loop
This change removes the use of the objects loop for calculating bucket
tallies. It has been superseded by a custom query.

Change-Id: I9ea4633006c9af3ea14d7de40871639e7b687c22
2023-05-22 15:18:03 -05:00
Michal Niewrzal
eabd9dd994 satellite/orders: remove unsed argument
Change-Id: I6c5221fc19f97ae6db5627d7239795ff663289e0
2023-05-22 14:35:08 +00:00
Cameron
09ec5f107d satellite/payments: Remove expired package credits
During billing, before invoice creation, check if users are part of a
package plan. If so, and if the package plan is expired, remove unused
credit from the user's balance. If the user has credit in addition to
the package credit, send an analytics event to notify someone to handle
the credit removal manually.

Change-Id: Iad71d791f67c9733f9d9e42f962c64b2780264cc
2023-05-22 11:58:37 +00:00
JT Olio
b502310fe5 cmd/inspector: remove
this is a very old tool built in the very early days
of v3, when we didn't know how the network would be
used. this tool anticipated being able to query remote
nodes for internal state. we don't do that. i don't
think anyone uses this.

Change-Id: Ie1ded3ecbedb09313f2d6fc721039e0f15e4ee85
2023-05-19 09:11:29 +00:00
paul cannon
3dc01bd25d satellite/repair: change how we log clumped pieces
rather than only logging the last_nets we see in clumpedPieces, this
will run through all the last_nets and log any that have more than one
node. This should have the same outcome, except the counts will be 1
higher (because FindClumpedPieces won't include the first node found in
a clumped network, and this will).

This should be quite a bit faster.

Change-Id: I6a7b2fd387e98963d5295c9ecfde80f2e1ee3b7a
2023-05-19 10:38:50 +02:00
paul cannon
c856d45cc0 satellite/overlay: fix GetNodesNetworkInOrder
We were using the UploadSelectionCache previously, which does _not_ have
all nodes, or even all online nodes, in it. So all nodes with less than
MinimumVersion, or with less than MinimumDiskSpace, or nodes suspended
for unknown audit errors, or nodes that have started graceful exit, were
all missing, and ended up having empty last_nets. Even with all that,
I'm kind of surprised how many nodes this involved, but using the upload
selection cache was definitely wrong.

This change uses the download selection cache instead, which excludes
nodes only when they are disqualified, gracefully exited (completely),
or offline.

Change-Id: Iaa07c988aa29c1eb05796ac48a6f19d69f5826c1
2023-05-19 08:08:08 +00:00
Michal Niewrzal
9b3488276d satellite/gracefulexit: use node alias instead id with observer
Using node alias helps using less cpu and memory.

Fixes https://github.com/storj/storj/issues/5654

Change-Id: If3a5c7810732cbb1bff4dcb78706c81aca56b71b
2023-05-18 22:37:46 +00:00
paul cannon
de737bdee9 satellite/repair: add flag for de-clumping behavior
It seems that the "what pieces are clumped" code does not work right, so
this logic is causing repair overload or other repair failures.

Hide it behind a flag while we figure out what is going on, so that
repair can still work in the meantime.

Change-Id: If83ef7895cba870353a67ab13573193d92fff80b
2023-05-18 21:02:36 +00:00
Michal Niewrzal
c0e7f463fe satellite/metabase: remove segmentsloop package
Last change to remove segments loop from codebase.

https://github.com/storj/storj/issues/5237

Change-Id: I77b12911b6b4e390a7385e6e8057c7587e74b70a
2023-05-18 19:08:29 +00:00
Moby von Briesen
19cb08b025 web/satellite: add "public linksharing URL"
* Update defaults for gateway credentials URL and linksharing URL to use
  storjsatelliteshare.io instead of storjshare.io
* Add new config for "public linksharing URL" and set it to
  link.storjshare.io
* Use "private" linksharing URL for actions within the object browser
* Use "public" linksharing URL for sharing files externally

Resolves https://github.com/storj/storj/issues/5805

Change-Id: I2c8fbd04141755b4751dcf4d054253a7ff8d6cf3
2023-05-18 07:02:26 +00:00