Commit Graph

8026 Commits

Author SHA1 Message Date
Jeff Wendling
32f683fe9d satellite/orders: filter nodes based on segment placement
this change adds code to CreateGetOrderLimits to filter
out any nodes that are not in the placement specified
by the segment. notably, it does not change the audit
or repair order limits. the list segments code had to be
changed to include getting the placement field from the
database.

Change-Id: Ice3e42a327811bb20928c619a72ed94e0c1464ac
2023-06-05 13:56:22 -04:00
Michal Niewrzal
337eb9be6a satellite/repair/checker: put into queue segment off placement
Checker when qualifying segment for repair is now looking at pieces
location and if they are outisde segment placement puts them into
repair queue.

Fixes https://github.com/storj/storj/issues/5895

Change-Id: If0d941b30ad94c5ef02fb1a03c7f3d04a2df25c7
2023-06-05 15:53:49 +00:00
JT Olio
1966340b2a storagenode: report fastopen support
Change-Id: Ic707bce32729540391d5139e7f3a82c693bcb834
2023-06-05 15:20:13 +00:00
Michal Niewrzal
128b0a86e3 satellite/repair/repairer: repair pieces out of placement
Segment repairer should take into account segment 'placement' field
and remove or repair pieces from nodes that are outside this placement.

In case when after considering pieces out of placement we are still above
repair threshold we are only updating segment pieces to remove
problematic pieces. Otherwise we are doing regular repair.

https://github.com/storj/storj/issues/5896

Change-Id: I72b652aff2e6b20be3ac6dbfb1d32c2840ce3d59
2023-06-05 14:48:36 +00:00
Egon Elbre
52cefb816c satellite/satellitedb: add utility for converting slices
Change-Id: I2654a9ef7c58016bd5af923c66f5f31819ab9b9d
2023-06-05 14:12:02 +00:00
Márton Elek
52598d1545 satellite/admin: introduce new placement rule
Change-Id: I582d38febd6187e06a46f14fd0f37531a2139c99
2023-06-05 13:34:26 +00:00
Stefan Benten
11324111e3 satellite/admin: secure accidental user email change
It currently is possible to create a violation with regards to
the uniqueness of the user account emails that is used for the
login.
When an update via the admin API is made, it currently is possible
to set the accounts email to an already occupied email address.
This will result in very flacky login behaviour, as well as creating
a lot of other related issues.

This small change adds a check to ensure the email is not attached to
any account.

Change-Id: I167be673082d59ef32cafe41047fce9f5ae534d0
2023-06-05 12:17:31 +00:00
Wilfred Asomani
2e3ed973de satellite/{consoleapi,web}: limit user input
This change limits the length of user input fields like search, email,
username. It also limits the receivable size of request payloads.
This is to prevent potential DDoS attacks resulting from receiving
large payloads.
Improvements are also made to the accounts page and register success
pages to display long names/emails better.

Issue: https://github.com/storj/storj-private/issues/201

Change-Id: I5d36eb83609b3605335a8d150b275c89deaf3b43
2023-06-05 11:43:56 +00:00
Márton Elek
61ad5751f6
cmd/uplink: exclude unnecessary spans from distributed tracing
We have a special method to exclude methods (which are called to frequently) from distributed traces.

https://github.com/storj/common/blob/main/tracing/excluded.go

But this works only, if we define the exclusion function during the initialization.

Let's do it in this patch.

Change-Id: Icf12202bd7213b5c0009332ce2755b267f2bdbae
2023-06-05 12:33:19 +02:00
JT Olio
08a63fa5d1 go.mod: bump storj.io/common and storj.io/uplink
Change-Id: Ifc33429ab3e6f5b52b685c800b7c96dff6b904f2
2023-06-02 18:01:16 -04:00
JT Olio
3566bbf968 versioncontrol: implement partial version rollout automation
see https://github.com/storj/storj/blob/main/docs/blueprints/rollout-automation.md
for context.

there are new config values, but the default values make the
system behave as if there were no changes. existing rollout
configs should work just fine.

Change-Id: I8477adf8b05b5affa5bca1c1a106810c934e5845
2023-06-02 16:01:33 +00:00
JT Olio
36038af3d1 go.mod: bump storj.io/common
Change-Id: I18d4d491cdca99f6694a20f3ead5fba3f75bb658
2023-06-02 11:23:02 -04:00
Vitalii
89457b3472 web/satellite: initial implementation of gallery view
Added new gallery view for object browser.
It is behind new feature flag.

TODO: add options dropdown and modals

Issue:
https://github.com/storj/storj/issues/5824

Change-Id: I21829c599cd904b833eaf429690c66c3da306a0f
2023-06-02 14:03:49 +03:00
Clement Sam
b64179c82a {storagenode/pieces,cmd/storagenode}: refactor lazyfilewalker commands and tests
With this change we are directly testing how the command
is executed when the args are passed

Change-Id: Ibb33926014c9d71c928e0fd374bf4edc5a8a1232
2023-06-02 00:11:53 +00:00
Vitalii
8c4a9f9277 web/satellite: improve onboarding CLI flow small screen adaptation
Improved onboarding CLI flow view on smaller screens.
Also improved notification boxes adaptation.

Issue:
https://github.com/storj/storj/issues/5894

Change-Id: I83dd8968df937d0d50535707e02914bd232fed21
2023-06-01 21:08:00 +00:00
Wilfred Asomani
eb4fd2180b satellite/{console,web}: prevent unneeded redirect to all projects dashboard
This change prevents the redirect to all projects dashboard when no
project is selected (if all projects dash is enabled).
Since a previously selected project id is saved in local storage, it is
used to store it's associated project in memory.
This change also makes a small change to a test that ignores potential
failures.

Issue: https://github.com/storj/storj/issues/5920

Change-Id: Ie758893dfb655893520c642fb47b934cd59f177e
2023-06-01 14:23:20 +00:00
Antonio Franco
aa64162b44
Ordersfiles common unittests (#5516)
* storagenode/orders/ordersfiles: unit test coverage
This change implements unit testing on common.go from the ordersfile package.

* storagenode/orders/ordersfiles: unit test coverage
This change implements the zeebo assert library instead of gotools as to not introduce a new dependency.

* storagenode/orders/ordersfiles: unit test coverage
This change implements the zeebo assert library instead of gotools as to not introduce a new dependency.
2023-06-01 12:27:14 +00:00
Moby von Briesen
563239d555 satellite/satellitedb: Add value_attributions.partner_id back
There were some dbx-generated queries on the old satellite version that
still reference this column, so we need to add it back until the next
version.

Change-Id: I78b19336d9ca0384936d6cc11f5c50e579b4f2ab
2023-05-31 17:31:33 -04:00
Lizzy Thomson
87f2da92b1 web/satellite: replace instances of bandwidth and download to egress
Improve consistency of UI to use the term egress instead of bandwidth
or download

Issue https://github.com/storj/storj/issues/5887

Change-Id: I898ab5d55e5ad8a2d2dd0420c8496d57bc2ecb61
2023-05-31 09:18:58 -06:00
Wilfred Asomani
09c87a1a0d web/satellite: update transactions table design
This change slightly modifies the token transactions table to match the
figma and account for bonuses.

Issue: https://github.com/storj/storj/issues/5755

Change-Id: I2bb27d5e331223a1e5e807ace09bbd6c3bc2c3b0
2023-05-31 14:25:14 +00:00
Moby von Briesen
058dbd4cb7 web/satellite: Disable limits area of dashboard
Add a config flag (default false) to hide the new limit cards (e.g.
segment, storage, bandwidth limits) from the UI. We need to investigate
some queries the egress card is using before enabling these everywhere.

Change-Id: I762e7d9e6a0a4315f1520e688b2bad32b100e5a0
2023-05-31 13:40:54 +03:00
Moby von Briesen
037b07712d satellite/console: Disable all projects dashboard by default
Waiting for basic table view support before enabling

https://github.com/storj/storj/issues/5927

Change-Id: I48388f733138b5aa784c7b2fd3c4d3c813b9e40f
2023-05-30 15:07:43 -04:00
Michal Niewrzal
fe21fd42f7 satellite/overlay: add GetNodesOutOfPlacement method
We would like to verify if nodes matches specific placement e.g. to
validate segment pieces are correctly geofenced.

https://github.com/storj/storj/issues/5896

Change-Id: I842767dccc121a3c60224f677ab55e5dc150c76e
2023-05-30 14:57:20 +02:00
Vitalii
e6959004c9 web/satellite: migrate webpack to vite
Migrated webpack to vite.
Replaced jest with vitest (because I couldn't resolve 'import.meta' issue)
Replaced legacy vue-svg-loader with vite-svg-loader
Replaced bip39 dep with bip39-english (it includes only english wordlist)
Replaced aws-sdk v2 with aws-sdk V3 (because v2 was throwing some weird error in console. Ref: https://stackoverflow.com/questions/75107933/aws-sdk-contributes-to-build-error-uncaught-typeerror-e-is-not-a-constructor)
Renamed VUE_APP_ENDPOINT_URL env variable to VITE_ENDPOINT_URL
Removed a ton of dependencies (like babel and jest-related stuff).

Tested in Chrome, Safari, Brave, Firefox and Opera browsers.

Additionally fixed logout errors from buckets and object browser routes.

TODO: try to remove util and stream-browserify dependencies and see if it works

Change-Id: I4562649a59eb0ba80c1a672d55c59fceb8c80b23
2023-05-30 09:25:30 +00:00
Wilfred Asomani
4791a6422d satellite/console: add STORJ bonus to transactions
This change includes STORJ bonuses to the list of transactions returned
by the /wallet/payments endpoint.

Issue: https://github.com/storj/storj/issues/5755

Change-Id: Icc95c2cb9dd9fc5ee7a373e68c1cf8a991e1aa58
2023-05-26 20:36:30 +00:00
Michal Niewrzal
c48bd81e5f satellite/satellitedb: update SelectAllStorageNodes* to set country code
Methods SelectAllStorageNodesUpload and SelectAllStorageNodesDownload
are not returning full info with overlay.SelectedNode because its
missing CountryCode.

Change-Id: Ie3cb396bf28d7ec4c6ab8927e5bb560236036aa6
2023-05-26 11:02:29 +00:00
Sean Harvey
403f5eff81
satellite/admin: add /api/apikeys/{apikey} GET endpoint
This allows scripted automation to get more details of the
API key such as project ID, and paid tier status.

Updates https://github.com/storj/gateway-mt/issues/321

Change-Id: I8a835752d4fd67382aca804b8c93e63de6c9a846
2023-05-26 17:18:05 +12:00
Jeremy Wharton
c3d72a269e satellite/{accounting,satellitedb}: create tallies for empty buckets
Tallies are now created for buckets with no objects. Previously, the
bucket tally collector skipped empty buckets since it created tallies
only using information from the objects table. Methods that used
bucket tallies when calculating usage costs would return incorrect
results because of this.

Change-Id: I0b37fe7159a11cc02a51562000dad9258555d9f9
2023-05-25 19:48:59 +00:00
Jeff Wendling
ccfe5cae49 cmd/uplink: pass in a maximum concurrent segments value
also change the config creation to be more robust to
changes that add defaults in the future by not fully
reconstructing the config value passed in to the
project.

Change-Id: I673e8b54ce0b951ae735bf4658525c477c26ac5a
2023-05-25 10:53:50 -04:00
Jeff Wendling
a0fbc87b31 cmd/uplink: add upload log file flag
Change-Id: I740eaf431cfe5bb3dcb746e4128dea5efc935257
2023-05-25 14:14:00 +00:00
Márton Elek
d6615cb698
makefile, docker: purge deleted (satellite) inspector tool from build files
the inspector tool of satellite is removed recently. We need to remove it from the build/docker files.

Change-Id: Icd75474ea64fe5f9dd4ed76d8597982108f93536
2023-05-25 11:04:32 +02:00
Lizzy Thomson
6f2d6a97a6 web/satellite: update modals for create projects
Update to use NewCreateProjectModal to match figma designs and update
CreateProjectPromptModal to handle free tier case and paid tier increase
project limits case

Issue https://github.com/storj/storj/issues/5814

Change-Id: I8c5f818d3867d0484b213d1bfc3ce51e904c0afa
2023-05-24 17:47:01 +00:00
Lizzy Thomson
7d745ac578 web/satellite: update wording from decentralized to distributed cloud
Issue https://github.com/storj/storj/issues/5873

Change-Id: I0cb1032a6513289482e56dabefff934a8d722134
2023-05-24 17:12:52 +00:00
Wilfred Asomani
715b9f0299 web/satellite: fix table cells styling
This fixes an issue where the "primary" class will be applied to the
wrong table cell. Also, truncation for long file names is fixed.

Change-Id: I6303f4b7d3a7a064f8dc79fac90b0bc2d5eb5024
2023-05-24 16:24:28 +00:00
Clement Sam
c6f67d4799 storagenode: fix lazyfilewalker failing with SIGPIPE
Lazyfilewalker was failing with SIGPIPE which was quite
misleading. The command was failing because the
the value of the --lower-io-priority flag was assumed
to be an arguement since it was passed as
"--lower-io-priority true" instead "--lower-io-priority=true"

Resolves https://github.com/storj/storj/issues/5900

Change-Id: Icf79fcce76dafee21659d76ee0ce19d8520c8f1d
2023-05-24 15:19:31 +00:00
Moby von Briesen
919278dbf9 satellite/satellitedb: drop partnerID columns
The last code referencing these columns was removed as of satellite
release v1.79, so it is safe to remove them now.

Resolves https://github.com/storj/storj/issues/5432

Change-Id: I2e9d641b2511a61e0b9482ef0f4955a73c290709
2023-05-24 14:38:57 +00:00
Moby von Briesen
714c9f40c1 satellite/console: Turn all projects dashboard on by default
Related issue: https://github.com/storj/storj/issues/5872

Change-Id: Idd78f78a63534e15f99ebf3015431bbe52ba18ef
2023-05-24 13:45:22 +00:00
Clement Sam
bf8072d3f3 cmd/storagenode: fix identity flag for graceful exit commands
Resolves https://github.com/storj/customer-issues/issues/780
Resolves https://github.com/storj/storj/issues/5807

Change-Id: I0af2b14f5b12ddc1e7a09afb2cf5ccf025f53502
2023-05-24 07:58:34 +00:00
Vitalii
98591f34f9 satellite/{web, console}: implemented new objects uploading hover component
Added new feature flag.
Implemented new objects uploading hover component.

Issue:
https://github.com/storj/storj/issues/5825

Change-Id: If43ade5db33aecba6c089ce727503518c59f6ffc
2023-05-23 15:52:41 +00:00
Jeremy Wharton
d8f64326f5 satellite/accounting/tally: remove objects loop
This change removes the use of the objects loop for calculating bucket
tallies. It has been superseded by a custom query.

Change-Id: I9ea4633006c9af3ea14d7de40871639e7b687c22
2023-05-22 15:18:03 -05:00
Michal Niewrzal
eabd9dd994 satellite/orders: remove unsed argument
Change-Id: I6c5221fc19f97ae6db5627d7239795ff663289e0
2023-05-22 14:35:08 +00:00
Cameron
09ec5f107d satellite/payments: Remove expired package credits
During billing, before invoice creation, check if users are part of a
package plan. If so, and if the package plan is expired, remove unused
credit from the user's balance. If the user has credit in addition to
the package credit, send an analytics event to notify someone to handle
the credit removal manually.

Change-Id: Iad71d791f67c9733f9d9e42f962c64b2780264cc
2023-05-22 11:58:37 +00:00
Wilfred Asomani
250704493d web/satellite: fix frozen status banners not showing
This change moves the call to fetch user's frozen status into the
getUser store action. It fixes an issue where the froze status banners
will not show.

Change-Id: I1a1853d5a89502e294848ceb6a8566120ecd4c10
2023-05-19 17:06:28 +00:00
JT Olio
b502310fe5 cmd/inspector: remove
this is a very old tool built in the very early days
of v3, when we didn't know how the network would be
used. this tool anticipated being able to query remote
nodes for internal state. we don't do that. i don't
think anyone uses this.

Change-Id: Ie1ded3ecbedb09313f2d6fc721039e0f15e4ee85
2023-05-19 09:11:29 +00:00
paul cannon
3dc01bd25d satellite/repair: change how we log clumped pieces
rather than only logging the last_nets we see in clumpedPieces, this
will run through all the last_nets and log any that have more than one
node. This should have the same outcome, except the counts will be 1
higher (because FindClumpedPieces won't include the first node found in
a clumped network, and this will).

This should be quite a bit faster.

Change-Id: I6a7b2fd387e98963d5295c9ecfde80f2e1ee3b7a
2023-05-19 10:38:50 +02:00
paul cannon
c856d45cc0 satellite/overlay: fix GetNodesNetworkInOrder
We were using the UploadSelectionCache previously, which does _not_ have
all nodes, or even all online nodes, in it. So all nodes with less than
MinimumVersion, or with less than MinimumDiskSpace, or nodes suspended
for unknown audit errors, or nodes that have started graceful exit, were
all missing, and ended up having empty last_nets. Even with all that,
I'm kind of surprised how many nodes this involved, but using the upload
selection cache was definitely wrong.

This change uses the download selection cache instead, which excludes
nodes only when they are disqualified, gracefully exited (completely),
or offline.

Change-Id: Iaa07c988aa29c1eb05796ac48a6f19d69f5826c1
2023-05-19 08:08:08 +00:00
Michal Niewrzal
9b3488276d satellite/gracefulexit: use node alias instead id with observer
Using node alias helps using less cpu and memory.

Fixes https://github.com/storj/storj/issues/5654

Change-Id: If3a5c7810732cbb1bff4dcb78706c81aca56b71b
2023-05-18 22:37:46 +00:00
paul cannon
de737bdee9 satellite/repair: add flag for de-clumping behavior
It seems that the "what pieces are clumped" code does not work right, so
this logic is causing repair overload or other repair failures.

Hide it behind a flag while we figure out what is going on, so that
repair can still work in the meantime.

Change-Id: If83ef7895cba870353a67ab13573193d92fff80b
2023-05-18 21:02:36 +00:00
Michal Niewrzal
c0e7f463fe satellite/metabase: remove segmentsloop package
Last change to remove segments loop from codebase.

https://github.com/storj/storj/issues/5237

Change-Id: I77b12911b6b4e390a7385e6e8057c7587e74b70a
2023-05-18 19:08:29 +00:00
Wilfred Asomani
09311b22ca web/satellite: add role column to project members table
A "Role" column has been added to the project members table to indicate
the roles of the members in the project.

Issue: https://github.com/storj/storj/issues/5743

Change-Id: I991b01e3d2cd792c21bd661feb7a225300d9a86b
2023-05-18 16:16:00 +00:00