Commit Graph

7980 Commits

Author SHA1 Message Date
Michal Niewrzal
9b3488276d satellite/gracefulexit: use node alias instead id with observer
Using node alias helps using less cpu and memory.

Fixes https://github.com/storj/storj/issues/5654

Change-Id: If3a5c7810732cbb1bff4dcb78706c81aca56b71b
2023-05-18 22:37:46 +00:00
paul cannon
de737bdee9 satellite/repair: add flag for de-clumping behavior
It seems that the "what pieces are clumped" code does not work right, so
this logic is causing repair overload or other repair failures.

Hide it behind a flag while we figure out what is going on, so that
repair can still work in the meantime.

Change-Id: If83ef7895cba870353a67ab13573193d92fff80b
2023-05-18 21:02:36 +00:00
Michal Niewrzal
c0e7f463fe satellite/metabase: remove segmentsloop package
Last change to remove segments loop from codebase.

https://github.com/storj/storj/issues/5237

Change-Id: I77b12911b6b4e390a7385e6e8057c7587e74b70a
2023-05-18 19:08:29 +00:00
Wilfred Asomani
09311b22ca web/satellite: add role column to project members table
A "Role" column has been added to the project members table to indicate
the roles of the members in the project.

Issue: https://github.com/storj/storj/issues/5743

Change-Id: I991b01e3d2cd792c21bd661feb7a225300d9a86b
2023-05-18 16:16:00 +00:00
Michal Niewrzal
cf5ff537e3 cmd/tools/metabase-verify: switch to ranged loop
We would like to remove segments loop so we need to refactor
our tools to use ranged loop.

To simplify change ranged loop is used with single range only.

https://github.com/storj/storj/issues/5237

Change-Id: I94d96d54f9d0e37b06def4f4fc16b71c5b79baba
2023-05-18 14:27:38 +00:00
Moby von Briesen
19cb08b025 web/satellite: add "public linksharing URL"
* Update defaults for gateway credentials URL and linksharing URL to use
  storjsatelliteshare.io instead of storjshare.io
* Add new config for "public linksharing URL" and set it to
  link.storjshare.io
* Use "private" linksharing URL for actions within the object browser
* Use "public" linksharing URL for sharing files externally

Resolves https://github.com/storj/storj/issues/5805

Change-Id: I2c8fbd04141755b4751dcf4d054253a7ff8d6cf3
2023-05-18 07:02:26 +00:00
Wilfred Asomani
a2a603aee8 web/satellite: rightly navigate to pricing plan page
A navigation is prevented to the onboarding if no project is selected.
However, an exception was not made for the pricing plan page and caused
the pricing plan page not to show even under passing conditions. This
change makes that exception for the pricing plan page.

Change-Id: I72ddb942a0059d748a13da0b6038b99b9b5064af
2023-05-17 19:35:37 +00:00
Vitalii
ace0ef89a2 web/satellite: migrated Vue 2 to Vue 3
Finally migrated web/satellite UI to use Vue 3.

Issue:
https://github.com/storj/storj/issues/5862

Change-Id: I609e2332142b9092d6efeb2d4b931e34cce846cc
2023-05-17 17:31:03 +00:00
paul cannon
958d8676d0 satellite/overlay: remove unnecessary test helper
Change-Id: I8439eec4ed440f60353fc620ca906a917a03613c
2023-05-17 17:04:54 +00:00
paul cannon
1f4f79b6b3 satellite/repair: don't mark clumped segments as irreparable
Clumped segments (segments with multiple pieces on the same subnet) may
need repair, but the clumped pieces are considered retrievable and we
don't need to call such segments irreparable.

We do want to know where they're coming from, though, if we can, because
we are seeing more than expected.

Change-Id: I41863b243f4bb007ef8929191a3fde1562565ef9
2023-05-17 16:24:15 +00:00
Jeremy Wharton
607b120116 satellite/satellitedb: add inviter ID column to project invites table
The project member invitations table has been modified to contain a
column for the ID of the user who sent the invitation. This ID is
required for us to return information about the inviter to the
satellite frontend.

References #5855

Change-Id: I928d987a8db2340f731ca65ce30173d4f90a9837
2023-05-17 15:46:40 +00:00
Lizzy Thomson
f87772a0d3 satellite/console: bug fix for using a API key to retrieve a session
Change-Id: I0202f75fb78f42af9e7cb00bb3ac73839340c977
2023-05-16 14:06:32 -06:00
paul cannon
75d10fe4fa satellite/overlay: use UploadSelectionCache for GetNodesNetworkInOrder
The query for GetNodesNetworkInOrder is causing far too much load on the
database. Since it is not critical that the repair checker have
perfectly up-to-date node network information, we can use a cache
instead.

Change-Id: I07ad45bfdeb46529da093941a06c2da8a00ce878
2023-05-16 17:32:09 +00:00
Wilfred Asomani
e9fc4e9d0f web/satellite: fix billing page nav on all dashboard
This change fixes an issue where clicking "Billing" on all projects
dashboard won't navigate to the billing page.
This also makes sure an error notification for projectID is not shown
after the user is back on the all projects dashboard from team page.

related: https://github.com/storj/storj/issues/5875

Change-Id: Ie6517c9006d1a780f307f6282f83c749d369adaf
2023-05-16 15:14:18 +00:00
Márton Elek
f1b9f6b672 satellite/satellitedb: paginating on ordered invoice
87d0789691 replaces offset usage with cursors.

But to continue the interation from a specific cursor, we need to iterate over ordered records.

(at least this is what I understood based on the failing tests)

87d0789691

Change-Id: Ic4da3a7c5f03386dd4c373c05102f05871900a3a
2023-05-16 14:05:34 +00:00
Michal Niewrzal
4bdbb25d83 satellite/metabase/rangedloop: move Segment definition
We will remove segments loop soon so we need first to move
Segment definition to rangedloop package.

https://github.com/storj/storj/issues/5237

Change-Id: Ibe6aad316ffb7073cc4de166f1f17b87aac07363
2023-05-16 12:37:17 +00:00
Lizzy Thomson
7e69b22dd4 satellite/console: retrieve a web UI session using a API key
retrieve a web UI session using a API key and add new test

Issue https://github.com/storj/storj/issues/5688

Change-Id: I0ad9c783f0573b87b212b8fb065cdbf7074b782c
2023-05-15 10:24:58 -06:00
Wilfred Asomani
81b2b067e6 satellite/satellitedb/dbx/gen: update to work with updated dbx
This change updates the replacer in satellite/satellitedb/dbx/gen/main.go
to work with an updated dbx.

Change-Id: I08e89d6d27e6f1d435416105fe5f622009add7ad
2023-05-15 14:53:06 +00:00
JT Olio
e90612c285 docs/blueprint: partial rollout automation
Change-Id: I355d7071da7e501948665ee153ed641722a3d264
2023-05-15 13:49:47 +00:00
Michal Niewrzal
2592aaef9c satellite/gc/bloomfilter: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I1a22ac4b242998e287b2b7d8167b64e850b61a0f
2023-05-15 11:46:26 +00:00
Wilfred Asomani
5fede0ce95 web/satellite: fix no project id provided error
This change fixes an error when all projects dashboard is enabled where
there will be no project selected and an error "projectid not provided"
will be thrown.
It navigates back to the all projects dashoard so the user can select
a project.

Issue: https://github.com/storj/storj/issues/5875

Change-Id: I144c4bbd3782dcdf40b2ed2be445c7f100c36b32
2023-05-15 10:23:32 +00:00
Wilfred Asomani
331a3348a0 web/satellite: mod project options menu for all projects dashboard
The project options has been modified to account for whether the all
projects dashboard is enabled or not. The "Manage projects" item is
hidden, and an "All projects" item is added if it is enabled.

Issue: https://github.com/storj/storj/issues/5866

Change-Id: Ic2e8337d2e5577d1c27231287ab3284145fa5b9b
2023-05-12 20:10:33 +00:00
Wilfred Asomani
b11ba50b94 web/satellite: hide sidebar on onboarding
The sidebar previously was showing on the onboarding pages when all
projects dashboard is enabled. This change hides it and also hides the
"session timeout" info banner which was also showing on onboarding.

Issue: https://github.com/storj/storj/issues/5874

Change-Id: Ie418a6f286262b59dce41aba98a5c9d0015cb984
2023-05-12 18:57:02 +00:00
Vitalii
277a4c6aa3 web/satellite: updated project dashboard data cards
Updated cards to show access grants count, buckets count, team size and billing status for current project

Issue:
https://github.com/storj/storj/issues/5865

Change-Id: I7e8d3aa3e05548d593576c3a0633e9eb51f38cff
2023-05-12 14:43:09 +03:00
Vitalii
083b3d6fc1 web/satellite: added modal to update single project limit
Added new modal where user can update storage or bandwidth limit (if Pro tier).

Issue:
https://github.com/storj/storj/issues/5827

Change-Id: Ic3ae6b7a73d055b4eac93d4e2faad7bf26cb9d73
2023-05-12 11:38:39 +00:00
Vitalii
b91f72f08a web/satellite: added limit cards on project dashboard
Added storage, bandwidth, segment, free tier and coupon cards to project dashboard.

Issue:
https://github.com/storj/storj/issues/5827

Change-Id: Iaf974273a34c8ed3faf1974876fa685074d5ae61
2023-05-12 14:36:53 +03:00
paul cannon
c2710cc78d satellite/audit: improve error handling
* Don't use rpcstatus.Unknown as an indicator of dial failure; instead,
  GetShare now indicates with a per-share field where a failure happened
  (DialFailure, RequestFailure, NoFailure). Use that information in
  Verify() to determine how to treat the source node.

* Add a test that replaces a storage node with a black hole, so that
  connections there will always time out. Make sure we handle that case
  correctly.

Refs: https://github.com/storj/storj/issues/5632
Change-Id: I513a53520fb48b7187d4c4d7e14e01e2cfc0a721
2023-05-11 22:55:26 +00:00
igor gaidaienko
bc30deee11 satellite/repair: update repair test to use blake3 algo
Update repair test to also use blake3 hashing algorithm
https://github.com/storj/storj/issues/5649

Change-Id: Id8299576f8be4cfd84ddf9a6b852e653628ada72
2023-05-11 08:45:48 +00:00
Cameron
52ff7a66a0 web/satellite: use color palette instead of hex colors for button
Change-Id: I1da9c1927c2768e86f3887477250ae833e29439e
2023-05-10 17:54:26 +00:00
Michal Niewrzal
87d0789691 satellite/payments/stripe: avoid full table scan while listing records
Stripe invoice project records while listing are causing full table scan
because of OFFSET caluse. This change is refactoring query to list using
cursor.

Change-Id: I6b73b9b2815173d7ef02cf615408778476eb3b7b
2023-05-09 14:16:55 +00:00
Clement Sam
c64f3f3132 {storagenode/console,web/storagenode}: fetch pricing model from storagenode API
Instead of the hardcoded payout rates that is assumed for all satellites,
this change adds a new endpoint for fetching the pricing model for
each satellite.
The pricing model is then displayed on the Info & Estimation table
on the dashboard

Updates https://github.com/storj/storj-private/issues/245

Change-Id: Iac7669e3e6eb690bbaad6e64bbbe42dfd775f078
2023-05-09 13:25:44 +00:00
Clement Sam
d80d674863 web/storagenode: get current month earnings from estimated payout
The current month estimated earnings on the main dashboard
was calculated on the frontend using hardcoded payout rates for
all satellites.
Instead, we could just use the calculate the current month earnings
from the estimated payout info.

Updates https://github.com/storj/storj-private/issues/245

Change-Id: Ie8ec3e4f2b86455c68f664690ef28bb352107bca
2023-05-09 12:54:20 +00:00
Clement Sam
1003d8213c storagenode/blobstore/filestore: add debug log to WalkNamespace when dir does not exist
This is particularly useful for monitoring the lazyfilewalker to
make sure it is not checking the wrong directory.

Updates https://github.com/storj/storj/issues/5349

Change-Id: I7e5fcfd4545ec4157d33a9225cd1bce607ccd154
2023-05-09 10:27:13 +00:00
Márton Elek
291b541960 earthfile: remove storage folder
Content of storage folder is moved to the storagenode with c1f1aacff (as it's used only from there).
We don't need to include it manually for storagenode builds.

This Earthfile is used for nightly deployment.

Change-Id: I2d255804ce5d87ab5a9b713c2d5ae0f3b3ccc9ca
2023-05-09 09:46:43 +00:00
Michal Niewrzal
fa16d947c9 satellite/satellitedb: add index on owner_id column for projects table
We have method which is getting projects owned by specific user but it's
causing full table scan because we don't have index on owner_id column.

Change-Id: Icb71c9ac5b73104a52241ed8ba126c995c10811f
2023-05-09 08:57:32 +00:00
Wilfred Asomani
5145be1eeb satellite/console: proper constraint error check for duplicate project members
The string check previously used to check for constraint errors is now
replaced with dbx.IsConstraintError check.

Change-Id: I553ccd69e3c02b6b54441bd9f929b85a155eaf00
2023-05-08 22:23:00 +00:00
Clement Sam
018b6eeeaf storagenode: add tests for lazyfilewalker
Updates https://github.com/storj/storj/issues/5349

Change-Id: I9544c14ba2acacd5b304f151ab29c70ff61adc5b
2023-05-08 21:50:40 +00:00
dlamarmorgan
cf5d2d792c satellite/payments/stripe/service.go: fix payment for multiple invoices
Fix an error that can occur when processing multiple invoices for the same user in a single invoice cycle when the user is paying with Storj tokens.

Change-Id: I54af8c7dde1965d994f687fdfc4e4b5ef4deeb2d
2023-05-08 21:17:14 +00:00
Egon Elbre
8beb78ec3f {satellite,multinode}/console: fix WriteHeader
w.Header().Set needs to be called before WriteHeader,
because WriteHeader sends all the headers and calls to
Set won't have any effect afterwards.

Change-Id: Ia6b1c5e2cd54201a6c3980d63de04a0095b2db9a
2023-05-08 18:18:30 +00:00
Jeremy Wharton
9c753163c2 satellite/console/dbcleanup: make chore clean up project invites
The console DB cleanup chore has been extended to remove old project
member invitation records.

Resolves #5816

Change-Id: Id0a748e40f5acf03b9b903265c653b072846ba19
2023-05-08 16:36:16 +00:00
Clement Sam
cf7ce81d09 cmd/storagenode: refactor lazyfilewalker commands to satisfy the execwrapper.Command interface
Follow-up change for https://review.dev.storj.io/c/storj/storj/+/10335

Updates https://github.com/storj/storj/issues/5349

Change-Id: Iadf55bae84ebc0803a0766830e596c396dfb332b
2023-05-08 15:09:53 +00:00
Márton Elek
56dbe7738d cmd/storagenode: use the right subcommand for restarting windows services
Commit 3cf89633e9 is changed how the cobra subcommands are created for storagenode (with prefering local variables instead of package level variables).

However, there is a bug which makes it impossible to restart Storagenode services on Windows: the refactored code creates the rootCmd/runCmd twice: therefore the ctx of the running process is not exactly the same as the ctx which supposed to be stopped / cancelled.

This patch fixes this problem with re-using exising, initialized command instead of creating a new one for cancellation.

Fixes: https://github.com/storj/storj/issues/5845

Change-Id: Ib8a4b80d4574e448f65c8558e927c0908c9c5eed
2023-05-08 14:09:37 +00:00
Clement Sam
5977357a60 web/storagenode: hide payout rates for historical payout data
This is a temporal fix to buy us some time while we fix the issues.

We may revert later or maybe override with a later commit.

Updates https://github.com/storj/storj-private/issues/245

Change-Id: I27b2b6811d24d5948161553bf889fe45af013925
2023-05-08 12:53:37 +00:00
Michal Niewrzal
36e046375c satellite/repair/checker: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I8583549973cd36aa0e0c482c20d7a75cb7568ab3
2023-05-08 12:19:13 +00:00
Cameron
e8e6dd056a web/satellite: remove project member modal
Create modal for removing project members

issue: https://github.com/storj/storj/issues/5751

Change-Id: I621b8e1de54270faebb7577b8431903c62a6e45c
2023-05-08 11:36:56 +00:00
Wilfred Asomani
fb1a0cc784 web/satellite: redesign and allow enter passphrase skip
This change allows users to skip the enter passphrase modal and opt in
for that choice to be remembered. The enter passphrase modal is also
redesigned.

Issue: https://github.com/storj/storj/issues/5818
Issue: https://github.com/storj/storj/issues/5616

Change-Id: Ie1cf6602682af3e9625656455fa3c096a24b95d3
2023-05-08 10:52:49 +00:00
Clement Sam
291e639ac2 storagenode/pieces/lazyfilewalker: add execwrapper package
The execwrapper package wraps the exec.Cmd and has a Command
interface that mimics the behaviour of the exec.Cmd.
This is useful for testing the lazyfilewalker subprocesses
by stubbing instead of spawning a real subprocess.

Updates https://github.com/storj/storj/issues/5349

Change-Id: I14084139c76a531f2b6d7163f9aa35c3f5e192d7
2023-05-06 02:02:23 +00:00
Clement Sam
ec8bfe6b94 storagenode/pieces: capture logger time Key in zapwrapper
Updates https://github.com/storj/storj/issues/5349

Change-Id: I426f38c0ae0f93d498317e3f66ba4f5724620758
2023-05-06 02:02:23 +00:00
Jeff Wendling
6ac72e42cd all: bump storj.io/drpc
this includes a fix to cancellation when soft cancel
is enabled.

Change-Id: I1b9ab071f623153a2bd715888f97ca20277a76d7
2023-05-06 01:02:50 +00:00
Moby von Briesen
432b12d1c4 satellite/satellitedb: make value_attributions.partner_id nullable
There is still a reference to partner_id in a query, which we cannot
move until dropping the "not null" constraint for it. This change adds
that migration.

Related to https://github.com/storj/storj/issues/5432

Change-Id: I98802a6e1bd59f3d9214de3db6688d9daf664a70
2023-05-05 23:42:50 +00:00