Commit Graph

8151 Commits

Author SHA1 Message Date
Michal Niewrzal
87d0789691 satellite/payments/stripe: avoid full table scan while listing records
Stripe invoice project records while listing are causing full table scan
because of OFFSET caluse. This change is refactoring query to list using
cursor.

Change-Id: I6b73b9b2815173d7ef02cf615408778476eb3b7b
2023-05-09 14:16:55 +00:00
Clement Sam
c64f3f3132 {storagenode/console,web/storagenode}: fetch pricing model from storagenode API
Instead of the hardcoded payout rates that is assumed for all satellites,
this change adds a new endpoint for fetching the pricing model for
each satellite.
The pricing model is then displayed on the Info & Estimation table
on the dashboard

Updates https://github.com/storj/storj-private/issues/245

Change-Id: Iac7669e3e6eb690bbaad6e64bbbe42dfd775f078
2023-05-09 13:25:44 +00:00
Clement Sam
d80d674863 web/storagenode: get current month earnings from estimated payout
The current month estimated earnings on the main dashboard
was calculated on the frontend using hardcoded payout rates for
all satellites.
Instead, we could just use the calculate the current month earnings
from the estimated payout info.

Updates https://github.com/storj/storj-private/issues/245

Change-Id: Ie8ec3e4f2b86455c68f664690ef28bb352107bca
2023-05-09 12:54:20 +00:00
Clement Sam
1003d8213c storagenode/blobstore/filestore: add debug log to WalkNamespace when dir does not exist
This is particularly useful for monitoring the lazyfilewalker to
make sure it is not checking the wrong directory.

Updates https://github.com/storj/storj/issues/5349

Change-Id: I7e5fcfd4545ec4157d33a9225cd1bce607ccd154
2023-05-09 10:27:13 +00:00
Márton Elek
291b541960 earthfile: remove storage folder
Content of storage folder is moved to the storagenode with c1f1aacff (as it's used only from there).
We don't need to include it manually for storagenode builds.

This Earthfile is used for nightly deployment.

Change-Id: I2d255804ce5d87ab5a9b713c2d5ae0f3b3ccc9ca
2023-05-09 09:46:43 +00:00
Michal Niewrzal
fa16d947c9 satellite/satellitedb: add index on owner_id column for projects table
We have method which is getting projects owned by specific user but it's
causing full table scan because we don't have index on owner_id column.

Change-Id: Icb71c9ac5b73104a52241ed8ba126c995c10811f
2023-05-09 08:57:32 +00:00
Wilfred Asomani
5145be1eeb satellite/console: proper constraint error check for duplicate project members
The string check previously used to check for constraint errors is now
replaced with dbx.IsConstraintError check.

Change-Id: I553ccd69e3c02b6b54441bd9f929b85a155eaf00
2023-05-08 22:23:00 +00:00
Clement Sam
018b6eeeaf storagenode: add tests for lazyfilewalker
Updates https://github.com/storj/storj/issues/5349

Change-Id: I9544c14ba2acacd5b304f151ab29c70ff61adc5b
2023-05-08 21:50:40 +00:00
dlamarmorgan
cf5d2d792c satellite/payments/stripe/service.go: fix payment for multiple invoices
Fix an error that can occur when processing multiple invoices for the same user in a single invoice cycle when the user is paying with Storj tokens.

Change-Id: I54af8c7dde1965d994f687fdfc4e4b5ef4deeb2d
2023-05-08 21:17:14 +00:00
Egon Elbre
8beb78ec3f {satellite,multinode}/console: fix WriteHeader
w.Header().Set needs to be called before WriteHeader,
because WriteHeader sends all the headers and calls to
Set won't have any effect afterwards.

Change-Id: Ia6b1c5e2cd54201a6c3980d63de04a0095b2db9a
2023-05-08 18:18:30 +00:00
Jeremy Wharton
9c753163c2 satellite/console/dbcleanup: make chore clean up project invites
The console DB cleanup chore has been extended to remove old project
member invitation records.

Resolves #5816

Change-Id: Id0a748e40f5acf03b9b903265c653b072846ba19
2023-05-08 16:36:16 +00:00
Clement Sam
cf7ce81d09 cmd/storagenode: refactor lazyfilewalker commands to satisfy the execwrapper.Command interface
Follow-up change for https://review.dev.storj.io/c/storj/storj/+/10335

Updates https://github.com/storj/storj/issues/5349

Change-Id: Iadf55bae84ebc0803a0766830e596c396dfb332b
2023-05-08 15:09:53 +00:00
Márton Elek
56dbe7738d cmd/storagenode: use the right subcommand for restarting windows services
Commit 3cf89633e9 is changed how the cobra subcommands are created for storagenode (with prefering local variables instead of package level variables).

However, there is a bug which makes it impossible to restart Storagenode services on Windows: the refactored code creates the rootCmd/runCmd twice: therefore the ctx of the running process is not exactly the same as the ctx which supposed to be stopped / cancelled.

This patch fixes this problem with re-using exising, initialized command instead of creating a new one for cancellation.

Fixes: https://github.com/storj/storj/issues/5845

Change-Id: Ib8a4b80d4574e448f65c8558e927c0908c9c5eed
2023-05-08 14:09:37 +00:00
Clement Sam
5977357a60 web/storagenode: hide payout rates for historical payout data
This is a temporal fix to buy us some time while we fix the issues.

We may revert later or maybe override with a later commit.

Updates https://github.com/storj/storj-private/issues/245

Change-Id: I27b2b6811d24d5948161553bf889fe45af013925
2023-05-08 12:53:37 +00:00
Michal Niewrzal
36e046375c satellite/repair/checker: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I8583549973cd36aa0e0c482c20d7a75cb7568ab3
2023-05-08 12:19:13 +00:00
Cameron
e8e6dd056a web/satellite: remove project member modal
Create modal for removing project members

issue: https://github.com/storj/storj/issues/5751

Change-Id: I621b8e1de54270faebb7577b8431903c62a6e45c
2023-05-08 11:36:56 +00:00
Wilfred Asomani
fb1a0cc784 web/satellite: redesign and allow enter passphrase skip
This change allows users to skip the enter passphrase modal and opt in
for that choice to be remembered. The enter passphrase modal is also
redesigned.

Issue: https://github.com/storj/storj/issues/5818
Issue: https://github.com/storj/storj/issues/5616

Change-Id: Ie1cf6602682af3e9625656455fa3c096a24b95d3
2023-05-08 10:52:49 +00:00
Clement Sam
291e639ac2 storagenode/pieces/lazyfilewalker: add execwrapper package
The execwrapper package wraps the exec.Cmd and has a Command
interface that mimics the behaviour of the exec.Cmd.
This is useful for testing the lazyfilewalker subprocesses
by stubbing instead of spawning a real subprocess.

Updates https://github.com/storj/storj/issues/5349

Change-Id: I14084139c76a531f2b6d7163f9aa35c3f5e192d7
2023-05-06 02:02:23 +00:00
Clement Sam
ec8bfe6b94 storagenode/pieces: capture logger time Key in zapwrapper
Updates https://github.com/storj/storj/issues/5349

Change-Id: I426f38c0ae0f93d498317e3f66ba4f5724620758
2023-05-06 02:02:23 +00:00
Jeff Wendling
6ac72e42cd all: bump storj.io/drpc
this includes a fix to cancellation when soft cancel
is enabled.

Change-Id: I1b9ab071f623153a2bd715888f97ca20277a76d7
2023-05-06 01:02:50 +00:00
Moby von Briesen
432b12d1c4 satellite/satellitedb: make value_attributions.partner_id nullable
There is still a reference to partner_id in a query, which we cannot
move until dropping the "not null" constraint for it. This change adds
that migration.

Related to https://github.com/storj/storj/issues/5432

Change-Id: I98802a6e1bd59f3d9214de3db6688d9daf664a70
2023-05-05 23:42:50 +00:00
Wilfred Asomani
e3573f44ea web/satellite: update project member invite modal
The project member invite modal has been updated to match the new figma
design.

Issue: https://github.com/storj/storj/issues/5744

Change-Id: I9ed7ad4ede0fb6f7f42f56b2d8386e12a5a5b6c1
2023-05-05 19:46:28 +00:00
Jeremy Wharton
f61230a670 satellite/console/dbcleanup: create console DB cleanup chore
A chore responsible for purging data from the console DB has been
implemented. Currently, it removes old records for unverified user
accounts. We plan to extend this functionality to include expired
project member invitations in the future.

Resolves #5790
References #5816

Change-Id: I1f3ef62fc96c10a42a383804b3b1d2846d7813f7
2023-05-05 19:11:53 +00:00
Cameron
2bb636684e web/satellite: update create bucket modal
issue: https://github.com/storj/storj/issues/5817

Change-Id: Ia86e0bad8de180fb4e6dde3ed194c9ce6f4f12f1
2023-05-05 17:50:58 +00:00
Wilfred Asomani
4ab433ba89 satellite/{web,console}: make duplicate member error readable
This change makes the error thrown when adding an existing member to a
project readable.

Issue: https://github.com/storj/storj/issues/5840

Change-Id: I4269495f9b7b09c77fbf1af1fc605e5c95bd7cbf
2023-05-05 16:46:12 +00:00
Egon Elbre
8b82dba602 storagenode/blobstore/filestore: add tracking of blobs
We've had issues with forgetting to close readers and writers.
Add leak tracking to find those pesky issues.

Change-Id: If6b0ad6e9958318a7e0affee9c6d0a1ece412b6d
2023-05-05 15:40:15 +03:00
Clement Sam
569c639cd5 web/storagenode: hide payout rates when no specific satellite is selected
We should not display payout rates when "All Satellites" option is
selected since some of the satellites may have different payout
rates.

Updates https://github.com/storj/storj-private/issues/245

Change-Id: I66cadf85aa8b57681ffbeac75c4ad97f8b057676
2023-05-05 01:00:41 +00:00
Wilfred Asomani
bf05040dd6 satellite/{console,db}: allow passphrasepromt toggling
This change adds the user's passphrase prompt setting to the
/account/settings endpoints.

Issue: https://github.com/storj/storj/issues/5616

Change-Id: I48d470d49e82096fd090b74da323b279e342546e
2023-05-04 19:26:54 +00:00
Paul Willoughby
d8081169cd satellite/metainfo: increase default MaxEncryptedObjectKeyLength
Allow a longer encrypted key length to reduce 'key length is too big'
errors in gateway-mt.  Gateway is enforcing an unencrypted key length
of 1024 bytes but when encrypted some keys are exceeding the current
limit.

updates https://github.com/storj/gateway-mt/issues/335

Change-Id: I861a2313e558c9f7d39569d21c7a3d429c83575c
2023-05-04 12:37:13 -06:00
Vitalii
a8ba7aa263 web/satellite: small styling updates for access grant flow
Fixed Copy and Download buttons.
Fixed wording styling.
Also fixed VButton's hover state if there is an icon.

Issue:
https://github.com/storj/storj/issues/5774

Change-Id: I9fbac12fe433817f4b5fc1d885f50e6f27e7f6a0
2023-05-04 17:33:06 +00:00
Vitalii
73898e0d86 web/satellite: make graph line more smooth
Made lines to be more smooth and also updates points styling

Issue:
https://github.com/storj/storj/issues/5826

Change-Id: I34229e772ceee5e48330f5662a584cd9dcd8c63d
2023-05-04 15:20:31 +00:00
Egon Elbre
511d74a76d satellite/metainfo: hardcode the prefix
This avoids a potential bug with regards to not returning the prefix
that the uplink expects.

Change-Id: I64683db57c5b22989db91b4f8e08530bf3b64ff1
2023-05-04 17:06:51 +03:00
Clement Sam
8a206f0aee web/multinode: fix wrong undistributed payout amount
Resolves https://github.com/storj/storj/issues/5849

Change-Id: I074f6ca9f9ac1e8ffa62649188a6059d9bcd4c25
2023-05-04 10:36:10 +00:00
Clement Sam
162af1e836 web/multinode: fix bandwidth graph disappearing on state change
Resolves https://github.com/storj/storj/issues/5847

Change-Id: I6731e7a39cd3ffedaff76b307c141a9e94d9b71f
2023-05-04 10:36:10 +00:00
Clement Sam
dab4288899 web/multinode: fix used space graph showing 0
Resolves https://github.com/storj/storj/issues/5566

Change-Id: Ib9c81fa2c92ed71ccf07e7a0c9b6f61c46294f09
2023-05-04 08:46:47 +00:00
Moby von Briesen
a229feac1e satellite/console: Pass pricing packages enabled config to frontend
Ensure that the value of "pricing packages enabled" flag on frontend is
the same as what is configured on the backend.

Change-Id: Id78771800a4973ebd3ad4e22f1953f6f71c75dd4
2023-05-03 18:13:38 -04:00
Wilfred Asomani
034431e4a8 web/satellite: update session timeout modals
This change opens up the "set session timeout" modal for users to set a
custom timeout duration if they haven't already.
It also fixes a few UI/UX issues on the modal including toggling on the
dashboard.

Issue: https://github.com/storj/storj/issues/5770

Change-Id: I0e71e191049b242e638ca36214d6dd33f78ae5fe
2023-05-03 13:59:50 +00:00
Clement Sam
e0542c2d24 storagenode: run garbage collection filewalker as a low I/O subprocess
Updates https://github.com/storj/storj/issues/5349

Change-Id: I7d810d737b17f0b74943765f7f7cc30b9fcf1425
2023-05-02 19:43:38 +00:00
Lizzy Thomson
25f2305e00 web/satellite: removed google fonts from satellite emails
update fonts in satellite emails by removing google fonts

Issue - https://github.com/storj/storj/issues/5690

Change-Id: I9243c172e52d780800300dca2ec78b5665979eda
2023-05-02 16:10:34 +00:00
Wilfred Asomani
bf630b7efa web/satellite: update team page design
This change the projects members (team) page to match the figma.

Issue: https://github.com/storj/storj/issues/5742

Change-Id: I5d4ef4478612a5c397b795c9b0b9b364fd16e4a4
2023-05-02 15:34:02 +00:00
Egon Elbre
ee28103b29 satellite/metabase: fix error message
The "object not found" included an additional prefix "metabase:", which
broke uplink error message detection.

Ideally, changing an internal error message shouldn't break the uplink.

Change-Id: I5ce7789cc11742d3435af1ec555bc96927f1bedc
2023-05-02 13:53:57 +00:00
Moby von Briesen
90692362c6 web/satellite: fix link
Link to end-to-end encryption page on docs.storj.io is incorrect. This
change fixes it.

Change-Id: I61c942e2d2b3bcd9ad0d51451027207a9eeb36f0
2023-04-28 18:51:22 -04:00
Jeff Wendling
80b3edf1d1 storagenode/piecestore: respect maximum chunk size requests
See https://review.dev.storj.io/c/storj/common/+/10297 for
more details.

Change-Id: Id5d19f029ae872780a554874592679191c1b5b2f
2023-04-28 16:31:41 -04:00
Vitalii
3f1166b5aa web/satellite: added update your session timeout banner
Added new banner to inform user that they can update their session timeout now.

Issue:
https://github.com/storj/storj/issues/5772

Change-Id: Icdf2164b80b12954d004537a4f31d30ef6bb12b8
2023-04-28 11:17:06 +00:00
Michal Niewrzal
98562d06c8 satellite/gc/bloomfilter: add sync observer
Current observer used with ranged loop is using massive amount of
memory because each range is generating separate set of bloom filters.
Each bloom filter can be up to 2MB of memory. That's a lot.

This change is initial change to reduce used memory by sharing bloom
filters between ranges and just synchronize access to them. This
implementation is rather simple and even naive but maybe it will be
enough without doing something more complex.

https://github.com/storj/storj/issues/5803

Change-Id: Ie62d19276aa9023076b1c97f712b788bce963cbe
2023-04-28 07:40:56 +00:00
Vitalii
30dee52256 web/satellite: rework delete access grant modal
Reworked delete access grant modal to be more simple.

Issue:
https://github.com/storj/storj/issues/5775

Change-Id: I415e75458736522f17914f252be531db5a2313f7
2023-04-27 21:26:38 +00:00
Vitalii
978b714128 web/satellite: rework open bucket modal styling
Reworked open bucket modal styling to correspond to create access grant flow

Issue:
https://github.com/storj/storj/issues/5819

Change-Id: I618b50472e45413a63a531ad959d9af69511d464
2023-04-27 16:10:27 +00:00
dlamarmorgan
b89e0e0272 payments/stripe: Remove apply coupon from generate invoice command
Remove the command to apply free tier coupons from the generate invoices
command. Applying free tier coupons should instead be done outside the
normal time window for invoice generation to save time during the
invoicing cycle.

Change-Id: If8fecc558411d5a6fff9d5689143d72f3b709e55
2023-04-27 08:24:12 -07:00
Michal Niewrzal
3cd79d987d satellite/gc/bloomfilter: extract BF upload logic
This is refactor/cleanup change before I will start working on adding
separate GC observer with optimized memory consumption.

https://github.com/storj/storj/issues/5803

Change-Id: I854cb3797802a32942c25f2765dbb72be88bacbd
2023-04-27 11:15:27 +02:00
Vitalii
48f920809f web/satellite: rework styling of manage project passphrase modals
Reworked styling of manage project passphrase modals to crrespond to create access grant flow.

Issue:
https://github.com/storj/storj/issues/5815

Change-Id: Iab02d2d94239d9e8c7f85a498fb3a3d83ece34e0
2023-04-26 20:39:11 +00:00