Commit Graph

7994 Commits

Author SHA1 Message Date
Jeremy Wharton
b3b619efc5 satellite/admin: use system-given port in OAuth test to fix flakiness
The test for the admin API's OAuth authorization behaviour has been
modified to use a random available port given by the system rather than
a hardcoded one. This prevents the test from accidentally using a port
that is already in use.

Change-Id: Iae017b2f397ae53f1a006bae1d0578d2ddfd0875
2023-04-14 13:14:19 -05:00
Jeremy Wharton
d62dd0b8e7 web/satellite: use frontend config in Vue components
References to the meta tag config values in Vue components have been
modified to instead refer to the frontend config fetched through the
satellite API.

References #5494

Change-Id: I00ecf81d4a0ba6bd07c827cecb2c689d923d67c0
2023-04-14 16:24:05 +00:00
Jeremy Wharton
34e6c5048b web/satellite,satellite/console/consoleweb: update static error pages
Our static error pages have been unified and updated to reflect our new
designs.

Change-Id: Ib7cf76e87c56180dd997a91a2829187dac99baff
2023-04-14 15:48:15 +00:00
Wilfred Asomani
4ee22e0ed8 satellite/admin: add tests to admin auth
This change tests authorization of the admin api.

Issue: https://github.com/storj/storj/issues/5699

Change-Id: Iecfe4c27a70ab1b48aeb5ed3251b51a3406140e8
2023-04-14 11:13:14 +00:00
Jeremy Wharton
49c5e3de9e web/satellite: update Vue error page design
A common error page component that reflects our new designs has been
added, replacing our old 404 and 50X Vue pages.

Change-Id: Ifc9071674eeda03c5d961c26dce9ff0c80b95c6e
2023-04-14 00:07:14 -05:00
Clement Sam
f076238748 storagenode: run used-space filewalker as a low IO subprocess
As part of fixing the IO priority of filewalker related
processes such as the garbage collection and used-space
calculation, this  change allows the initial used-space
calculation to run as a separate subprocess with lower
IO priority.

This can be enabled with the `--storage2.enable-lazy-filewalker`
config item. It falls back to the old behaviour when the
subprocess fails.

Updates https://github.com/storj/storj/issues/5349

Change-Id: Ia6ee98ce912de3e89fc5ca670cf4a30be73b36a6
2023-04-14 04:16:14 +00:00
Vitalii
7dc2a5e20b web/satellite: use access grants pinia module instead of old vuex module
Start using new pinia module instead of old vuex module

Change-Id: I72e98fcf6910b1a601d005ecc24cec37a8016478
2023-04-13 23:59:48 +00:00
Egon Elbre
eecb055dfd satellite/buckets: move Bucket definition
Move Bucket struct definition.

Updates https://github.com/storj/storj/issues/5291

Change-Id: I6bfc5ce287793ea479f2cb8b17878ba3cf6b63e0
2023-04-13 17:55:40 -04:00
Jeff Wendling
54ef1c8ca2 cmd/uplink: use new upload code path
the parallelism and parallelism-chunk-size flags
which used to control how many parts to split a
segment into and many to perform in parallel
are now deprecated and replaced by
maximum-concurrent-pieces and long-tail-margin.

now, for an individual transfer, the total number
of piece uploads that transfer will perform is
controlled by maximum-concurrent-pieces, and
segments within that transfer will automatically
be performed in parallel. so if you used to set
your parallelism to n, a good value for the pieces
might be something approximately like 130*n, and
the parallelism-chunk-size is unnecessary.

Change-Id: Ibe724ca70b07eba89dad551eb612a1db988b18b9
2023-04-13 16:52:38 -04:00
Jeremy Wharton
eeeac5fa39 web/satellite: disable removal of SVG viewBox
The webpack loader we use for SVGs uses an optimizer that strips the
viewBox attribute by default in order to reduce the byte count.
However, this prevents us from scaling SVGs using CSS. Because of this,
viewBox removal has been disabled.

Change-Id: I443eb83c762d3e0c84bb0fb6705e8f089d8a9405
2023-04-13 13:22:19 -05:00
Jeremy Wharton
34b2a369f7 satellite/payments/stripe: apply egress discount to project charges
The project charge information used by the satellite frontend to
compute project cost estimates has been updated to account for
configured egress discounts.

Resolves storj/storj-private#215

Change-Id: Ic90b015d65f5bea104ac96fb0cea545b3f9f1f8f
2023-04-13 17:34:50 +00:00
JT Olio
16ccffb6f7 satellite/metainfo: test to confirm that retried pieces can't be submitted with originals
closes https://github.com/storj/storj/issues/5709

Change-Id: I2030825d85987d97ae8c10f02ab1f70086cbcb97
2023-04-13 14:48:32 +00:00
Michal Niewrzal
a21afeddd1 satellite/payments/stripe: avoid full table scan while listing
Query to list (with pages) stripe customers were doing full table scan
because Offset clause was used. This refactoring changed listing to
use cursor instead Offset.

Change-Id: I14688e6c533bc932ba0d209a061562f080b4cf54
2023-04-13 12:36:31 +02:00
Vitalii
fc3d31e732 web/satellite: use app pinia module instead of vuex module
Use new pinia module instead of old vuex module.
Removed 'do you want to leave object browser with ongoing uploads?' feature for now because it doesn't make sense with project level passphrase

Change-Id: I2249ee9a497d3fa7c59b583849e70366491c3eeb
2023-04-12 17:08:50 +00:00
Egon Elbre
46debe7f24 ci: bump release to Go 1.20.3
Change-Id: I9fe3d45f53d8624d6fe9d6eb9a5f2e017012bd47
2023-04-12 19:06:26 +03:00
Jeremy Wharton
c92fdba972 satellite/payments/stripe: apply egress discount to invoice items
Invoicing has been modified to account for an egress discount ratio,
which specifies the fraction of the amount of data stored that egress
usage should be discounted.

The egress discount ratio measures discounted egress units per storage
unit-months, so a ratio of 0.5 with 2 MB-months of storage would
discount 1 MB of egress.

Resolves storj/storj-private#224

Change-Id: I43d7d6719391c303712c082709aef77249c65f62
2023-04-12 15:28:00 +00:00
Jeremy Wharton
6e9dae06d5 satellite/payments: add egress discount ratio to price override config
This change allows for specifying the ratio of free egress per unit
of storage within a price override configuration.

References storj/storj-private#215
References storj/storj-private#224

Change-Id: Ib1c79f77ec8bb11dd5b2f9dace13800b0b3ce942
2023-04-12 14:52:19 +00:00
Vitalii
64c798e912 web/satellite: use billing pinia module instead of vuex module
Use new billing pinia module instead of old payments vuex module

Change-Id: I03ee443345259ca96414bfb0771ace751ad36e58
2023-04-12 11:07:39 +00:00
Vitalii
e25da61b11 web/satellite: use project members pinia module instead of vuex module
Start using pinia module instead of vuex module

Change-Id: Ic51ecbf58583957360d4b5ac9c6e21885fbb0522
2023-04-12 11:06:17 +00:00
Jeremy Wharton
59eab37b95 satellite/console/consoleweb: remove bucket usage report page
No component has referenced this page since 9dab10e and we do not
anticipate this changing, so this page can be safely removed.

Resolves #5768

Change-Id: I57acb5e4d0977d74df46aaf67606a19ec0f10bcf
2023-04-11 18:17:03 +00:00
Lizzy Thomson
1c2ad79bf8 web/satellite: fix banner container CSS issue
Hides empty banner container when there are no banners being displayed

Change-Id: Ib64912e80ff1d8073ecfeb772939a0bce781d684
2023-04-11 10:54:58 -06:00
Egon Elbre
8fba740332 storagenode/blobstore/testblobs: don't error checks in BadDB
We automatically start a chore to check whether the blobstore is
writeable and readable, however, we don't want to fail the tests due to
that reason. Usually we want to test some other failure.

There probably should be some nicer way to achieve this, but this is an
easier fix.

Change-Id: I77ada75329f88d3ea52edd2022e811e337c5255a
2023-04-11 16:14:31 +03:00
Lizzy Thomson
e2ecee41b5 web/satellite: update notifications to be aligned with dashboard
Update notification and banners to be aligned with inner dashboard
content

Issue https://github.com/storj/storj/issues/5636

Change-Id: I0e9fe73835e691bfa9aae05c1cfb08a5b9ab68e3
2023-04-11 10:50:52 +00:00
Vitalii
9c20c4ecb4 web/satellite: use pinia users module instead of vuex users module
Start using pinia users module instead of vuex pinia module.

Change-Id: I2a61aafa4b8f71cccd2a5825e11b315f59767289
2023-04-11 07:55:40 +00:00
Vitalii
5d860fddbd web/satellite: introduce project members pinia module
Added new pinia module to replace old vuex module in the future.

Change-Id: I73a5f6a36bc661a278591112ecd572e94700c442
2023-04-11 07:07:04 +00:00
Vitalii
199a28eb44 web/satellite: introduce billing pinia module
Added new billing pinia module to replace vuex payments module in the future

Change-Id: I7b03d6bd6f946c8e1d75c7501c0a32dde5b96832
2023-04-11 06:14:28 +00:00
Vitalii
cf1677fd05 web/satellite: combine buckets and objects pinia modules. Rename files module to object browser module
Combined buckets and objects pinia modules since they are responsible for the same thing.
Rename files module to object browser module to make it more clear.

Change-Id: Id55ec2ab7dbffaec95e5724340068891752a9576
2023-04-11 05:20:45 +00:00
Moby von Briesen
8893576dd2 web/satellite: Add clarification to pro account description
Clarify that bandwidth cost/limits relate specifically to egress
bandwidth (ingress bandwidth is free and does not affect bw limit).

Change-Id: I95a8a07ac3607ff41417d894067800f81246b7a2
2023-04-10 17:15:09 +00:00
Andrew Harding
8c5924d6ea satellite/payments/billing: add bonus transactions
- only applies to storjscan transactions
- applies a 10% bonus by default
- bonus transactions have a distinct source "type" to allow for
  filtering on the frontend

Fixes: https://github.com/storj/storj/issues/5702

Change-Id: I32d65f776c58bcb41227ff5bc77a8e4cb62a9add
2023-04-10 16:26:56 +00:00
Andrew Harding
e676b5c893 cmd/uplink: progress bars for recursive copy
```
$ uplink cp -r -t=3 files/ sj://files
uploading 6 files...
files/bar/buz   (2 of 6) 134.22 MB / 134.22 MB [============================================] 100.00% 36.43 MiB/s
files/bar/baz   (1 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 18.39 MiB/s
files/boo       (3 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 20.42 MiB/s
files/foo       (4 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 57.83 MiB/s
files/glue/flew (5 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 55.01 MiB/s
files/stew      (6 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 91.43 MiB/s
```

Change-Id: Ibd9d07a1291f7a599bd27fba93c1b2e0f17dc787
2023-04-10 15:13:22 +00:00
Moby von Briesen
97cefed5c2 web/satellite: Remove BillingNotification component
This notification has been around for long enough that it is no longer
necessary.

Change-Id: I747d11f872d5bb1643dba599bafb36ac1b8ad9ee
2023-04-10 14:12:45 +00:00
Wilfred Asomani
085bc0c4cb web/satellite: fix browser hover and click behaviour
This change adds checkboxes to file items in the browser so they can be
selected that way instead of the previous click behaviour, which now
opens the object modal for files and open folder for folders. This
change also features some styling fixes for the browser.

Issue: https://github.com/storj/storj/issues/5726

Change-Id: I5b38208a8e8673d8212c749586bdb7e169c086f8
2023-04-09 21:03:46 +00:00
JT Olio
1d63395fd1 storagenode/peer: don't require CA whitelist any longer
this change makes it so that the storage node no longer
cares if the cert of peers it talks to has been signed
by the sno registration server. this is fine because
the only reason a storage node would talk to a peer
besides the explicitly configured satellites is because
a satellite told it to.

we have already disabled this on uplinks (uplinks don't
care about the peer ca whitelist), and we are starting
to consider disabling this on satellites entirely.
however, before we really can disable it on satellites,
we need to disable it on storage nodes so that graceful
exit and node to node transfers can work correctly.

Change-Id: I2e0a0781bd247e574b82f0065aafb88804e59c71
2023-04-07 21:53:00 +00:00
Wilfred Asomani
5b65e10563 satellite/console: enable session timeout configuration
This another account endpoint; patch /auth/account/settings. to handle
changing a user's settings, including their session timeout config.

Issue: https://github.com/storj/storj/issues/5560

Change-Id: I747b4e919cf7cef7c867ac9d282837ef51bed67e
2023-04-07 21:05:36 +00:00
Jeremy Wharton
b2a4a3543f testsuite/playwright-ui: add .gitignore
This change adds a .gitignore file to the testsuite/playwright-ui
directory. It is configured to ignore Node dependencies and test
results.

Change-Id: Ib200128ebe30064b1827fd71a5a7df163dcb4f69
2023-04-07 17:44:13 +00:00
Wilfred Asomani
6f8dff5832 web/satellite: enable session timeout setting
This change adds a new setting on the account settings page to change
session timeout duration. The old settings page is replaced with a new
one used on the all projects dashboard page. Also, onboarding API
endpoints and store action have been changed to be generic to include
session timeout setting.

Issue: https://github.com/storj/storj/issues/5560

Change-Id: I9026e61c6f86e4be5f9357e5d20e280eab2c29ea
2023-04-07 00:03:17 +00:00
Jeremy Wharton
857a292e52 web/satellite: show loading screen when fetching frontend config
The satellite UI now shows a loading screen instead of the Vue router
view when fetching the frontend config. This prevents components that
rely on the config from being prematurely rendered.

References #5494

Change-Id: Ifeb06288a6895f98a7cab8f321cd68078c73c696
2023-04-06 21:24:58 +00:00
paul cannon
915f3952af satellite/repair: repair pieces on the same last_net
We avoid putting more than one piece of a segment on the same /24
network (or /64 for ipv6). However, it is possible for multiple pieces
of the same segment to move to the same network over time. Nodes can
change addresses, or segments could be uploaded with dev settings, etc.
We will call such pieces "clumped", as they are clumped into the same
net, and are much more likely to be lost or preserved together.

This change teaches the repair checker to recognize segments which have
clumped pieces, and put them in the repair queue. It also teaches the
repair worker to repair such segments (treating clumped pieces as
"retrievable but unhealthy"; i.e., they will be replaced on new nodes if
possible).

Refs: https://github.com/storj/storj/issues/5391
Change-Id: Iaa9e339fee8f80f4ad39895438e9f18606338908
2023-04-06 17:34:25 +00:00
Michal Niewrzal
8a50a3baa3 satellite/payments: rename 'stripecoinpayments' package to 'stripe'
Automatic rename. May require some more cleanups later.

Change-Id: I18220a4278056d25c41fb137832bb81f2b876ac1
2023-04-06 16:51:43 +00:00
Michal Niewrzal
31f5e2cb65 satellite/metainfo: limit uploads to the same location
We would like to have ability to limit burst uploads to the single
object (the same location). This change we are limiting such upload to
one per second.

Change-Id: Ib9351df1017cbc07d7fc2f846c2dbdbfcd3a360c
2023-04-06 15:49:11 +00:00
Egon Elbre
c1f1aacffe private/kvstore: move storage package
There's no reason it should be at the top-level.

Change-Id: I35b06e7baa0e425c6ff9a82964d0a1570d4eb6d0
2023-04-06 17:26:29 +03:00
Egon Elbre
48256c91b5 storage: move errors to better locations
Change-Id: Ia44570949a8f6bb50220dc838c5b6aa21e851a4d
2023-04-06 17:26:29 +03:00
Egon Elbre
ea4a9e61c3 storage: delete unused code
Change-Id: Ic85a09fa31ff0bf3e99bce685a14c03598c2a962
2023-04-06 17:26:29 +03:00
Egon Elbre
5498424e1f storage: implement Range
By implementing a simpler Range, we can get rid of many methods
defined for storage.KeyValueStore.

Initially KeyValueStore was used to manage objects, hence it required
much more complex behaviour. However, we've moved away from this
implementation, hence we can significantly simplify everything.

Change-Id: Iaaa8c9c83d8601ad3c894767f17c0f5b356f7253
2023-04-06 17:26:29 +03:00
Márton Elek
e5995947d0 go.mod: bump storj.io/private and storj.io/uplink
Change-Id: I01790c1af8d6302b3c9ffee0482c952bb9a5d8da
2023-04-06 13:44:34 +00:00
Clement Sam
3cf89633e9 cmd/storagenode: refactor main.go
The cmd/storagenode/main.go is a big mess right now with so many
unneeded config structures initialized and shared by several
subcommands.

There are many instances where the config structure of one subcommand
is mistakenly used for another subcommand.

This changes is an attempt to clean up the main.go by moving the
subcommands to a separate `cmd_*.go` files with separate config structures
for each subcommand.

Resolves https://github.com/storj/storj/issues/5756

Change-Id: I85adf2439acba271c023c269739f7fa3c6d49f9d
2023-04-06 12:48:23 +00:00
Vitalii
45a8ac7f57 web/satellite: migrate CreateProject component to use SFC composition api
Change-Id: I2e2bb654d1d759fb66ecbcbece1afb38557e271b
2023-04-06 11:53:19 +00:00
Vitalii
1a8e5544b9 web/satellite: migrate ProjectDashboard component to use SFC composition api
Change-Id: I8f7a523d0ea80664a7ccd6ebfb74f46a2553ac31
2023-04-06 11:52:28 +00:00
Wilfred Asomani
1dc5930d6c web/satellite: fix clicks on project/billing history tables
This change fixes an issue where clicking on the name of a project will
perform no action. And another where clicking "Invoice Pdf" on billing
history table will download twice.

Issue: https://github.com/storj/storj/issues/5698

Change-Id: I1a05b994d4855da5be309b1b6d4eb4c70f6f946b
2023-04-06 11:03:23 +00:00
Wilfred Asomani
ff88404d32 web/satellite: change project limit upperbound
This change makes the upperbound of the project limit slider the max
between the current project limit and the default limit for paid tier.

Issue: https://github.com/storj/storj/issues/5553

Change-Id: I6ca31ec86810ee3577370ab5cd042fce8daf1b90
2023-04-06 10:10:22 +00:00