Commit Graph

7548 Commits

Author SHA1 Message Date
Cameron
b24ce98944 satellite/admin: add Groups to config
add new config to the satellite admin: --admin.groups.limit-update.
This can be used as an alternate means of authentication if the request
is coming from the oauth proxy.

Change-Id: Ic2de13862e6414244b060c66a0f2bed72097cbad
2023-02-27 16:07:20 +00:00
Egon Elbre
1cef2334ff satellite/satellitedb/dbx: document billing
Change-Id: Ief90cc0a6bebaa8df9dd9255565eaa42dc0abc9e
2023-02-27 15:13:23 +00:00
Egon Elbre
4aa519a69f satellite/satellitedb/dbx: document node payments
Change-Id: Ie65ec960f1683f5735a2b5959fedbde326007d04
2023-02-27 14:13:35 +00:00
Vitalii
a1c0fa96d2 web/satellite: S3 credentials step for new access grant flow
Added s3 credentials generated step for new access grant flow.
We show access key secret key and endpoint.
They can be copied or downloaded to .txt file.

Change-Id: I647fb0a86453134702812e8ea9425b8417382a65
2023-02-27 15:15:58 +02:00
Vitalii
430b9646dc web/{satellite, storagenode, multinode}: more linter fixes
Added rule to restrict trailing white spaces.
Added rule to always use '===' instead of '=='.
Added rule to have correct spacing around commas.
Added rule to have spaces around '=>'.
Added rule to not have spaces in parentheses.
Added rule to have space before blocks '{}'.

Change-Id: Ibdbd204d591f0356e0745d89672e0e0579b59ce1
2023-02-27 12:37:32 +00:00
Vitalii
bd0ad21c24 web/satellite: CLI access step for new access grant flow
Added CLI access created step for new access grant flow.
We show satellite address and restricted key there.
They can be copied to clipboard or downloaded.

Change-Id: Ic1942abfa575017fbb57abb9f09e8de66039bb8d
2023-02-27 13:59:47 +02:00
Egon Elbre
1f439f5ae9 satellite/satellitedb/dbx: document audit tables
Change-Id: Iae0f2d7eaddb30cf43a3a7cd678f1dcfb8256a45
2023-02-27 09:56:39 +00:00
JT Olio
8befcaab27 go.mod: bump storj.io/{common,drpc,uplink}
this set of changes re-enables noise with fixes

Change-Id: I122142d202dd76a83b678ac8d709407b0791ed0b
2023-02-24 23:17:16 +00:00
Vitalii
991211a2b0 web/satellite: access created step for new access grant flow
Added access created step to new access grant flow.
Also added some really tiny fixes for current project passphrase flow.

Change-Id: I94eb25b53cf7cc0c6ea7b9c13762ca76736ea7ac
2023-02-24 22:26:42 +00:00
Vitalii
fa2f7336cd web/satellite: encryption info step for new access grant flow
Added middleware info step in case if 'S3' access type is selected.
It's possible to toggle checkbox to not show it again in current browser,

Change-Id: I78b297b373f14715dc67b247baa8e684bea3a265
2023-02-24 21:49:27 +00:00
JT Olio
f3c5cd9df2 cmd/satellite: limit time spent restoring trash from a node
okay to do now that
https://review.dev.storj.io/c/storj/storj/+/7882
is widely deployed

Change-Id: I26c9a982b3999186e7c34a30515d151bf18a8648
2023-02-24 20:56:09 +00:00
Jeremy Wharton
cbbd5ab1ef satellite/payments/stripecoinpayments: undo price override removal
Commit fb59974 disabled usage price overrides because of a failing
test. This change reenables it while resolving the issue that caused
the test to fail.

The previous version of the test passed Gerrit verification and was
merged, but it failed for the primary Jenkins pipeline after merge.

This is due to a difference in how the Jenkins build runs Cockroach
and Postgres for each pipeline.

This commit rewrites the test to be safe for concurrent execution by
ensuring any mutable variables are defined within each test so that
shared state across tests is reduced.

Change-Id: Ia4566c9cd2d698afdb2caa4b7e2808b17e18de4e
2023-02-24 16:54:51 +00:00
Vitalii
17ec326fd4 web/satellite: make inactivity modal appear on top of other modals
This change makes sure that inactivity modal appears on top of everything else including other modals.
If it is the last element of main dashboard wrapper then it will be the last element in DOM tree which makes it appear on top of everything.

Issue:
https://github.com/storj/storj/issues/5549

Change-Id: Ibb0c6e2036c0be09aae17118f672dbed4ffc5b62
2023-02-24 16:03:31 +00:00
paul cannon
20bcdeb8b1 satellite/repair: fix flaky test TestECREpairerGetOffline
It was possible to get into a situation where successfulPieces =
es.RequiredCount(), errorCount < minFailures, and inProgress == 0 (when
the succeeding gets all completed before the failures), whereupon the
last goroutine in the limiter would sit and wait forever for another
goroutine to finish.

This change corrects the handling of that situation.

As an aside, this is really pretty confusing code and we should think
about redoing the whole function.

Change-Id: Ifa3d3ad92bc755e563fd06b2aa01ef6147075a69
2023-02-24 09:05:21 -06:00
Egon Elbre
4a6e34bb2c satellite/satellitedb/dbx: document peer_identity table
Change-Id: I112ef00cb80638a8168dd46a531522b0785b537a
2023-02-24 14:06:46 +00:00
Egon Elbre
cf324c5b6e satellite/satellitedb/dbx: document node tables
Also, fix nodevents iota usage. Type is directly used in the database
hence we should avoid iota, because it's easy to mess-up the constant
values.

Change-Id: I32bf6d0502fbf3d18660dab5b1dd0b5627b444df
2023-02-24 14:05:55 +00:00
Egon Elbre
e4914e9b7a satellite/satellitedb/dbx: document accounting tables
Change-Id: I94478892a37623882d306a811e272f51865e48dd
2023-02-24 16:04:29 +02:00
Márton Elek
dfe3c87b87 satellite/metabase/rangedloop: save observer durations to eventkit
Change-Id: I2d027cd7f7421e2c38dcc1277d640d4183af4cec
2023-02-24 13:30:19 +00:00
Egon Elbre
4e70ef83fb satellite/satellitedb/dbx: split accounting.dbx by project and node
Change-Id: Ia3dee52abf98a41ea62d94af0a155f4f09661224
2023-02-24 09:08:20 +00:00
Cameron
8842985571 satellite/console/consoleweb: create purchase-package endpoint
Add new purchase-package endpoint to Server. The endpoint can be enabled
or disabled by a new config, --console.pricing-packages-enabled.
The purchase-package endpoint applies a coupon and adds and charges a
credit card if user's useragent is a partner with a configured package
plan.

github issue: https://github.com/storj/storj-private/issues/125

Change-Id: I0d6ccccd6874ddba360c45f338fd1c44f95e135a
2023-02-23 15:42:36 -05:00
JT Olio
529e3674e4 storagenode/piecestore: handle upload write if provided in first message
we may in the future want to accept writes and commits as part of the
initial request message, just like
https://review.dev.storj.io/c/storj/storj/+/9245

this change is forward compatible but continues to work with
existing clients.

Change-Id: Ifd3ac8606d498a43bb35d0a3751859656e1e8995
2023-02-23 18:33:08 +00:00
Egon Elbre
1a2ad602d7 satellite/satellitedb/dbx: document gracefulexit tables
Change-Id: Ia76b67fa85e797c1d75438ae882bdfe83a6ee057
2023-02-23 19:14:51 +02:00
Jeremy Wharton
fb5997484e satellite/payments/stripecoinpayments: remove usage price overriding
Project usage price overriding has been removed because it produces
incorrect results when tested. It should not be re-implemented until
the issues it causes are resolved.

Change-Id: Ic92eff374c9af4fea3bf32782a72303a7978b055
2023-02-23 13:36:59 +00:00
Ivan Fraixedes
16d3fcde70 satellite/admin/ui/lib: Fix bug in API
The update user API function was defining a parameter that the API
calling function doesn't receive, so all the values were shifted by
1 after this extra parameter and the calling function was sending these
  values to the next one updating the fields that the user was not
  expecting.

  The API server doesn't allow to update this removed parameter, so it
  was likely a bug caused by copy-pasta.

Change-Id: I52a2b66abc1f6e5cb0070dc6fd230c30d1133dcf
2023-02-23 10:59:18 +00:00
Jeremy Wharton
209b669e84 satellite/console/consoleweb: create and serve frontend config
A new struct containing configuration values for the satellite frontend
is populated and served to the client. In the future, this struct will
be used to configure the frontend rather than using Go templates and
meta tags to do so.

References #5494

Change-Id: Ie2c8a140d53b20fab159d34f2e3767e5dbd43bdb
2023-02-22 23:24:35 -06:00
Vitalii
a9ca643e3f web/satellite: added passphrase generated screen for new access grant flow
Added passphrase generated screen.
Passphrase is a 12-word mnemonic in this case and it is generated with bip39 lib.
It can be copied to clipboard or downloaded as a .txt file.

Change-Id: I031f6c0e92f4f783c07a2d8d35c0433c1d9a81ff
2023-02-22 14:18:34 +02:00
Michal Niewrzal
16b7901fde satellite/metabase: add piece size calculation to segment
This code is essentially replacement for eestream.CalcPieceSize. To call
eestream.CalcPieceSize we need eestream.RedundancyStrategy which is not
trivial to get as it requires infectious.FEC. For example infectious.FEC
creation is visible on GE loop observer CPU profile because we were
doing this for each segment in DB.

New method was added to storj.Redundancy and here we are just wiring it
with metabase Segment.

BenchmarkSegmentPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize-8         	    5822	    189189 ns/op	    9776 B/op	       8 allocs/op
BenchmarkSegmentPieceSize/segment.PieceSize
BenchmarkSegmentPieceSize/segment.PieceSize-8              	94721329	        11.49 ns/op	       0 B/op	       0 allocs/op

Change-Id: I5a8b4237aedd1424c54ed0af448061a236b00295
2023-02-22 11:04:02 +00:00
Michal Niewrzal
3abe7ac0da private/testplanet: add uplink config to use different path cipher
From time to time it would be handy to test uplink operations
without encrypting paths.

Change-Id: I00c64fafa133cb711fcdcfe915c3bfbe5eb75d81
2023-02-22 10:28:25 +00:00
Vitalii
a1f46fe721 web/satellite: enter passphrase step for new access grant flow
Added generic enter passphrase component for new access grant flow to fulfill 2 cases where user enters project level passphrase and a new passphrase.

Change-Id: I10e89d0e32690b0423462ad76c20dafdb6ff096c
2023-02-22 00:00:41 +00:00
Wilfred Asomani
81d7f751da web/satellite: clean up popup constants
This change removes unused modal and dropdown action/mutation constants

Change-Id: Ib4f789ec967727d3619e9fd7769caf99dc9531ac
2023-02-21 23:11:02 +00:00
Cameron
19d4fcc530 satellite/console: return card from AddCreditCard
Change-Id: I1a2667d26e2864606cfafc21a146a6be02798715
2023-02-21 20:48:15 +00:00
Cameron
e2cf486fcb satellite/console: add Purchase method to console.Payments
Purchase collects a payment from user using the specified price,
description, and payment method. This is implemented at the lower layers
using the payments.Invoices interface. There are also additions to
stripecoinpayments stripemock to simulate errors returned from stripe
invoices New and Pay. If an invoice exists with the same description, if
it is a draft, it is deleted and a new one is created and paid. If it is
open, pay it and don't create a new one.

Change-Id: Ic3147434bc44a0777ecbedb3a4ed4c768eb02ea3
2023-02-21 20:10:42 +00:00
Jeremy Wharton
3fa31c2077 satellite/console/consoleweb: remove trailing slash from URLs
This change removes the trailing slash from the account activation and
password recovery URLs, making them consistent with the rest. The URLs'
previous forms are still supported, however, in order to not invalidate
emails containing them.

Resolves storj/customer-issues#491

Change-Id: Ie774a87698d8e9edd1836611968fc3911c6cc56f
2023-02-21 19:15:36 +00:00
Cameron
38f94c1c29 satellite/reputation: if node is DQd or exited skip applying audit
If a node is already disqualified or exited, don't bother going through
the audit reputation code. If there's an additional reputation update to
be made based on the automatic offline audit these nodes would get, the
overlay DB rejects it anyway.

Change-Id: I48ba151aa640b32f87e1285521be465d1f692a78
2023-02-21 18:31:21 +00:00
Cameron
26f80f8e26 satellite/payments: add Invoices Delete
Add Invoices method Delete to delete draft invoices.

Change-Id: I221c6208df98ab11e648d25b94c52a411bf65b9a
2023-02-21 17:54:42 +00:00
Cameron
78e68cdca2 satellite/payments: add constants for invoice status
Change-Id: Iaf94d57d99caff3d435a44c56bd47c7b715ff4c0
2023-02-21 17:04:23 +00:00
JT Olio
f905878ff9 go.mod: bump storj.io/uplink
this change brings in:

 * private/piecestore: disable noise
 * uplink: fix noise version report
 * private/piecestore: fix noise
 * private/metaclient: honor context cancellation on retries
 * private/metaclient: fix multi-passphrase ListObjects paging
 * private/metaclient: fix broken ListUploads method

it also adds:

 * private/storage/streams/batchaggregator: introduce Aggregator
 * private/storage/streams/streambatcher: introduce Batcher
 * private/storage/streams: pieceupload package
 * private/storage/streams: segmenttracker package

but these commits provably change no existing behavior, since they
only add unused, new packages and types.

Change-Id: Id9efc65117b7e1ddcb3ffc8cc15d3e665ed11d7d
2023-02-21 16:09:19 +00:00
Michal Niewrzal
06f4aede9b satellite/audit: use NodeAlias instead pure NodeID
This is first attempt to use AliasPieces inastead Pieces with
segments/range loop. So far we were always using Pieces
which are always converted from AliasPieces for easy use.
Side effect is that using NodeID with loop observers is heavy
e.g. we are using maps which behaves slower with NodeIDs.

We are starting with audit observer because it's easy to change
it as in feact it doesn't need access to real NodeID at all. We just
need to reference node in some way and this way is NodeAlias.

Results of BenchmarkRemoteSegment:
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    1.79µs ± 6%    0.03µs ± 4%  -98.29%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8     0.00B          0.00B          ~     (all equal)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      0.00           0.00          ~     (all equal)

Change-Id: Ib7fc87e568a4d3a9af27b5e3b644ea68ab6db7aa
2023-02-21 15:31:59 +00:00
Vitalii
6f12ba59ba web/satellite: third step on new access grant flow
Added third step of new access grant flow where user selects further passphrase option from this list:
- use existing passphrase
- set my project passphrase
- generate new passphrase
- enter new passphrase

Note: In case of 'use existing passphrase' option access grant will be generated and user will be redirected to success screen (not implemented yet).

Change-Id: Idc238bb469f3e7a87a6523783cee4963bfe0445d
2023-02-21 16:07:42 +02:00
Vitalii
00021e6c75 web/satellite: fix unlock feature inside object browser
Fixed unlock feature inside object browser.

Bug: Clicking unlock button doesn't do anything but it should open switch passphrase modal.

Change-Id: Ie7fa0f842a393c4d887f4dac63fb59edf16229ea
2023-02-21 13:27:26 +00:00
Michal Niewrzal
8aa6110835 tests/backwardcompatibility: use noquic tag for release binaries
Older releases are not compiling with latest Go version if quic is used.
We need to add noquic tag to be able to compile older release with
latest Go version.

Change-Id: Id5768fcaa5c1f7cf3e6fbb633e7ca60309b7a37c
2023-02-21 13:26:46 +01:00
Egon Elbre
0cdef95d55 all: fix math/rand deprecations
Change-Id: I4b966375697c0d409ce24cc7604f806973f8f22a
2023-02-17 15:05:54 +02:00
Michal Niewrzal
aba2f14595 satellite/metabase/rangedloop: few additions for monitoring
Additional elements added:
* monkit metric for observers methods like Start/Fork/Join/Finish to
be able to check how much time those methods are taking
* few more logs e.g. entries with processed range
* segmentsProcessed metric to be able to check loop progress

Change-Id: I65dd51f7f5c4bdbb4014fbf04e5b6b10bdb035ec
2023-02-17 08:46:00 +00:00
Vitalii
9ac314e482 web/satellite: second step of new access grant flow
Added second step where user sets permissions, buckets and date caveats.

Change-Id: I2f959a5bbd46b47b52ea9cafc866b13d28b3e972
2023-02-16 23:12:11 +00:00
Lizzy Thomson
155b927c5e satellite/console: add notifications for free account limits
Add notifications for free account limits for segment usage
and update to follow the figma designs.

Issue: https://github.com/storj/storj/issues/5482

Change-Id: I8a2fe38d609d53e09bf5074484cedc343223bffd
2023-02-16 13:31:01 -07:00
Cameron
d48f745668 satellite/payments: trigger errors in stripemock payment methods
stripecoinpayments.mockStripeClient implements the
stripecoinpayments.StripeClient interface which is used to interact with
stripe's APIs in production. We use it in tests to simulate API calls to
stripe. This change introduces new stripecoinpayments package level
constants: TestPaymentMethodsNewFailure and
TestPaymentMethodsAttachFailure. These can be passed as part of the
arguments their respective stripe mock methods to signal to the method
that it should return an error. This is useful so that we can test how
layers on top of the StripeClient interface handle errors.

Change-Id: Ib64c08ba1e91f31e755b66a1ad563c3b6e77f6f0
2023-02-16 10:00:20 -05:00
Cameron
8d9b61c78e satellite/console: ApplyCoupon and ApplyFreeTierCoupon
Add new console service payments methods ApplyCoupon and
ApplyFreeTierCoupon. ApplyCoupon applies a coupon to an account based on
the coupon ID passed to it. ApplyFreeTierCoupon applies the satellite
configured free tier coupon to the account.

Change-Id: Ic221092278553a79207ac2a0c9229c374d76c881
2023-02-16 10:00:20 -05:00
paul cannon
6bb6479690 satellite/repair: fix flakiness in tests
Several tests using `(*ECRepairer).Get()` have begun to exhibit flaky
results. The tests are expecting to see failures in certain cases, but
the failures are not present. It appears that the cause of this is that,
sometimes, the fastest good nodes are able to satisfy the repairer
(providing RequiredCount pieces) before the repairer is able to identify
the problem scenario we have laid out.

In this commit, we add an argument to `(*ECRepairer).Get()` which
specifies how many failure results are expected. In normal/production
conditions, this parameter will be 0, meaning Get need not wait for
any errors and should only report those that arrived while waiting for
RequiredCount pieces (the existing behavior). But in these tests, we can
request that Get() wait for enough results to see the errors we are
expecting.

Refs: https://github.com/storj/storj/issues/5593
Change-Id: I2920edb6b5a344491786aab794d1be6372c07cf8
2023-02-16 07:33:47 +00:00
paul cannon
ab2e793555 satellite/audit: test delay before Reverify
We are supposed to wait for some amount of time after a timed-out audit
before retrying the audit on the contained node. We are also supposed to
wait for some amount of time before subsequent retries, if they are
necessary. The test added here tries to assure that those delays happen,
as far as it is possible to assure that a delay will happen in computer
code.

The previous behavior of the system was, in fact, to carry out
Reverifies as soon as a worker could retrieve the job from the
reverification queue. That's not a very major problem, as subsequent
retries do have a delay and the node does get several retries. Still, it
was not ideal, and this test exposed that mismatch with expectations, so
this commit includes a minor change to effect that pause between verify
and the first reverify.

Refs: https://github.com/storj/storj/issues/5499
Change-Id: I83bb79c166a458ba59a2db2d17c85eca43ca90f0
2023-02-15 23:16:23 +00:00
Cameron
dd0013bdda bump stripe-go version to v72.90.0
v72.90.0 grants access to InvoiceParams.PendingInvoiceItemsBehavior
which allows the caller to specify whether the invoice should apply all
pending invoice items.

Change-Id: I7c8295d38791ed7daa3d5b1fce15438eb42a962f
2023-02-15 21:16:56 +00:00