Commit Graph

7568 Commits

Author SHA1 Message Date
Cameron
19d4fcc530 satellite/console: return card from AddCreditCard
Change-Id: I1a2667d26e2864606cfafc21a146a6be02798715
2023-02-21 20:48:15 +00:00
Cameron
e2cf486fcb satellite/console: add Purchase method to console.Payments
Purchase collects a payment from user using the specified price,
description, and payment method. This is implemented at the lower layers
using the payments.Invoices interface. There are also additions to
stripecoinpayments stripemock to simulate errors returned from stripe
invoices New and Pay. If an invoice exists with the same description, if
it is a draft, it is deleted and a new one is created and paid. If it is
open, pay it and don't create a new one.

Change-Id: Ic3147434bc44a0777ecbedb3a4ed4c768eb02ea3
2023-02-21 20:10:42 +00:00
Jeremy Wharton
3fa31c2077 satellite/console/consoleweb: remove trailing slash from URLs
This change removes the trailing slash from the account activation and
password recovery URLs, making them consistent with the rest. The URLs'
previous forms are still supported, however, in order to not invalidate
emails containing them.

Resolves storj/customer-issues#491

Change-Id: Ie774a87698d8e9edd1836611968fc3911c6cc56f
2023-02-21 19:15:36 +00:00
Cameron
38f94c1c29 satellite/reputation: if node is DQd or exited skip applying audit
If a node is already disqualified or exited, don't bother going through
the audit reputation code. If there's an additional reputation update to
be made based on the automatic offline audit these nodes would get, the
overlay DB rejects it anyway.

Change-Id: I48ba151aa640b32f87e1285521be465d1f692a78
2023-02-21 18:31:21 +00:00
Cameron
26f80f8e26 satellite/payments: add Invoices Delete
Add Invoices method Delete to delete draft invoices.

Change-Id: I221c6208df98ab11e648d25b94c52a411bf65b9a
2023-02-21 17:54:42 +00:00
Cameron
78e68cdca2 satellite/payments: add constants for invoice status
Change-Id: Iaf94d57d99caff3d435a44c56bd47c7b715ff4c0
2023-02-21 17:04:23 +00:00
JT Olio
f905878ff9 go.mod: bump storj.io/uplink
this change brings in:

 * private/piecestore: disable noise
 * uplink: fix noise version report
 * private/piecestore: fix noise
 * private/metaclient: honor context cancellation on retries
 * private/metaclient: fix multi-passphrase ListObjects paging
 * private/metaclient: fix broken ListUploads method

it also adds:

 * private/storage/streams/batchaggregator: introduce Aggregator
 * private/storage/streams/streambatcher: introduce Batcher
 * private/storage/streams: pieceupload package
 * private/storage/streams: segmenttracker package

but these commits provably change no existing behavior, since they
only add unused, new packages and types.

Change-Id: Id9efc65117b7e1ddcb3ffc8cc15d3e665ed11d7d
2023-02-21 16:09:19 +00:00
Michal Niewrzal
06f4aede9b satellite/audit: use NodeAlias instead pure NodeID
This is first attempt to use AliasPieces inastead Pieces with
segments/range loop. So far we were always using Pieces
which are always converted from AliasPieces for easy use.
Side effect is that using NodeID with loop observers is heavy
e.g. we are using maps which behaves slower with NodeIDs.

We are starting with audit observer because it's easy to change
it as in feact it doesn't need access to real NodeID at all. We just
need to reference node in some way and this way is NodeAlias.

Results of BenchmarkRemoteSegment:
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    1.79µs ± 6%    0.03µs ± 4%  -98.29%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8     0.00B          0.00B          ~     (all equal)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      0.00           0.00          ~     (all equal)

Change-Id: Ib7fc87e568a4d3a9af27b5e3b644ea68ab6db7aa
2023-02-21 15:31:59 +00:00
Vitalii
6f12ba59ba web/satellite: third step on new access grant flow
Added third step of new access grant flow where user selects further passphrase option from this list:
- use existing passphrase
- set my project passphrase
- generate new passphrase
- enter new passphrase

Note: In case of 'use existing passphrase' option access grant will be generated and user will be redirected to success screen (not implemented yet).

Change-Id: Idc238bb469f3e7a87a6523783cee4963bfe0445d
2023-02-21 16:07:42 +02:00
Vitalii
00021e6c75 web/satellite: fix unlock feature inside object browser
Fixed unlock feature inside object browser.

Bug: Clicking unlock button doesn't do anything but it should open switch passphrase modal.

Change-Id: Ie7fa0f842a393c4d887f4dac63fb59edf16229ea
2023-02-21 13:27:26 +00:00
Michal Niewrzal
8aa6110835 tests/backwardcompatibility: use noquic tag for release binaries
Older releases are not compiling with latest Go version if quic is used.
We need to add noquic tag to be able to compile older release with
latest Go version.

Change-Id: Id5768fcaa5c1f7cf3e6fbb633e7ca60309b7a37c
2023-02-21 13:26:46 +01:00
Egon Elbre
0cdef95d55 all: fix math/rand deprecations
Change-Id: I4b966375697c0d409ce24cc7604f806973f8f22a
2023-02-17 15:05:54 +02:00
Michal Niewrzal
aba2f14595 satellite/metabase/rangedloop: few additions for monitoring
Additional elements added:
* monkit metric for observers methods like Start/Fork/Join/Finish to
be able to check how much time those methods are taking
* few more logs e.g. entries with processed range
* segmentsProcessed metric to be able to check loop progress

Change-Id: I65dd51f7f5c4bdbb4014fbf04e5b6b10bdb035ec
2023-02-17 08:46:00 +00:00
Vitalii
9ac314e482 web/satellite: second step of new access grant flow
Added second step where user sets permissions, buckets and date caveats.

Change-Id: I2f959a5bbd46b47b52ea9cafc866b13d28b3e972
2023-02-16 23:12:11 +00:00
Lizzy Thomson
155b927c5e satellite/console: add notifications for free account limits
Add notifications for free account limits for segment usage
and update to follow the figma designs.

Issue: https://github.com/storj/storj/issues/5482

Change-Id: I8a2fe38d609d53e09bf5074484cedc343223bffd
2023-02-16 13:31:01 -07:00
Cameron
d48f745668 satellite/payments: trigger errors in stripemock payment methods
stripecoinpayments.mockStripeClient implements the
stripecoinpayments.StripeClient interface which is used to interact with
stripe's APIs in production. We use it in tests to simulate API calls to
stripe. This change introduces new stripecoinpayments package level
constants: TestPaymentMethodsNewFailure and
TestPaymentMethodsAttachFailure. These can be passed as part of the
arguments their respective stripe mock methods to signal to the method
that it should return an error. This is useful so that we can test how
layers on top of the StripeClient interface handle errors.

Change-Id: Ib64c08ba1e91f31e755b66a1ad563c3b6e77f6f0
2023-02-16 10:00:20 -05:00
Cameron
8d9b61c78e satellite/console: ApplyCoupon and ApplyFreeTierCoupon
Add new console service payments methods ApplyCoupon and
ApplyFreeTierCoupon. ApplyCoupon applies a coupon to an account based on
the coupon ID passed to it. ApplyFreeTierCoupon applies the satellite
configured free tier coupon to the account.

Change-Id: Ic221092278553a79207ac2a0c9229c374d76c881
2023-02-16 10:00:20 -05:00
paul cannon
6bb6479690 satellite/repair: fix flakiness in tests
Several tests using `(*ECRepairer).Get()` have begun to exhibit flaky
results. The tests are expecting to see failures in certain cases, but
the failures are not present. It appears that the cause of this is that,
sometimes, the fastest good nodes are able to satisfy the repairer
(providing RequiredCount pieces) before the repairer is able to identify
the problem scenario we have laid out.

In this commit, we add an argument to `(*ECRepairer).Get()` which
specifies how many failure results are expected. In normal/production
conditions, this parameter will be 0, meaning Get need not wait for
any errors and should only report those that arrived while waiting for
RequiredCount pieces (the existing behavior). But in these tests, we can
request that Get() wait for enough results to see the errors we are
expecting.

Refs: https://github.com/storj/storj/issues/5593
Change-Id: I2920edb6b5a344491786aab794d1be6372c07cf8
2023-02-16 07:33:47 +00:00
paul cannon
ab2e793555 satellite/audit: test delay before Reverify
We are supposed to wait for some amount of time after a timed-out audit
before retrying the audit on the contained node. We are also supposed to
wait for some amount of time before subsequent retries, if they are
necessary. The test added here tries to assure that those delays happen,
as far as it is possible to assure that a delay will happen in computer
code.

The previous behavior of the system was, in fact, to carry out
Reverifies as soon as a worker could retrieve the job from the
reverification queue. That's not a very major problem, as subsequent
retries do have a delay and the node does get several retries. Still, it
was not ideal, and this test exposed that mismatch with expectations, so
this commit includes a minor change to effect that pause between verify
and the first reverify.

Refs: https://github.com/storj/storj/issues/5499
Change-Id: I83bb79c166a458ba59a2db2d17c85eca43ca90f0
2023-02-15 23:16:23 +00:00
Cameron
dd0013bdda bump stripe-go version to v72.90.0
v72.90.0 grants access to InvoiceParams.PendingInvoiceItemsBehavior
which allows the caller to specify whether the invoice should apply all
pending invoice items.

Change-Id: I7c8295d38791ed7daa3d5b1fce15438eb42a962f
2023-02-15 21:16:56 +00:00
JT Olio
77bf88e916 satellite/overlay: check node difficulty before entering database
closes https://github.com/storj/storj/issues/5568

Change-Id: Id413637c2678e7a7cf8dbf414e082c687c8e8a39
2023-02-15 17:46:25 +00:00
Michal Niewrzal
0a525292e4 satellite/metainfo: fix duplicates while listing pending objects
While working on fixing listing for committed objects we didn't fix
the same case for pending objects. For case were we have many
pending objects under different locations we need to set cursor
version to highest value to avoid duplicates.

For case where we have many pending objects under the same location
we will need to make a separate fix.

https://github.com/storj/storj/issues/5570

Change-Id: Id5c8eb728868e8e1177fdbcf65a493142be4eaf0
2023-02-15 16:52:53 +00:00
Wilfred Asomani
9138e84fb1 satellite/{web,console}: feature flag for all projects dashboard
Adds a feature flag for the new all projects dashboard. It defaults to false.

Issue: https://github.com/storj/storj/issues/5514

Change-Id: I160904eccae7d30e05b734e69600725702b16aca
2023-02-15 15:53:31 +00:00
Stefan Benten
f3c752091f Jenkinsfile,Makefile: bump to go v1.19.6
Change-Id: I9654f691538e73d30569dc1a3f430795ee9ff7b7
2023-02-15 14:00:52 +00:00
Lizzy Thomson
1b22994631 satellite/admin: update user's limits and project limits from admin api
Create an endpoint in the Admin Api to be able to update a user’s limits
for all existing and new projects. Also added a GET endpoint to return
user's limits.

Fixes: https://github.com/storj/storj/issues/5395

Change-Id: I2c093dc08ebf79a4318391e63a37da4d2b403547
2023-02-15 11:38:57 +00:00
Malcolm Bouzi
109da3c798 web/satellite: refactor appstate store module to use enum for popups
This change refactors appstate, removing the big list of booleans
Previously used to toggle modals and dropdowns, replacing them with enums.

Issue: https://github.com/storj/storj/issues/5244

Change-Id: I4cffb7ab3ad7712f7ff79dd7486df938ca63830e
2023-02-15 09:25:39 +00:00
paul cannon
0e1c99f75a storage/filestore: fix panic on fs error in EmptyTrash
fileInfo is, of course, nil here, so we can't use fileInfo.Name() to
report the location of the problem.

Change-Id: Ia858a3795b127da0fc812d0a158b36ad344ff76b
2023-02-14 18:09:19 -06:00
Vitalii
8c7e031ee3 web/satellite: extended open bucket functionality
Upon opening bucket we try to decrypt objects with provided passphrase.
If overall object count is higher than decrypted objects count then we show a passphrase warning.
Also updated banner inside object browser if there are unencrypted objects.
Added 'Object locked' line item which toggles switch passphrase modal on click.

Issue:
https://github.com/storj/storj/issues/5513

Change-Id: I00fce23cee04d4e2f09cc4612f733318a9f270fc
2023-02-14 19:04:36 +02:00
Michal Niewrzal
41bcc6bb62 satellite/metainfo: fix duplicates while listing committed objects
We have an issue where object can appear in two different listing pages.
It's because protobuf listing cursor doesn't have version included and
now we can have internally versions higher than 1. On satellite side
version 1 was always used as a default cursor version.

As a workaround for existing implementation of libuplink library we will
use always maximum version for listing cursor on satellite side.

Fixing protobuf and libuplink implementation will happen later.

https://github.com/storj/storj/issues/5570

Change-Id: Ibd27b174556c9d8b8bd60fab8cff7862fd11e994
2023-02-14 14:47:27 +01:00
Wilfred Asomani
4a67b57103 satellite/{console,web}: make bucket endpoints support publicID
This change adds support for project public id to the bucket-names and
usage-report endpoints.

Issue: https://github.com/storj/storj/issues/5578

Change-Id: I2429ebebe52dfc8217fc40f4691e7bc473b805fb
2023-02-13 23:00:03 +00:00
Michal Niewrzal
94d341bcf3 satellite: use ranged loop with GC-GF peer
Peer for generating bloom filters will be able to use ranged loop.

As an addition some cleanup were made:
* remove unused parts of GC BF peer (identity, version control)
* added missing Close method for ranged loop service
* some additional tests added

https://github.com/storj/storj/issues/5545

Change-Id: I9a3d85f5fffd2ebc7f2bf7ed024220117ab2be29
2023-02-13 18:32:21 +00:00
Márton Elek
8f8e97de23 satellite/metainfo: support desired node number for download object/segment
This modification introduce support of the new "desired node" field of download segment/object.

This can be used to request more nodes than the suggested minimum. It can be used to achieve better performance in exchange of using more bandwidth. (more parallel downloads).

Change-Id: Ia167d6979e6d70a597c85070a4ccd1c3a573e406
2023-02-13 13:57:48 +00:00
Márton Elek
ca6e3a9e88 satellite/orders: create mock based unit test
Most of our (~integration) tests based on testplanet runner.

However running testplanet for each test make the testing process slow.
It seems to be better to use real unit tests (without db dependency) when it's possible.

This patch makes small modification to make it possible to test orders.Service with real unit test.

As the existing unit test of `service.go` is isolated with `_test` package name, it's moved to an `_integration_test.go` file to make place for the unit test.

Change-Id: Ia69f26a34e2c48d230d8d36c2040dd02a60455a6
2023-02-13 13:24:30 +00:00
JT Olio
ca13eca718 go.mod: bump libuplink to include noise
Change-Id: If5bceb139ce6fdf6c0792b4bb536bc61b54e32bb
2023-02-10 17:03:51 +00:00
Cameron
ce23ce0dc9 satellite/nodeevents: minor changes to dedupe node IDs
`idsMap` renamed to `seen`. The map key is now storj.NodeID rather than
string.

Change-Id: Ia120d21c94eeecd1b855910bf27123dcd1dc7f97
2023-02-10 16:20:10 +00:00
Egon Elbre
491fa78840 satellite/satellitedb/dbx: add comments for node_reputation.dbx
Change-Id: If9c2bb605a9c562c114a59c3f3cec0b01b765689
2023-02-10 17:10:34 +02:00
Stefan Benten
f104e68c8d Jenkinsfile,Makefile: bump to v1.19.5
Change-Id: I1e936a2c25f569a47bf2c98a49aefe83679c7deb
2023-02-10 15:36:07 +02:00
Egon Elbre
a8caea8b9c satellite/satellitedb: remove gob migration exceptions
Change-Id: I2958e2145edaa501d510314f9bfd0e5280ea82ec
2023-02-10 15:05:18 +02:00
paul cannon
362d73f783 satellite/audit: add tests for concurrent audits
This commit introduces tests that perform multiple concurrent audits
against the same storage node, to make sure that doing so does not
create incorrect outcomes.

Refs: https://github.com/storj/storj/issues/5495
Change-Id: Iaae49e042306bfa59bdf04c1a1540667488e51e5
2023-02-09 19:58:50 +00:00
Moby von Briesen
4cb825a6ea satellite/console/consoleweb: Add user ID rate limiting to some endpoints
Add rate-limiting based on UserID to add credit card and change password
endpoints.

Change-Id: I5d493aeeefc03729abb66266b5e3fd9715d592ad
2023-02-09 18:50:13 +00:00
Egon Elbre
b14019b8c5 satellite/{metabase/rangedloop,metainfo/piecedeletion}: fix flaky tests
TestLoopContinuesAfterObserverError was failing due to system
granularity measuring the duration as 0.

TestDialer_DialTimeout was failing due to connection failure came with a
delay and wasn't being handled.

Change-Id: I4638c86f5d021a86c3d3529fab13cf3608f35c40
2023-02-09 16:07:00 +02:00
Vitalii
450772f034 web/satellite: first step of new access grant flow
Added icons and typings.
Added a first step (Create a new access) of the new flow.

Change-Id: Ic60073dd20fd9ac8728151eb9a2f182366c46e99
2023-02-09 14:39:59 +02:00
Kaloyan Raev
b1d4a159a6 satellite/metainfo: ListUploads returns incorrect UploadID
ListUploads returns incorrect UploadID if Expires was set in
BeginUpload. DB is truncating expiration date to microseconds precision
so we need to do this also in code.

Change-Id: Iee0cf45cb705342f6bb9a2f745acca91cce6ff52
2023-02-09 10:33:08 +00:00
Vitalii
65e3cfb9c6 satellite/{web, console}: feature flag for new access grant flow
Added a feature flag for new create access grant flow.
Also added some initial setup.

Change-Id: I7f738181c8a83f5a724f9e562427445cae146b6f
2023-02-09 00:35:35 +00:00
Cameron
38275bd710 satellite/nodeevents: deduplicate node IDs
Remove dupulicate node IDs when sending notifications.

Change-Id: I76c642e342081f8fdf8248443de8383b9e88eed2
2023-02-08 23:14:18 +00:00
Lizzy Thomson
f733cb255b satellite/satellitedb: drop unused last_verification_reminder column
(updated to migration step 225) remove users table column
last_verification_reminder and update migrate_test.go

Issue: https://github.com/storj/storj/issues/4560

Change-Id: Icbd8b70a8243b75a11c481a3b87df0ebe9402c5f
2023-02-08 16:23:09 +00:00
Egon Elbre
dd3d051170 satellite/satellitedb/dbx: add docs for user.dbx
Change-Id: I346ceaedf6963169bb9d6d1435454016239d4cb1
2023-02-08 17:30:29 +02:00
Egon Elbre
a1bd36bb64 satellite/satellitedb/dbx: add docs for repair.dbx
Change-Id: Icffcdebb7dcc1d10eb8c97212acc6ae309b2f92c
2023-02-08 17:30:27 +02:00
Egon Elbre
0e00c7b8da certificate/authorization,cmd/certificates: remove gob code
Now that we've migrated data we can remove gob handling.

Change-Id: I55f772b7a0dda71c51db0683bad3db66b89867ac
2023-02-08 15:05:40 +00:00
Michal Niewrzal
252c437b0e satellite/orders: ignore context canceled when updating bucket bandwidth
Orders from storage nodes are received by SettlementWithWindowFinal method. There is a stream which receives all orders and after getting
all orders we are inserting into DB storagenode and bucket bandwidth. Problem is with bucket bandwidth which is stored through cache which is often using context from SettlementWithWindowFinal stream to perform DB inserts and its doing this in separate goroutine. Because of that is possible that  SettlementWithWindowFinal is finished before flushing was finished and context is canceled while doing insert into DB

Change-Id: I3a72c86390e9aedc060f6b082bb059f1406231ee
2023-02-08 13:21:42 +00:00