Commit Graph

3148 Commits

Author SHA1 Message Date
Lizzy Thomson
7e2d98988b cmd/tools: remove migration tool code for generating missing salt
Remove generate-missing-project-salt migration tool code and related
tests. This migration has already been run and this code is no longer
needed.

Issue https://github.com/storj/storj-private/issues/163

Change-Id: I4e36dcd95a07c5305c597113a7fd08148e100ccc
2023-03-10 16:38:54 +00:00
Michal Niewrzal
0696256769 satellite/accounting: fix project limit cache usage
It looks that at some point we broke how project limits cache is used
and we were missing cache in most critical paths (upload/download).
This is fix for this issue.

I also adjusted cache methods naming.

Change-Id: Ic98372779a39365d0920fe3943f1f7a68b064173
2023-03-10 14:42:38 +00:00
Wilfred Asomani
b1d4763086 satellite/{web,payments}: show STORJ balance
This change adds a card to the billing overview page, which shows the
user's token balance from coinpayments.

Issue: https://github.com/storj/storj-private/issues/151

Change-Id: I11e295b48791b32b745cb7a11c5b4aad6b56618e
2023-03-10 13:01:15 +00:00
paul cannon
b98a09fa58 satellite/geoip: skip test on macOS
This is how we deal with the other tests that expect all 127.* addresses
to work locally.

Change-Id: Id62bb07a56044dfadcc87089f0f13684eeb07f3c
2023-03-10 03:08:17 +00:00
paul cannon
fd6ce6b9a5 scripts/tests: fix test-sim-rolling-upgrade.sh
This test involves a satellite with dev defaults (DistinctIP=no) being
upgraded past commit 2522ff09b6, which
means we need to run the dev-defaults-satellite-upgrade migration SQL
to avoid getting DistinctIP=yes behavior (which breaks the tests).

Change-Id: I29fb596d1ffa568dad635d98cfe9abacd3aaa48f
2023-03-09 23:35:36 +00:00
Michal Niewrzal
0c177ef91f satellite: cleanup orders dependencies
Only API peer needs access to order DB (and rollups cache) because it's
only place where we are creating orders for PUT and GET operations. For
other peers like auditor and repairer we can set noop implementation to
reduce number of dependencies needed for them.

Change-Id: Ic32d1879f0b97ffc4516f401898e31e95ae892e4
2023-03-09 13:34:21 +00:00
Márton Elek
ffaf15a3b0 satellite/overlay: remove unused mail service from overlay
It was surprising that `satellite auditor` complained about SMTP mail settings, even if it's not supposed to sending any mail.

Looks like we can remove the mail service dependency, as it's not a hard requirement for overlay.Service.

Change-Id: I29a52eeff3f967ddb2d74a09458dc0ee2f051bd7
2023-03-09 12:17:35 +00:00
Márton Elek
788f5fde01 satellite/peer: better error handling for SMTP parameters
I tried to configure a satellite service and got this error:

```
DEBUG   process/exec_conf.go:408        Unrecoverable error     {"error": "missing port in address"}
```

It took some time to realize that I forgot to set the SMTPServerAddress.

This patch makes it easier to detect similar problem (detailed error message), and makes SMTP parameters optional if no real mail sending is used (simulated or nomail)

Change-Id: I32535a7c8d6529e19e4d919806f42ba430d074a5
2023-03-09 11:29:07 +00:00
paul cannon
2522ff09b6 satellite/overlay: configurable meaning of last_net
Up to now, we have been implementing the DistinctIP preference with code
in two places:

 1. On check-in, the last_net is determined by taking the /24 or /64
    (in ResolveIPAndNetwork()) and we store it with the node record.
 2. On node selection, a preference parameter defines whether to return
    results that are distinct on last_net.

It can be observed that we have never yet had the need to switch from
DistinctIP to !DistinctIP, or from !DistinctIP to DistinctIP, on the
same satellite, and we will probably never need to do so in an automated
way. It can also be observed that this arrangement makes tests more
complicated, because we often have to arrange for test nodes to have IP
addresses in different /24 networks (a particular pain on macOS).

Those two considerations, plus some pending work on the repair framework
that will make repair take last_net into consideration, motivate this
change.

With this change, in the #2 place, we will _always_ return results that
are distinct on last_net. We implement the DistinctIP preference, then,
by making the #1 place (ResolveIPAndNetwork()) more flexible. When
DistinctIP is enabled, last_net will be calculated as it was before. But
when DistinctIP is _off_, last_net can be the same as address (IP and
port). That will effectively implement !DistinctIP because every
record will have a distinct last_net already.

As a side effect, this flexibility will allow us to change the rules
about last_net construction arbitrarily. We can do tests where last_net
is set to the source IP, or to a /30 prefix, or a /16 prefix, etc., and
be able to exercise the production logic without requiring a virtual
network bridge.

This change should be safe to make without any migration code, because
all known production satellite deployments use DistinctIP, and the
associated last_net values will not change for them. They will only
change for satellites with !DistinctIP, which are mostly test
deployments that can be recreated trivially. For those satellites which
are both permanent and !DistinctIP, node selection will suddenly start
acting as though DistinctIP is enabled, until the operator runs a single
SQL update "UPDATE nodes SET last_net = last_ip_port". That can be done
either before or after deploying software with this change.

I also assert that this will not hurt performance for production
deployments. It's true that adding the distinct requirement to node
selection makes things a little slower, but the distinct requirement is
already present for all production deployments, and they will see no
change.

Refs: https://github.com/storj/storj/issues/5391
Change-Id: I0e7e92498c3da768df5b4d5fb213dcd2d4862924
2023-03-09 02:20:12 +00:00
Michal Niewrzal
67ad792d1a satellite/rangedloop: migrate segments verification from segment loop
Segments loop have build-in sanity check to verify if number of segments
processed by loop is roughly fine. We want to have the same verification
for ranged loop.

https://github.com/storj/storj/issues/5544

Change-Id: Ia19edc0fb4aa8dc45993498a8e6a4eb5928485e9
2023-03-08 17:00:11 +00:00
Cameron
f4ea730e69 satellite/console: use custom session duration
If the session_minutes column in user_settings is not null, use this
value to determine the expiration when creating or refreshing a session.

github issue: https://github.com/storj/storj/issues/5487

Change-Id: I1f777f7ffcd4018b0c40f0b26c8a50de85a31909
2023-03-08 14:55:02 +00:00
Erik van Velzen
464ceb1c0e satellite/gc: improve comments
Change-Id: I9e71c9bee3447f78365ba1593e4a4ef55b28356f
2023-03-08 13:15:13 +00:00
Vitalii
a01d47b3ed satellite:{console, web}: remove old project dashboard
Removed old project dashboard along with a feature flag.

Change-Id: I9292b19caf101a84042d8c3006c27bc44db8d6b5
2023-03-07 13:34:59 +02:00
Vitalii
b6721633f1 satellite/payments: return public project ID for project charges
We should return public project id instead of regular one when quering project charges so that it is consistent on a client side.
This is a fix for an issue with displaying project name on a Billing screen.

Issue:
https://github.com/storj/storj/issues/5641

Change-Id: Ic0c544dbe7369aa88f3a7c97a2f5de5bc854e1fd
2023-03-06 22:50:21 +00:00
JT Olio
5b0cada4b3 repairer: monitor non-nil limit amount
Change-Id: I1a7b7a4a6716783449704cd8a7823090109a14de
2023-03-06 20:39:45 +00:00
Michal Niewrzal
bc8f8f62b5 satellite/orders: cleanup after altering primary key
We changed primary key for bucket_bandwidth_rollups table. Now we
need to do some cleanup in places like structs, sorting methods or SQL
queries.

Change-Id: Ida4f874f161356df193379a53507602e04db1668
2023-03-06 16:03:11 +00:00
Egon Elbre
63fa386b0a satellite/payments/stripecoinpayments: avoid mock cross-talk in tests
The tests were using global variables for keeping the mock state, which
was indexed by the satellite ID. However, the satellite ID-s are
deterministic and it's possible for two tests end up using the same
mocks.

Instead make the mock creation not depend on the satellite ID and
instead require it being configured via paymentsconfig.

This fixes TestAutoFreezeChore failure.

Change-Id: I531d3550a934fbb36cff2973be96fd43b7edc44a
2023-03-03 20:20:48 +02:00
Michal Niewrzal
d54ccfa92b satellite/satellitedb: change bucket_bandwidth_rollups primary key
In the past we set incorrect primary key for bucket_bandwidth_rollups
table and bucket name was first column of key instead of project id.
This change fixes this problem.

We will alter primary key manually for production satellites. That's
why migration code is checking if change was already applied.

https://github.com/storj/storj/issues/5332

Change-Id: I7dd555eb0c2f6fda1c90eaed08fa847b7f273bc7
2023-03-03 16:25:35 +00:00
Moby von Briesen
e9628c133d satellite/console: Add comments to generated api
Clarify that this functionality is not currently implemented within the
Satellite UI

Change-Id: I94096ab9b4d8f00195b55f5e875d76930de848e4
2023-03-03 09:29:11 -05:00
Michal Niewrzal
5a9577bb43 satellite/metabase/rangedloop: test loop boundaries
Fixes https://github.com/storj/storj/issues/5620

Change-Id: Ibf23e8866c1a8a8e78281c3d07e81ffc7db30fcf
2023-03-03 09:56:01 +00:00
Jeremy Wharton
3b37a23d7b satellite/console/consoleweb: add pricing package availability endpoint
An endpoint has been added that returns whether a pricing package is
available for a user to purchase. This will be used to conditionally
skip the pricing plan selection step of the onboarding tour.

Change-Id: I8c02a4e474e5f0f80778453b7daf674c8da64306
2023-03-02 23:51:09 +00:00
Cameron
fb471d7d8d satellite/console: delete password recovery token when password is changed
If a user changes password while a password recovery token exists,
delete the token so the reset password email doesn't work.

github issue: https://github.com/storj/storj-private/issues/149

Change-Id: Ibb0c13076e182c46501c1d7ac92d3fecc7d5ee58
2023-03-02 22:47:55 +00:00
Lizzy Thomson
54f0f47839 satellite/satellitedb: add passphrase_prompt column to user_settings
Add passphrase_prompt column to user_settings table to store a boolean
whether a user would like to be prompted to enter a passphrase when
entering a new project.

Issue https://github.com/storj/storj/issues/5613

Change-Id: I5d086df277aff458453343c7c38a379e7b1b8bf9
2023-03-01 14:53:18 -07:00
Michal Niewrzal
b46c0fb78f satellite/orders: don't cancel flushing bandwidth orders
Earlier we made a change to not cancel flushing orders when flushing
was triggered by orders endpoint method but we missed a case
where it can be also triggered (and canceled) by metainfo endpoints
method. This change moves ignoring context cancellation deeper.

Change-Id: Id43176f552efc3167345783f73aab885411ac247
2023-03-01 17:10:05 +00:00
Wilfred Asomani
faeea88265 satellite/{db,analytics,payments}: add chore for auto account freeze
This change adds a new chore that will check for failed invoices and
potentially freeze corresponding accounts.
It makes slight modifications to stripemock.go and invoices.go (adding
stripe CustomerID to the Invoice struct).

Issue: https://github.com/storj/storj-private/issues/140

Change-Id: I161f4037881222003bd231559c75f43360509894
2023-03-01 09:31:27 +00:00
Jeremy Wharton
31ec4fae9e web/satellite: add pricing plan selection to onboarding tour
A pricing plan selection step for users with a recognized partner has
been added to the beginning of the onboarding tour. Once visited, users
have the option of purchasing the pricing plan associated with their
partner or proceeding as a paid or free tier user.

Resolves storj-private#118
Resolves storj-private#126

Change-Id: I3b423194d96deaf87cf9807a766bf4d04fbcf86d
2023-02-28 20:49:59 -06:00
JT Olio
4362761fc7 satellite/audit: fix go1.19 dial timeouts and log more
Change-Id: Ide17c1b8e0ca8c86f305bea1b4ae553cc4cb60d0
2023-02-28 17:09:47 +00:00
Egon Elbre
f13d0f7df0 satellite/satellitedb/dbx: document oauth tables
Change-Id: Ide327699f4b7b35f46c9356d558221ece5bd77a9
2023-02-28 09:07:58 +00:00
Cameron
b24ce98944 satellite/admin: add Groups to config
add new config to the satellite admin: --admin.groups.limit-update.
This can be used as an alternate means of authentication if the request
is coming from the oauth proxy.

Change-Id: Ic2de13862e6414244b060c66a0f2bed72097cbad
2023-02-27 16:07:20 +00:00
Egon Elbre
1cef2334ff satellite/satellitedb/dbx: document billing
Change-Id: Ief90cc0a6bebaa8df9dd9255565eaa42dc0abc9e
2023-02-27 15:13:23 +00:00
Egon Elbre
4aa519a69f satellite/satellitedb/dbx: document node payments
Change-Id: Ie65ec960f1683f5735a2b5959fedbde326007d04
2023-02-27 14:13:35 +00:00
Egon Elbre
1f439f5ae9 satellite/satellitedb/dbx: document audit tables
Change-Id: Iae0f2d7eaddb30cf43a3a7cd678f1dcfb8256a45
2023-02-27 09:56:39 +00:00
Jeremy Wharton
cbbd5ab1ef satellite/payments/stripecoinpayments: undo price override removal
Commit fb59974 disabled usage price overrides because of a failing
test. This change reenables it while resolving the issue that caused
the test to fail.

The previous version of the test passed Gerrit verification and was
merged, but it failed for the primary Jenkins pipeline after merge.

This is due to a difference in how the Jenkins build runs Cockroach
and Postgres for each pipeline.

This commit rewrites the test to be safe for concurrent execution by
ensuring any mutable variables are defined within each test so that
shared state across tests is reduced.

Change-Id: Ia4566c9cd2d698afdb2caa4b7e2808b17e18de4e
2023-02-24 16:54:51 +00:00
paul cannon
20bcdeb8b1 satellite/repair: fix flaky test TestECREpairerGetOffline
It was possible to get into a situation where successfulPieces =
es.RequiredCount(), errorCount < minFailures, and inProgress == 0 (when
the succeeding gets all completed before the failures), whereupon the
last goroutine in the limiter would sit and wait forever for another
goroutine to finish.

This change corrects the handling of that situation.

As an aside, this is really pretty confusing code and we should think
about redoing the whole function.

Change-Id: Ifa3d3ad92bc755e563fd06b2aa01ef6147075a69
2023-02-24 09:05:21 -06:00
Egon Elbre
4a6e34bb2c satellite/satellitedb/dbx: document peer_identity table
Change-Id: I112ef00cb80638a8168dd46a531522b0785b537a
2023-02-24 14:06:46 +00:00
Egon Elbre
cf324c5b6e satellite/satellitedb/dbx: document node tables
Also, fix nodevents iota usage. Type is directly used in the database
hence we should avoid iota, because it's easy to mess-up the constant
values.

Change-Id: I32bf6d0502fbf3d18660dab5b1dd0b5627b444df
2023-02-24 14:05:55 +00:00
Egon Elbre
e4914e9b7a satellite/satellitedb/dbx: document accounting tables
Change-Id: I94478892a37623882d306a811e272f51865e48dd
2023-02-24 16:04:29 +02:00
Márton Elek
dfe3c87b87 satellite/metabase/rangedloop: save observer durations to eventkit
Change-Id: I2d027cd7f7421e2c38dcc1277d640d4183af4cec
2023-02-24 13:30:19 +00:00
Egon Elbre
4e70ef83fb satellite/satellitedb/dbx: split accounting.dbx by project and node
Change-Id: Ia3dee52abf98a41ea62d94af0a155f4f09661224
2023-02-24 09:08:20 +00:00
Cameron
8842985571 satellite/console/consoleweb: create purchase-package endpoint
Add new purchase-package endpoint to Server. The endpoint can be enabled
or disabled by a new config, --console.pricing-packages-enabled.
The purchase-package endpoint applies a coupon and adds and charges a
credit card if user's useragent is a partner with a configured package
plan.

github issue: https://github.com/storj/storj-private/issues/125

Change-Id: I0d6ccccd6874ddba360c45f338fd1c44f95e135a
2023-02-23 15:42:36 -05:00
Egon Elbre
1a2ad602d7 satellite/satellitedb/dbx: document gracefulexit tables
Change-Id: Ia76b67fa85e797c1d75438ae882bdfe83a6ee057
2023-02-23 19:14:51 +02:00
Jeremy Wharton
fb5997484e satellite/payments/stripecoinpayments: remove usage price overriding
Project usage price overriding has been removed because it produces
incorrect results when tested. It should not be re-implemented until
the issues it causes are resolved.

Change-Id: Ic92eff374c9af4fea3bf32782a72303a7978b055
2023-02-23 13:36:59 +00:00
Ivan Fraixedes
16d3fcde70 satellite/admin/ui/lib: Fix bug in API
The update user API function was defining a parameter that the API
calling function doesn't receive, so all the values were shifted by
1 after this extra parameter and the calling function was sending these
  values to the next one updating the fields that the user was not
  expecting.

  The API server doesn't allow to update this removed parameter, so it
  was likely a bug caused by copy-pasta.

Change-Id: I52a2b66abc1f6e5cb0070dc6fd230c30d1133dcf
2023-02-23 10:59:18 +00:00
Jeremy Wharton
209b669e84 satellite/console/consoleweb: create and serve frontend config
A new struct containing configuration values for the satellite frontend
is populated and served to the client. In the future, this struct will
be used to configure the frontend rather than using Go templates and
meta tags to do so.

References #5494

Change-Id: Ie2c8a140d53b20fab159d34f2e3767e5dbd43bdb
2023-02-22 23:24:35 -06:00
Michal Niewrzal
16b7901fde satellite/metabase: add piece size calculation to segment
This code is essentially replacement for eestream.CalcPieceSize. To call
eestream.CalcPieceSize we need eestream.RedundancyStrategy which is not
trivial to get as it requires infectious.FEC. For example infectious.FEC
creation is visible on GE loop observer CPU profile because we were
doing this for each segment in DB.

New method was added to storj.Redundancy and here we are just wiring it
with metabase Segment.

BenchmarkSegmentPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize-8         	    5822	    189189 ns/op	    9776 B/op	       8 allocs/op
BenchmarkSegmentPieceSize/segment.PieceSize
BenchmarkSegmentPieceSize/segment.PieceSize-8              	94721329	        11.49 ns/op	       0 B/op	       0 allocs/op

Change-Id: I5a8b4237aedd1424c54ed0af448061a236b00295
2023-02-22 11:04:02 +00:00
Cameron
19d4fcc530 satellite/console: return card from AddCreditCard
Change-Id: I1a2667d26e2864606cfafc21a146a6be02798715
2023-02-21 20:48:15 +00:00
Cameron
e2cf486fcb satellite/console: add Purchase method to console.Payments
Purchase collects a payment from user using the specified price,
description, and payment method. This is implemented at the lower layers
using the payments.Invoices interface. There are also additions to
stripecoinpayments stripemock to simulate errors returned from stripe
invoices New and Pay. If an invoice exists with the same description, if
it is a draft, it is deleted and a new one is created and paid. If it is
open, pay it and don't create a new one.

Change-Id: Ic3147434bc44a0777ecbedb3a4ed4c768eb02ea3
2023-02-21 20:10:42 +00:00
Jeremy Wharton
3fa31c2077 satellite/console/consoleweb: remove trailing slash from URLs
This change removes the trailing slash from the account activation and
password recovery URLs, making them consistent with the rest. The URLs'
previous forms are still supported, however, in order to not invalidate
emails containing them.

Resolves storj/customer-issues#491

Change-Id: Ie774a87698d8e9edd1836611968fc3911c6cc56f
2023-02-21 19:15:36 +00:00
Cameron
38f94c1c29 satellite/reputation: if node is DQd or exited skip applying audit
If a node is already disqualified or exited, don't bother going through
the audit reputation code. If there's an additional reputation update to
be made based on the automatic offline audit these nodes would get, the
overlay DB rejects it anyway.

Change-Id: I48ba151aa640b32f87e1285521be465d1f692a78
2023-02-21 18:31:21 +00:00
Cameron
26f80f8e26 satellite/payments: add Invoices Delete
Add Invoices method Delete to delete draft invoices.

Change-Id: I221c6208df98ab11e648d25b94c52a411bf65b9a
2023-02-21 17:54:42 +00:00
Cameron
78e68cdca2 satellite/payments: add constants for invoice status
Change-Id: Iaf94d57d99caff3d435a44c56bd47c7b715ff4c0
2023-02-21 17:04:23 +00:00
JT Olio
f905878ff9 go.mod: bump storj.io/uplink
this change brings in:

 * private/piecestore: disable noise
 * uplink: fix noise version report
 * private/piecestore: fix noise
 * private/metaclient: honor context cancellation on retries
 * private/metaclient: fix multi-passphrase ListObjects paging
 * private/metaclient: fix broken ListUploads method

it also adds:

 * private/storage/streams/batchaggregator: introduce Aggregator
 * private/storage/streams/streambatcher: introduce Batcher
 * private/storage/streams: pieceupload package
 * private/storage/streams: segmenttracker package

but these commits provably change no existing behavior, since they
only add unused, new packages and types.

Change-Id: Id9efc65117b7e1ddcb3ffc8cc15d3e665ed11d7d
2023-02-21 16:09:19 +00:00
Michal Niewrzal
06f4aede9b satellite/audit: use NodeAlias instead pure NodeID
This is first attempt to use AliasPieces inastead Pieces with
segments/range loop. So far we were always using Pieces
which are always converted from AliasPieces for easy use.
Side effect is that using NodeID with loop observers is heavy
e.g. we are using maps which behaves slower with NodeIDs.

We are starting with audit observer because it's easy to change
it as in feact it doesn't need access to real NodeID at all. We just
need to reference node in some way and this way is NodeAlias.

Results of BenchmarkRemoteSegment:
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    1.79µs ± 6%    0.03µs ± 4%  -98.29%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8     0.00B          0.00B          ~     (all equal)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      0.00           0.00          ~     (all equal)

Change-Id: Ib7fc87e568a4d3a9af27b5e3b644ea68ab6db7aa
2023-02-21 15:31:59 +00:00
Egon Elbre
0cdef95d55 all: fix math/rand deprecations
Change-Id: I4b966375697c0d409ce24cc7604f806973f8f22a
2023-02-17 15:05:54 +02:00
Michal Niewrzal
aba2f14595 satellite/metabase/rangedloop: few additions for monitoring
Additional elements added:
* monkit metric for observers methods like Start/Fork/Join/Finish to
be able to check how much time those methods are taking
* few more logs e.g. entries with processed range
* segmentsProcessed metric to be able to check loop progress

Change-Id: I65dd51f7f5c4bdbb4014fbf04e5b6b10bdb035ec
2023-02-17 08:46:00 +00:00
Lizzy Thomson
155b927c5e satellite/console: add notifications for free account limits
Add notifications for free account limits for segment usage
and update to follow the figma designs.

Issue: https://github.com/storj/storj/issues/5482

Change-Id: I8a2fe38d609d53e09bf5074484cedc343223bffd
2023-02-16 13:31:01 -07:00
Cameron
d48f745668 satellite/payments: trigger errors in stripemock payment methods
stripecoinpayments.mockStripeClient implements the
stripecoinpayments.StripeClient interface which is used to interact with
stripe's APIs in production. We use it in tests to simulate API calls to
stripe. This change introduces new stripecoinpayments package level
constants: TestPaymentMethodsNewFailure and
TestPaymentMethodsAttachFailure. These can be passed as part of the
arguments their respective stripe mock methods to signal to the method
that it should return an error. This is useful so that we can test how
layers on top of the StripeClient interface handle errors.

Change-Id: Ib64c08ba1e91f31e755b66a1ad563c3b6e77f6f0
2023-02-16 10:00:20 -05:00
Cameron
8d9b61c78e satellite/console: ApplyCoupon and ApplyFreeTierCoupon
Add new console service payments methods ApplyCoupon and
ApplyFreeTierCoupon. ApplyCoupon applies a coupon to an account based on
the coupon ID passed to it. ApplyFreeTierCoupon applies the satellite
configured free tier coupon to the account.

Change-Id: Ic221092278553a79207ac2a0c9229c374d76c881
2023-02-16 10:00:20 -05:00
paul cannon
6bb6479690 satellite/repair: fix flakiness in tests
Several tests using `(*ECRepairer).Get()` have begun to exhibit flaky
results. The tests are expecting to see failures in certain cases, but
the failures are not present. It appears that the cause of this is that,
sometimes, the fastest good nodes are able to satisfy the repairer
(providing RequiredCount pieces) before the repairer is able to identify
the problem scenario we have laid out.

In this commit, we add an argument to `(*ECRepairer).Get()` which
specifies how many failure results are expected. In normal/production
conditions, this parameter will be 0, meaning Get need not wait for
any errors and should only report those that arrived while waiting for
RequiredCount pieces (the existing behavior). But in these tests, we can
request that Get() wait for enough results to see the errors we are
expecting.

Refs: https://github.com/storj/storj/issues/5593
Change-Id: I2920edb6b5a344491786aab794d1be6372c07cf8
2023-02-16 07:33:47 +00:00
paul cannon
ab2e793555 satellite/audit: test delay before Reverify
We are supposed to wait for some amount of time after a timed-out audit
before retrying the audit on the contained node. We are also supposed to
wait for some amount of time before subsequent retries, if they are
necessary. The test added here tries to assure that those delays happen,
as far as it is possible to assure that a delay will happen in computer
code.

The previous behavior of the system was, in fact, to carry out
Reverifies as soon as a worker could retrieve the job from the
reverification queue. That's not a very major problem, as subsequent
retries do have a delay and the node does get several retries. Still, it
was not ideal, and this test exposed that mismatch with expectations, so
this commit includes a minor change to effect that pause between verify
and the first reverify.

Refs: https://github.com/storj/storj/issues/5499
Change-Id: I83bb79c166a458ba59a2db2d17c85eca43ca90f0
2023-02-15 23:16:23 +00:00
JT Olio
77bf88e916 satellite/overlay: check node difficulty before entering database
closes https://github.com/storj/storj/issues/5568

Change-Id: Id413637c2678e7a7cf8dbf414e082c687c8e8a39
2023-02-15 17:46:25 +00:00
Michal Niewrzal
0a525292e4 satellite/metainfo: fix duplicates while listing pending objects
While working on fixing listing for committed objects we didn't fix
the same case for pending objects. For case were we have many
pending objects under different locations we need to set cursor
version to highest value to avoid duplicates.

For case where we have many pending objects under the same location
we will need to make a separate fix.

https://github.com/storj/storj/issues/5570

Change-Id: Id5c8eb728868e8e1177fdbcf65a493142be4eaf0
2023-02-15 16:52:53 +00:00
Wilfred Asomani
9138e84fb1 satellite/{web,console}: feature flag for all projects dashboard
Adds a feature flag for the new all projects dashboard. It defaults to false.

Issue: https://github.com/storj/storj/issues/5514

Change-Id: I160904eccae7d30e05b734e69600725702b16aca
2023-02-15 15:53:31 +00:00
Lizzy Thomson
1b22994631 satellite/admin: update user's limits and project limits from admin api
Create an endpoint in the Admin Api to be able to update a user’s limits
for all existing and new projects. Also added a GET endpoint to return
user's limits.

Fixes: https://github.com/storj/storj/issues/5395

Change-Id: I2c093dc08ebf79a4318391e63a37da4d2b403547
2023-02-15 11:38:57 +00:00
Michal Niewrzal
41bcc6bb62 satellite/metainfo: fix duplicates while listing committed objects
We have an issue where object can appear in two different listing pages.
It's because protobuf listing cursor doesn't have version included and
now we can have internally versions higher than 1. On satellite side
version 1 was always used as a default cursor version.

As a workaround for existing implementation of libuplink library we will
use always maximum version for listing cursor on satellite side.

Fixing protobuf and libuplink implementation will happen later.

https://github.com/storj/storj/issues/5570

Change-Id: Ibd27b174556c9d8b8bd60fab8cff7862fd11e994
2023-02-14 14:47:27 +01:00
Wilfred Asomani
4a67b57103 satellite/{console,web}: make bucket endpoints support publicID
This change adds support for project public id to the bucket-names and
usage-report endpoints.

Issue: https://github.com/storj/storj/issues/5578

Change-Id: I2429ebebe52dfc8217fc40f4691e7bc473b805fb
2023-02-13 23:00:03 +00:00
Michal Niewrzal
94d341bcf3 satellite: use ranged loop with GC-GF peer
Peer for generating bloom filters will be able to use ranged loop.

As an addition some cleanup were made:
* remove unused parts of GC BF peer (identity, version control)
* added missing Close method for ranged loop service
* some additional tests added

https://github.com/storj/storj/issues/5545

Change-Id: I9a3d85f5fffd2ebc7f2bf7ed024220117ab2be29
2023-02-13 18:32:21 +00:00
Márton Elek
8f8e97de23 satellite/metainfo: support desired node number for download object/segment
This modification introduce support of the new "desired node" field of download segment/object.

This can be used to request more nodes than the suggested minimum. It can be used to achieve better performance in exchange of using more bandwidth. (more parallel downloads).

Change-Id: Ia167d6979e6d70a597c85070a4ccd1c3a573e406
2023-02-13 13:57:48 +00:00
Márton Elek
ca6e3a9e88 satellite/orders: create mock based unit test
Most of our (~integration) tests based on testplanet runner.

However running testplanet for each test make the testing process slow.
It seems to be better to use real unit tests (without db dependency) when it's possible.

This patch makes small modification to make it possible to test orders.Service with real unit test.

As the existing unit test of `service.go` is isolated with `_test` package name, it's moved to an `_integration_test.go` file to make place for the unit test.

Change-Id: Ia69f26a34e2c48d230d8d36c2040dd02a60455a6
2023-02-13 13:24:30 +00:00
Cameron
ce23ce0dc9 satellite/nodeevents: minor changes to dedupe node IDs
`idsMap` renamed to `seen`. The map key is now storj.NodeID rather than
string.

Change-Id: Ia120d21c94eeecd1b855910bf27123dcd1dc7f97
2023-02-10 16:20:10 +00:00
Egon Elbre
491fa78840 satellite/satellitedb/dbx: add comments for node_reputation.dbx
Change-Id: If9c2bb605a9c562c114a59c3f3cec0b01b765689
2023-02-10 17:10:34 +02:00
Egon Elbre
a8caea8b9c satellite/satellitedb: remove gob migration exceptions
Change-Id: I2958e2145edaa501d510314f9bfd0e5280ea82ec
2023-02-10 15:05:18 +02:00
paul cannon
362d73f783 satellite/audit: add tests for concurrent audits
This commit introduces tests that perform multiple concurrent audits
against the same storage node, to make sure that doing so does not
create incorrect outcomes.

Refs: https://github.com/storj/storj/issues/5495
Change-Id: Iaae49e042306bfa59bdf04c1a1540667488e51e5
2023-02-09 19:58:50 +00:00
Moby von Briesen
4cb825a6ea satellite/console/consoleweb: Add user ID rate limiting to some endpoints
Add rate-limiting based on UserID to add credit card and change password
endpoints.

Change-Id: I5d493aeeefc03729abb66266b5e3fd9715d592ad
2023-02-09 18:50:13 +00:00
Egon Elbre
b14019b8c5 satellite/{metabase/rangedloop,metainfo/piecedeletion}: fix flaky tests
TestLoopContinuesAfterObserverError was failing due to system
granularity measuring the duration as 0.

TestDialer_DialTimeout was failing due to connection failure came with a
delay and wasn't being handled.

Change-Id: I4638c86f5d021a86c3d3529fab13cf3608f35c40
2023-02-09 16:07:00 +02:00
Kaloyan Raev
b1d4a159a6 satellite/metainfo: ListUploads returns incorrect UploadID
ListUploads returns incorrect UploadID if Expires was set in
BeginUpload. DB is truncating expiration date to microseconds precision
so we need to do this also in code.

Change-Id: Iee0cf45cb705342f6bb9a2f745acca91cce6ff52
2023-02-09 10:33:08 +00:00
Vitalii
65e3cfb9c6 satellite/{web, console}: feature flag for new access grant flow
Added a feature flag for new create access grant flow.
Also added some initial setup.

Change-Id: I7f738181c8a83f5a724f9e562427445cae146b6f
2023-02-09 00:35:35 +00:00
Cameron
38275bd710 satellite/nodeevents: deduplicate node IDs
Remove dupulicate node IDs when sending notifications.

Change-Id: I76c642e342081f8fdf8248443de8383b9e88eed2
2023-02-08 23:14:18 +00:00
Lizzy Thomson
f733cb255b satellite/satellitedb: drop unused last_verification_reminder column
(updated to migration step 225) remove users table column
last_verification_reminder and update migrate_test.go

Issue: https://github.com/storj/storj/issues/4560

Change-Id: Icbd8b70a8243b75a11c481a3b87df0ebe9402c5f
2023-02-08 16:23:09 +00:00
Egon Elbre
dd3d051170 satellite/satellitedb/dbx: add docs for user.dbx
Change-Id: I346ceaedf6963169bb9d6d1435454016239d4cb1
2023-02-08 17:30:29 +02:00
Egon Elbre
a1bd36bb64 satellite/satellitedb/dbx: add docs for repair.dbx
Change-Id: Icffcdebb7dcc1d10eb8c97212acc6ae309b2f92c
2023-02-08 17:30:27 +02:00
Michal Niewrzal
252c437b0e satellite/orders: ignore context canceled when updating bucket bandwidth
Orders from storage nodes are received by SettlementWithWindowFinal method. There is a stream which receives all orders and after getting
all orders we are inserting into DB storagenode and bucket bandwidth. Problem is with bucket bandwidth which is stored through cache which is often using context from SettlementWithWindowFinal stream to perform DB inserts and its doing this in separate goroutine. Because of that is possible that  SettlementWithWindowFinal is finished before flushing was finished and context is canceled while doing insert into DB

Change-Id: I3a72c86390e9aedc060f6b082bb059f1406231ee
2023-02-08 13:21:42 +00:00
Wilfred Asomani
8ec3bc5803 web/satellite: move banners into page content
The banners on the web satellite previously hovered over the page
and overlapped other content as a result. This change moves them
into the page content so it overlaps nothing and is scrollable off-screen
This change also makes the upgrade banner only show on the dashboard and
only if the user joined more than seven days ago.

Issue: https://github.com/storj/storj/issues/5525

Change-Id: I7278c31201f09d3515d907b833622b04c6de8557
2023-02-08 09:22:58 +00:00
Jeremy Wharton
49916b02f0 satellite/{console,satellitedb}: migration, methods for user settings
This change adds a database migration for a new table that stores
configurations for a user, the first of which is the session duration.
Database methods are implemented to interact with this table.

Resolves #5472

Change-Id: I01049265f385ea5de65907da1bc3bcf426d3c577
2023-02-07 21:48:29 +00:00
JT Olio
522aed083d private/server,satellite/contact,misc: use new storj/common noise helpers
this change uses the new storj/common noise helpers, which:
 * add a security fix (require an expected node id for validating
   noise key attestations)
 * stops doing an unnecessary order signature validation (it's
   already been done inside of PutPiece)
 * removes some duplicate code

Change-Id: I5e67a08ff216cd9c5b0b82e40b4d9de664b6b0fc
2023-02-07 09:53:45 -05:00
Egon Elbre
dcebcf770d satellite/metabase: add database comments
Change-Id: I78e19fb9301044d076e3eabf7e92fa6b715ddf03
2023-02-07 11:42:21 +02:00
paul cannon
d6f8be1ec6 satellite/audit: add ContainmentSyncChore
We will be needing an infrequent chore to check which nodes are in the
reverify queue and synchronize that set with the 'contained' field in
the nodes db, since it is easily possible for them to get out of sync.
(We can't require that the reverification queue table be in the same
database as the nodes table, so maintaining consistency with SQL
transactions is out. Plus, even if they were in the same database, using
such SQL transactions to maintain consistency would be slow and
unwieldy.)

This commit adds the actual chore.

Refs: https://github.com/storj/storj/issues/5431
Change-Id: Id78b40bf69fae1ac39010e3b553315db8a1472bd
2023-02-07 01:18:49 +00:00
Egon Elbre
3146ad7f2e satellite/satellitedb: cleanup testing access
Previously we were exposing the testing facilities via interface casting
the necessary parts, however, when things are not part of the main
satellite.DB interface they need to be manually propagated. Rather than
relying on using hidden methods lets expose things as long as they don't
create a direct dependency to the database driver.

Change-Id: I2eb7d8b60f4b64de1320c2d32581f7be267c0f57
2023-02-06 14:36:11 +02:00
Wilfred Asomani
e8cd096eec satellite/{console,web}: make grapql mutations support publicId
This updates project related graphql mutations and queries to support
project publicId while maintaining support for project ID. The frontend
is updated to use only publicId when using these mutations/queries.

Issues:
https://github.com/storj/storj/issues/5409
https://github.com/storj/storj/issues/5413

Change-Id: Ib6241db157de3b37c86a4a98c9f682bf4a047b62
2023-02-06 11:57:44 +00:00
paul cannon
0e2fef977f satellite/overlay: add SetAllContainedNodes method to overlay.DB
We will be needing an infrequent chore to check which nodes are in the
reverify queue and synchronize that set with the 'contained' field in
the nodes db, since it is easily possible for them to get out of sync.
(We can't require that the reverification queue table be in the same
database as the nodes table, so maintaining consistency with SQL
transactions is out. Plus, even if they were in the same database, using
such SQL transactions to maintain consistency would be slow and
unwieldy.)

This commit adds a method to the overlay allowing the caller to set the
contained status of all nodes in the nodes table at once. This is valid
because our definition of "contained" now depends solely on whether a
node appears at least once in the reverification queue. Only rows whose
contained field does not match the expectation will be updated; the
contained timestamp will not be updated for a node which is supposed to
be contained and was already contained.

Change-Id: I8cabe56ad897b6027e11aa5b17175295391aa3ac
2023-02-06 10:18:54 +00:00
Moby von Briesen
395426977f satellite/reputation: Alter logic for hasReputationChanged
This change modifies "reputation change trigger" logic for dq and
suspension in order to:
* count only nil -> not-nil and not-nil -> nil as "reputation changes"
* do not count the timestamp updating (not-nil -> not-nil) as a
  reputation change

Change-Id: Ie57c97e80e5f3b368e2f2dc004e680a2f9416d1c
2023-02-03 21:17:57 -05:00
Jeremy Wharton
add3034b43 satellite/payments/stripecoinpayments: forbid replacing partner coupons
Users with a partner package plan should be unable to replace their
plan's coupon. This change enforces this behavior by rejecting coupon
application attempts from users that meet this criteria.

Change-Id: I6383d19f2c7fbd9e1a2826473b2f867ea8a8ea3e
2023-02-03 17:15:01 +00:00
Egon Elbre
873a202530 mod: bump storj.io/common
This bumps common, such that things build with Go 1.20.

Also, adds `go vet` checks for testsuite/storjscan and testsuite/ui.

The latest golang.org/x/bcrypt has a check that the new password is less
than 72 bytes, because bcrypt silently discarded them. This means our
own password validation has the same limitation. Old passwords should
still work fine.

Change-Id: Ibb8735b15eeb91460145906b81ae4e365e9ac418
2023-02-03 16:49:41 +02:00
Michal Niewrzal
36cbf4f0b8 private/testplanet: enable full table scan detection
Testplanet tests will print into logs (WARN) if full table scan will
be detected. Test won't be failed automatically. That's because
currently we have multiple queries which are doing full table scan and it's not trivial to change.

https://github.com/storj/storj/issues/5471

Change-Id: Ia2fcbfb9102424d58f95e00071329454a8c1066e
2023-02-03 12:11:18 +00:00
Michal Niewrzal
828654f98b gc: remove segment loop dependency
While splitting GC into two parts, bloom filter generation and bloom
filter sending, we missed to remove segments loop dependency from boom
filter sending part.

Change-Id: I48f213b0c0f8583e24fbb2bdcbfced518abe74cb
2023-02-03 10:31:16 +00:00
Michal Niewrzal
0b519fbd05 satellite/satellitedb: create index for interval_day for project_bandwidth_daily_rollup
To improve deletion of old entries in project_bandwidth_daily_rollup
we need index on `interval_day` column which is used to find those old
entries.

As an addition we are changing interval how often deletion is executed
from 7 to 1 day. We would like to have smaller portion of data to
delete.

Fixes https://github.com/storj/storj/issues/5465

Change-Id: Ie18ebe859887b93d6e4e6065a61fb9214c7ad27a
2023-02-03 10:02:58 +00:00
JT Olio
686faeedbd satellite/overlay: return noise info with selected nodes
we have two more fields in the database (noise_proto and
noise_public_key) that now need to go into pb.NodeAddress when
returning AddressedOrderLimits.

the only real complication is making sure type conversions between
database types and NodeURLs and so on don't lose this new
pb.NodeAddress field (NoiseInfo). otherwise this is a relatively
straightforward commit

Change-Id: I45b59d7b2d3ae21c2e6eb95497f07cd388d454b3
2023-02-02 15:46:27 +00:00
Cameron
de44c6ab58 satellite/admin: update README
add descriptions for freeze and unfreeze user endpoints and regenerate
table of contents. This additionally added the oauth endpoints and one project limits POST parameter to the ToC

Change-Id: I05025f1f3a11c3775a4f59b01569ccb419d72858
2023-02-02 14:40:24 +00:00
Michal Niewrzal
c1fffe881a satellite/satellitedb/satellitedbtest: enable full table scan detection
Satellite DB tests will print into logs (WARN) if full table scan will
be detected. Test won't be failed automatically. That's because currently
we have multiple queries which are doing full table scan and it's not
trivial to change.

We may change that behavior when we will figure out how to skip
specific query from detection or we will fix all problematic queries.

https://github.com/storj/storj/issues/5471

Change-Id: Icafe782257a0d353e8bcdf6fa8a19c20b1091a0b
2023-02-02 12:14:05 +00:00
Michal Niewrzal
4e0c7b2d90 satellite/metabase/metabasetest: fix race while running tests
Change-Id: I4fdb328443617bc9710ee6b9168b31870fe336d9
2023-02-02 10:35:28 +00:00