Commit Graph

3206 Commits

Author SHA1 Message Date
Jeremy Wharton
34b2a369f7 satellite/payments/stripe: apply egress discount to project charges
The project charge information used by the satellite frontend to
compute project cost estimates has been updated to account for
configured egress discounts.

Resolves storj/storj-private#215

Change-Id: Ic90b015d65f5bea104ac96fb0cea545b3f9f1f8f
2023-04-13 17:34:50 +00:00
JT Olio
16ccffb6f7 satellite/metainfo: test to confirm that retried pieces can't be submitted with originals
closes https://github.com/storj/storj/issues/5709

Change-Id: I2030825d85987d97ae8c10f02ab1f70086cbcb97
2023-04-13 14:48:32 +00:00
Michal Niewrzal
a21afeddd1 satellite/payments/stripe: avoid full table scan while listing
Query to list (with pages) stripe customers were doing full table scan
because Offset clause was used. This refactoring changed listing to
use cursor instead Offset.

Change-Id: I14688e6c533bc932ba0d209a061562f080b4cf54
2023-04-13 12:36:31 +02:00
Jeremy Wharton
c92fdba972 satellite/payments/stripe: apply egress discount to invoice items
Invoicing has been modified to account for an egress discount ratio,
which specifies the fraction of the amount of data stored that egress
usage should be discounted.

The egress discount ratio measures discounted egress units per storage
unit-months, so a ratio of 0.5 with 2 MB-months of storage would
discount 1 MB of egress.

Resolves storj/storj-private#224

Change-Id: I43d7d6719391c303712c082709aef77249c65f62
2023-04-12 15:28:00 +00:00
Jeremy Wharton
6e9dae06d5 satellite/payments: add egress discount ratio to price override config
This change allows for specifying the ratio of free egress per unit
of storage within a price override configuration.

References storj/storj-private#215
References storj/storj-private#224

Change-Id: Ib1c79f77ec8bb11dd5b2f9dace13800b0b3ce942
2023-04-12 14:52:19 +00:00
Jeremy Wharton
59eab37b95 satellite/console/consoleweb: remove bucket usage report page
No component has referenced this page since 9dab10e and we do not
anticipate this changing, so this page can be safely removed.

Resolves #5768

Change-Id: I57acb5e4d0977d74df46aaf67606a19ec0f10bcf
2023-04-11 18:17:03 +00:00
Egon Elbre
8fba740332 storagenode/blobstore/testblobs: don't error checks in BadDB
We automatically start a chore to check whether the blobstore is
writeable and readable, however, we don't want to fail the tests due to
that reason. Usually we want to test some other failure.

There probably should be some nicer way to achieve this, but this is an
easier fix.

Change-Id: I77ada75329f88d3ea52edd2022e811e337c5255a
2023-04-11 16:14:31 +03:00
Andrew Harding
8c5924d6ea satellite/payments/billing: add bonus transactions
- only applies to storjscan transactions
- applies a 10% bonus by default
- bonus transactions have a distinct source "type" to allow for
  filtering on the frontend

Fixes: https://github.com/storj/storj/issues/5702

Change-Id: I32d65f776c58bcb41227ff5bc77a8e4cb62a9add
2023-04-10 16:26:56 +00:00
Wilfred Asomani
5b65e10563 satellite/console: enable session timeout configuration
This another account endpoint; patch /auth/account/settings. to handle
changing a user's settings, including their session timeout config.

Issue: https://github.com/storj/storj/issues/5560

Change-Id: I747b4e919cf7cef7c867ac9d282837ef51bed67e
2023-04-07 21:05:36 +00:00
paul cannon
915f3952af satellite/repair: repair pieces on the same last_net
We avoid putting more than one piece of a segment on the same /24
network (or /64 for ipv6). However, it is possible for multiple pieces
of the same segment to move to the same network over time. Nodes can
change addresses, or segments could be uploaded with dev settings, etc.
We will call such pieces "clumped", as they are clumped into the same
net, and are much more likely to be lost or preserved together.

This change teaches the repair checker to recognize segments which have
clumped pieces, and put them in the repair queue. It also teaches the
repair worker to repair such segments (treating clumped pieces as
"retrievable but unhealthy"; i.e., they will be replaced on new nodes if
possible).

Refs: https://github.com/storj/storj/issues/5391
Change-Id: Iaa9e339fee8f80f4ad39895438e9f18606338908
2023-04-06 17:34:25 +00:00
Michal Niewrzal
8a50a3baa3 satellite/payments: rename 'stripecoinpayments' package to 'stripe'
Automatic rename. May require some more cleanups later.

Change-Id: I18220a4278056d25c41fb137832bb81f2b876ac1
2023-04-06 16:51:43 +00:00
Michal Niewrzal
31f5e2cb65 satellite/metainfo: limit uploads to the same location
We would like to have ability to limit burst uploads to the single
object (the same location). This change we are limiting such upload to
one per second.

Change-Id: Ib9351df1017cbc07d7fc2f846c2dbdbfcd3a360c
2023-04-06 15:49:11 +00:00
Egon Elbre
48256c91b5 storage: move errors to better locations
Change-Id: Ia44570949a8f6bb50220dc838c5b6aa21e851a4d
2023-04-06 17:26:29 +03:00
Egon Elbre
f5020de57c storagenode/blobstore: move blob store logic
The blobstore implementation is entirely related to storagenode, so the
rightful place is together with the storagenode implementation.

Fixes https://github.com/storj/storj/issues/5754

Change-Id: Ie6637b0262cf37af6c3e558556c7604d9dc3613d
2023-04-05 18:06:20 +00:00
Cameron
f550f7e159 web/satellite: sort api keys
add ability to sort api keys on the access management page by ascending
or descending name or creation date. Additionally, edit api keys query
when ordering by name to order by lower(name) so names starting with
capital letters are not treated differently from lower case when
ordering.

Change-Id: I81dbb87587a24fb7097313f76bad116b1f20d306
2023-04-05 11:55:17 -04:00
Michal Niewrzal
c4c391e154 satellite/accounting/live: replace address parsing with redis util
With this change we are replacing parsing code with existing go-redis
util.

We also switch redis client to version 9.

Change-Id: Ie4a651e3ae6960e68958c690873925d319b70e10
2023-04-05 13:20:11 +00:00
paul cannon
9e6955cc17 satellite/repair: fix flaky TestFailedDataRepair and friends
The following tests should be made less flaky by this change:

- TestFailedDataRepair
- TestOfflineNodeDataRepair
- TestUnknownErrorDataRepair
- TestMissingPieceDataRepair_Succeed
- TestMissingPieceDataRepair
- TestCorruptDataRepair_Succeed
- TestCorruptDataRepair_Failed

This follows on to a change in commit 6bb64796. Nearly all tests in the
repair suite used to rely on events happening in a certain order. After
some of our performance work, those things no longer happen in that
expected order every time. This caused much flakiness.

The fix in 6bb64796 was sufficient for the tests operating directly on
an `*ECRepairer` instance, but not for the tests that make use of the
repairer by way of the repair queue and the repair worker. These tests
needed a different way to indicate the number of expected failures. This
change provides that different way.

Refs: https://github.com/storj/storj/issues/5736
Refs: https://github.com/storj/storj/issues/5718
Refs: https://github.com/storj/storj/issues/5715
Refs: https://github.com/storj/storj/issues/5609
Change-Id: Iddcf5be3a3ace7ad35fddb513ab53dd3f2f0eb0e
2023-04-04 18:08:52 +00:00
Jeremy Wharton
f2ae202024 satellite/payments,web/satellite: separate UI cost estimates by partner
Components related to project usage costs have been updated to show
different estimations for each partner, and the satellite has been
updated to send the client the information it needs to do this.

Previously, project costs in the satellite frontend were estimated
using only the price model corresponding to the partner that the user
registered with. This caused users who had a project containing
differently-attributed buckets to see an incorrect price estimation.

Resolves storj/storj-private#186

Change-Id: I2531643bc49f24fcb2e5f87e528b552285b6ff20
2023-04-04 15:56:16 +00:00
Michal Niewrzal
4c05293d8b satellite: set names for lrucache metrics
Fixes https://github.com/storj/storj/issues/5685

Change-Id: I9c19b7644b0cd3297448982b920c97abcdce9119
2023-04-04 11:55:20 +00:00
Michal Niewrzal
4db3d33d3e satellite/accounting: check bucket existance with GetProjectObjectsSegments
Check if project have any buckets before trying to get stats from
bucket_storage_tallies table.

Change-Id: If87159e29236a1816c3fb99c2d889ff735737100
2023-04-03 20:06:11 +00:00
Jeff Wendling
7b06575f6f satellite/meta{base,info}: reduce db round trips for download
This combines the ListStreamPositions and GetSegmentByPosition
calls with a ListSegments call that now knows how to return
only the segments within a Range, just like ListStreamPositions.

It would theoretically be possible to also include the
GetObjectLastCommitted call by having it do one of three
queries based on the incoming request range, but that would
mean duplicating the data for the object in every single
row that is returned for each segment in the range.

One gross thing that ListSegments has to do now is update the
first segment returned with the information from any ancestor
segment because GetSegmentByPosition used to do that. It only
updates the first segment so that it doesn't do O(N) database
queries. It seems difficult to have it do a single query to
update all of the segments at once. I'm not certain this change
should be merged on this basis alone.

This change has made me think a couple of things should happen:

1. Server side copy with ancestor segments strikes again
   making the code less clear and potentially more buggy
   or inefficient for a rare case (empirically <0.1%)

2. The download code requests individual segments from
   the satellite lazily as part of its download which
   requires the satellite telling it the locations of
   all of the segments which requires the satellite
   querying the locations of all of the segments. Instead
   the download RPC could return the orders for all of
   the segments for a range and the download code could
   issue N download calls rather than 1 download call and
   N get segment calls. I believe both sides of the code
   paths would be simpler and more efficient this way.

3. In looking at the timing information for downloads when
   testing this, we really need to focus on getting the
   auth key and bandwidth limit verification times down.
   Here's the timing I saw:

       - 42ms: validate auth
       - 52ms: bandwidth usage checking
       - 14ms: get object info
       - 26ms: get segment position info
       - 26ms: getting the first segment full info
       - 20ms: unaccounted for by spans
       - 6ms:  creating the orders

   This change will remove 26ms, but there's a good 90ms
   in just validation. With improved semantics hitting the
   database only once and improved validation, a download
   rpc taking ~30ms seems doable compared to our current
   ~200ms.

Change-Id: I4109dba082eaedb79e634c61dbf86efa93ab1222
2023-04-03 16:49:00 +00:00
Michal Niewrzal
d8724d67f3 satellite/metainfo: add metric about download range type
Change-Id: I8f0d589664232f6cedbf3f41212872735e7f4ac5
2023-04-03 10:04:03 +00:00
Ivan Fraixedes
d17b41d027
satellite/admin: Fix response code for Get bucket
Get bucket was returning a "bad request" HTTP status code when the
bucket doesn't exists.

We have to return HTTP "Not found" status code.

Change-Id: If717d99276b02a1e59a9b71ebc909bd6d8d9390b
2023-04-03 10:22:22 +02:00
Cameron
c2cd213c4f satellite/console: purchase package grants credit
Instead of granting a coupon when purchasing a package, grant credit.
This changes paymentsconfig.PackagePlan to use credit amount rather than
coupon ID. Add additional check to see if a paid invoice with the
description exists. If so, don't create and pay another invoice.

Change-Id: I81df24984c519c773db5fc8e9070bd7797070ec2
2023-03-31 22:29:54 +00:00
Cameron
a2e3247471 satellite/payments: add Balances interface
Add and implement interface to manage customer balances. Adds ability to
add credit to a user's balance, list balance transactions, and get the
balance.

Change-Id: I7fd65d07868bb2b7489d1141a5e9049514d6984e
2023-03-31 21:51:37 +00:00
Jeremy Wharton
e0bb410192 satellite/payments/stripecoinpayments: parallelize invoice methods
Invoicing-related payment service methods have been modified to send
Stripe API requests in parallel.

Additionally, randomness has been added to the Stripe backend wrapper's
exponential backoff strategy in order to reduce the effects of the
thundering herd problem, which arises when executing many simultaneous
API calls.

Resolves #5156

Change-Id: I568f933284f4229ef41c155377ca0cc33f0eb5a4
2023-03-31 17:14:22 +00:00
Cameron
21249e6c00 Revert "satellite/payments: add partnered field to coupon struct"
We are changing package plans to use credits instead of coupons.

This reverts commit 73ff35f160.

Change-Id: I1a5bc27a3a198e1aa09afbb06670dde776f66f36
2023-03-31 16:03:48 +00:00
Cameron
549732228d satellite/satellitedb: add package info to customers db
Add columns package_plan and purchased_package_at to stripe_customers
table and add methods to update and select these values from console
service and payments accounts.

Change-Id: I1e89909055cc3054bfb7baa33c9dca3dfdc7336e
2023-03-31 15:51:30 +00:00
Andrew Harding
f5e110f2aa satellite/satellitedb: support insert for many billing txs in one db tx
Implementing https://github.com/storj/storj/issues/5702 means adding a
bonus billing transaction for each storjscan transaction being recorded.
To do this idempotently, we need to the ability to for both the
storjscan and bonus transaction to be committed together.

This change updates the billing database to allow multiple billing
transactions to be inserted under the same database transaction.

Change-Id: I941864f47fc64d65aab076eec2e96fd04fcc7aac
2023-03-31 14:48:17 +00:00
Lizzy Thomson
d8a4ca62b8 satellite/accounting: update GetProjectSegmentLimit to fix bug
Update GetProjectSegmentLimit to reflect segment limit instead of
usage limit.

Fixes https://github.com/storj/storj/issues/5713

Change-Id: If4dfc5b45227595f3ae2b3ac7d267ea541fd748a
2023-03-31 11:47:21 +00:00
Moby von Briesen
277612f4ff satellite/console: Update onboarding logic
A row in the new `user_settings` table does not always exist for a user,
even if they have been around for a while.
Since `user_settings` is now what defines the state of a user's
onboarding flow, prior to this fix, even old users would receive the
onboarding flow again.
This change appropriately updates `user_settings` for users who already
have projects, and thus have already gone through the onboarding flow. A
brand new user will still be navigated to the beginning of onboarding.

Change-Id: Ie745d280f6b8094ec60c200c2dca8d018d51f7d1
2023-03-31 05:51:31 -04:00
Clement Sam
e5c43722dc storagenode/pieces: introduce FileWalker
FileWalker implements methods to walk over pieces in
in a storage directory.

This is just a refactor to separate filewalker functions
from pieces.Store. This is needed to simplify the work
to create a separate filewalker subprocess and reduce the
number of config flags passed to the subprocess.

You might want to check https://review.dev.storj.io/c/storj/storj/+/9773

Change-Id: I4e9567024e54fc7c0bb21a7c27182ef745839fff
2023-03-30 18:33:52 +00:00
Michal Niewrzal
dc9cbe36ae satellite/metabase: sort stats by creation time not number of entries
By accident query to get latest table stats was ordered by 'row_count'
column instead 'create'. We need latest stats so we need ordering by
creation time.

Change-Id: I9d0a0edda8bab59c3d96b7a15cd6502ed51633fc
2023-03-30 12:31:45 +00:00
Lizzy Thomson
63533c628b Satellite/console: Reduce free account limits
Update free account limits to segments(10k), storage(25gb), and
bandwidth(25gb)

Issue: https://github.com/storj/storj-private/issues/182

Change-Id: I59250fc9f1eebc812bc85ecfbc2924f8c4369064
2023-03-29 15:24:24 +00:00
Michal Niewrzal
54b6e1614a satellite/metainfo: drop MultipleVersions config flag
This flag was in general one time switch to enable versions internally.
New we can remove it as it makes code more complex.

Change-Id: I740b6e8fae80d5fac51d9425793b02678357490e
2023-03-29 13:45:09 +00:00
paul cannon
ae5947327b satellite/accounting: Use metabase.AliasPiece with tally observer
We want to eliminate usages of LoopSegmentEntry.Pieces, because it is
costing a lot of cpu time to look up node IDs with every piece of every
segment we read.

In this change, we are eliminating use of LoopSegmentEntry.Pieces in the
node tally observer (both the ranged loop and segments loop variants).
It is not necessary to have a fully resolved nodeID until it is time to
store totals in the database. We can use NodeAliases as the map key
instead, and resolve NodeIDs just before storing totals.

Refs: https://github.com/storj/storj/issues/5622
Change-Id: Iec12aa393072436d7c22cc5a4ae1b63966cbcc18
2023-03-29 12:24:05 +00:00
Wilfred Asomani
e750f6f145 satellite/accountfreeze: change price threshold default
The default price threshold has been changed to 10000 cents. The help
text has also been modified to clarify that this value is in cents.

Change-Id: I3b605056c860a3afef7202950180bc8b863fe16a
2023-03-28 16:05:08 +00:00
Wilfred Asomani
fd1a94a84d satellite/accountfreeze: add debug logs
A debug log has been added to the autofreeze chore to show numbers of
invoices, users involved, users frozen, users warned and users bypassed
due to larger than config amount owed.

Issue: https://github.com/storj/storj/issues/5700

Change-Id: I4e628606fe234b6d239b76167a4781ca9e276308
2023-03-28 15:30:31 +00:00
Wilfred Asomani
ed70a03844 satellite/{console,db,analytics}: better warning handling
This handles cases where a user is warned and triggers payment for their
account. Previously, only a frozen account will trigger this payment,
and will be unfrozen on successful payment. Now, accounts in warning
state trigger payments and are removed from that state on successful payment.

Issue: https://github.com/storj/storj/issues/5691

Change-Id: Icc2107f5d256657d176d8b0dd0a43a470eb01277
2023-03-28 14:30:38 +00:00
Vitalii
e4d9f8686d satellite/{web, console}: remove old access grant and billing flows
Removed old flows along with feature flags.

Issue: https://github.com/storj/storj/issues/5694

Change-Id: Ib3a1cbb12435500bdc2c540bb67615c64ca19e5e
2023-03-27 19:08:35 +00:00
Jeremy Wharton
ef5ebc9f64 satellite/satellitedb: fix panic when calculating project usage
Previously, our code produced a Go panic when attempting to index an
empty slice of bucket user agent entries. A length check has been
added to ensure that the slice is not accessed beyond its bounds.

Change-Id: I904ca275b934be1b05393a45d99ff415dcafc926
2023-03-27 07:44:50 +00:00
Wilfred Asomani
41bfbbe772 satellite/console: add endpoint to get user settings
This change adds an endpoint that gets a user's settings. It will
create a new settings entry if no settings exists. There's also a new
endpoint to change a user's onboarding status.

Issue: https://github.com/storj/storj/issues/5661

Change-Id: I9941bb9d61994af46244003f3ef4fcfe7d36918e
2023-03-24 22:19:08 +00:00
Wilfred Asomani
7c68b51c82 satellite/{console,db}: add onboardings columns
This change adds onboarding_start, onboarding_end and onboarding_step
columns to the user_settings table. the first two are used to determine
if a user should go through onboarding, the last will be used to as the
step a user got to before exiting onboarding without finishing.

Issue: https://github.com/storj/storj/issues/5660

Change-Id: I8070c880d0d2fc22086f24087c962f57c695cc50
2023-03-23 17:50:31 +00:00
Jeremy Wharton
01f0e602b4 satellite/payments/stripecoinpayments: retry API calls with exp backoff
A Stripe backend implementation has been added that uses an exponential
backoff strategy to retry failed API calls. This behavior can be
configured in the satellite config.

References #5156

Change-Id: I16ff21a39775ea331c442457f976be0c95a7b695
2023-03-22 07:20:27 -05:00
Kaloyan Raev
98b82486bd satellite/overlay: update country code on every node check-in
We have a specific issue that a user uploaded a file to a bucket
geo-fenced to EU and one of the pieces appeared to be on a node in the
US. The country code of this node is set to SE (Sweden) in the satellite
DB. It turns out that some time ago MaxMind changed the country code of
this node's IP from Sweden to US, but this change hasn't been reflected
in the satellite's database.

So far the satellite updates the country code of a node only if its IP
changes. It was assumed that if the IP does not change, its country code
shouldn't change too. This turned to be a wrong assumption.

With this change, the satellite will look up the MaxMindDB on every
check-in to see if the country code of the node's IP has changed.

Change-Id: Icdf659b09be9fc6ad14601902032b35ba5ea78c4
2023-03-22 08:38:51 +00:00
JT Olio
2a63225b98 satellite/{contact,satellitedb}: preserve node message debounce support
Change-Id: I453ad35fda4e61d068db0c476dd86b50d7f2d705
2023-03-20 16:13:06 +00:00
Wilfred Asomani
bf45e53169 satellite/{web,payments}: display correct stripe balance
This correctly displays the stripe balance in dollars, which was
previously displayed in cents.

Change-Id: Ifb14a63a90d3701201c616780cee466525c8be18
2023-03-17 11:54:08 -05:00
Moby von Briesen
0980a2f9ce satellitedb: Add index to users table
Add a combined index on normalized_email,status to improve performance of
common "get user" query used for the satellite UI.

Change-Id: I24a20d7826e0a68a68c2f95b5847eb819921e7c0
2023-03-16 09:08:49 -04:00
Vitalii
6c49cc883e satellite/console: enable new access grant flow by default
Flip feature flag to enable new acces grant flow by default

Change-Id: I66a963ecd368d1b9889b7ee2d88ee9806366efac
2023-03-15 15:36:21 +02:00
Jeremy Wharton
b95ef36541 satellite/payments/stripecoinpayments: add ctx to Stripe method calls
This commit ensures that all invocations of Stripe library methods
include a context. This allows us to control the timeout and
cancellation of the underlying HTTP requests made by the Stripe
library.

References #5156

Change-Id: I8ddb317f3f2cbb06cfab869fbebdaf2ad78b7999
2023-03-14 17:52:17 +00:00
Michal Niewrzal
e0577eedf5 satellite/metainfo: add missing metrics for metainfo requests
Change-Id: Icba9aa1ee552c7c180aa781de5ef9b87c4ec2f2d
2023-03-14 13:07:58 +00:00
Michal Niewrzal
ee720040c9 satellite/metainfo: use project limit cache with limiter
Metainfo needs to know rate and burst limit to be able to limit users
requests. We made cache for per project limiter but to make single
instance we need to know about limits. So far we were doing direct DB
call to get rate/burst limit for project but it's generating lots of
DB requests and can be easily cached as we even have project limit cache.

This change extends project limit cache with rate/burst limit and starts
using this change while creating project limiter instance for metainfo.

Because data size kept in project limit cache is quite small this change
also bumps a bit default capacity of the cache.

Fixes https://github.com/storj/storj/issues/5663

Change-Id: Icb42ec1632bfa0c9f74857b559083dcbd054d071
2023-03-14 08:11:11 +00:00
Michal Niewrzal
e181f4b90e satellite/accounting: test that project limit cache is really used
We just fixed case were project limit cache was not used properly. This
is test case to cover that fix.

Change-Id: Iee467f0a46836860a14ab6238a9842ffbf54ed4c
2023-03-13 17:40:55 +00:00
paul cannon
97e20bc579 scripts/tests: fix rollingupgrade test even more
This might be pretty awful, but at least it is a complete and non-flaky
solution.

**Only when using the rollingupgrade test** (which implies a throwaway
satellite and also a PostgreSQL backend), create a trigger on the nodes
table which forces last_net to be equal to last_ip_port always.

Change-Id: I8448cf131e46576d96a414d06780270c7b2b1892
2023-03-13 15:49:07 +00:00
Michal Niewrzal
06b51258be satellite/metabase: use table stats if are up to date
Currently, to get number of entries in segments table we are doing
heavy SELECT count(*) operation. For biggest satellite it's taking
25min now. We are using this method to get stat before and after
segments loop so it adds almost 1h to overall loop time.

With current version of crdb we are using this additional code won't be
used because global configuration for stats refresh rate is inaccurate
for such large table like `segments`. Soon we should be able to upgrade
crdb and be able to adjust refresh rate per table and configure it to
satisfy defined threshold.

https://github.com/storj/storj/issues/5544

Change-Id: I05cfd9154f08894d2bc56bf716b436d1b03b87f1
2023-03-13 14:54:13 +00:00
Jeremy Wharton
73ff35f160 satellite/payments: add partnered field to coupon struct
A field has been added to the coupon struct indicating whether it is
associated with a partner's pricing package. This is required to
alter the appearance of partner coupons in the satellite frontend.

References storj/storj-private#172

Change-Id: Ie48ae3902aaa108abf9a399242a0cd98cb53d1c3
2023-03-13 02:54:26 -05:00
Wilfred Asomani
0177ef79da satellite/{console,analytics,accountfreeze}: send unfreeze event
This change sends an event to segment for when a user is unfrozen.
It also moves freeze and warning event triggers from the autofreeze
chore to the account freeze service.

Change-Id: I5c0522b921b7baf52d6db5eb7ef841c08644a461
2023-03-10 23:21:20 +00:00
Cameron
4e94e6188c satellite/admin: restrict api access based on user groups
This change reworks the allowedAuthorization function to check what
groups the user is a part of to determine if authorization should be
granted. By wrapping each handler with withAuth, we can specify the
allowed groups for each api method individually.

github issue: https://github.com/storj/storj/issues/5565

Change-Id: I1804dda04d5b16d19e93bd7199fb3fc89fca1294
2023-03-10 16:16:04 -05:00
Lizzy Thomson
7e2d98988b cmd/tools: remove migration tool code for generating missing salt
Remove generate-missing-project-salt migration tool code and related
tests. This migration has already been run and this code is no longer
needed.

Issue https://github.com/storj/storj-private/issues/163

Change-Id: I4e36dcd95a07c5305c597113a7fd08148e100ccc
2023-03-10 16:38:54 +00:00
Michal Niewrzal
0696256769 satellite/accounting: fix project limit cache usage
It looks that at some point we broke how project limits cache is used
and we were missing cache in most critical paths (upload/download).
This is fix for this issue.

I also adjusted cache methods naming.

Change-Id: Ic98372779a39365d0920fe3943f1f7a68b064173
2023-03-10 14:42:38 +00:00
Wilfred Asomani
b1d4763086 satellite/{web,payments}: show STORJ balance
This change adds a card to the billing overview page, which shows the
user's token balance from coinpayments.

Issue: https://github.com/storj/storj-private/issues/151

Change-Id: I11e295b48791b32b745cb7a11c5b4aad6b56618e
2023-03-10 13:01:15 +00:00
paul cannon
b98a09fa58 satellite/geoip: skip test on macOS
This is how we deal with the other tests that expect all 127.* addresses
to work locally.

Change-Id: Id62bb07a56044dfadcc87089f0f13684eeb07f3c
2023-03-10 03:08:17 +00:00
paul cannon
fd6ce6b9a5 scripts/tests: fix test-sim-rolling-upgrade.sh
This test involves a satellite with dev defaults (DistinctIP=no) being
upgraded past commit 2522ff09b6, which
means we need to run the dev-defaults-satellite-upgrade migration SQL
to avoid getting DistinctIP=yes behavior (which breaks the tests).

Change-Id: I29fb596d1ffa568dad635d98cfe9abacd3aaa48f
2023-03-09 23:35:36 +00:00
Michal Niewrzal
0c177ef91f satellite: cleanup orders dependencies
Only API peer needs access to order DB (and rollups cache) because it's
only place where we are creating orders for PUT and GET operations. For
other peers like auditor and repairer we can set noop implementation to
reduce number of dependencies needed for them.

Change-Id: Ic32d1879f0b97ffc4516f401898e31e95ae892e4
2023-03-09 13:34:21 +00:00
Márton Elek
ffaf15a3b0 satellite/overlay: remove unused mail service from overlay
It was surprising that `satellite auditor` complained about SMTP mail settings, even if it's not supposed to sending any mail.

Looks like we can remove the mail service dependency, as it's not a hard requirement for overlay.Service.

Change-Id: I29a52eeff3f967ddb2d74a09458dc0ee2f051bd7
2023-03-09 12:17:35 +00:00
Márton Elek
788f5fde01 satellite/peer: better error handling for SMTP parameters
I tried to configure a satellite service and got this error:

```
DEBUG   process/exec_conf.go:408        Unrecoverable error     {"error": "missing port in address"}
```

It took some time to realize that I forgot to set the SMTPServerAddress.

This patch makes it easier to detect similar problem (detailed error message), and makes SMTP parameters optional if no real mail sending is used (simulated or nomail)

Change-Id: I32535a7c8d6529e19e4d919806f42ba430d074a5
2023-03-09 11:29:07 +00:00
paul cannon
2522ff09b6 satellite/overlay: configurable meaning of last_net
Up to now, we have been implementing the DistinctIP preference with code
in two places:

 1. On check-in, the last_net is determined by taking the /24 or /64
    (in ResolveIPAndNetwork()) and we store it with the node record.
 2. On node selection, a preference parameter defines whether to return
    results that are distinct on last_net.

It can be observed that we have never yet had the need to switch from
DistinctIP to !DistinctIP, or from !DistinctIP to DistinctIP, on the
same satellite, and we will probably never need to do so in an automated
way. It can also be observed that this arrangement makes tests more
complicated, because we often have to arrange for test nodes to have IP
addresses in different /24 networks (a particular pain on macOS).

Those two considerations, plus some pending work on the repair framework
that will make repair take last_net into consideration, motivate this
change.

With this change, in the #2 place, we will _always_ return results that
are distinct on last_net. We implement the DistinctIP preference, then,
by making the #1 place (ResolveIPAndNetwork()) more flexible. When
DistinctIP is enabled, last_net will be calculated as it was before. But
when DistinctIP is _off_, last_net can be the same as address (IP and
port). That will effectively implement !DistinctIP because every
record will have a distinct last_net already.

As a side effect, this flexibility will allow us to change the rules
about last_net construction arbitrarily. We can do tests where last_net
is set to the source IP, or to a /30 prefix, or a /16 prefix, etc., and
be able to exercise the production logic without requiring a virtual
network bridge.

This change should be safe to make without any migration code, because
all known production satellite deployments use DistinctIP, and the
associated last_net values will not change for them. They will only
change for satellites with !DistinctIP, which are mostly test
deployments that can be recreated trivially. For those satellites which
are both permanent and !DistinctIP, node selection will suddenly start
acting as though DistinctIP is enabled, until the operator runs a single
SQL update "UPDATE nodes SET last_net = last_ip_port". That can be done
either before or after deploying software with this change.

I also assert that this will not hurt performance for production
deployments. It's true that adding the distinct requirement to node
selection makes things a little slower, but the distinct requirement is
already present for all production deployments, and they will see no
change.

Refs: https://github.com/storj/storj/issues/5391
Change-Id: I0e7e92498c3da768df5b4d5fb213dcd2d4862924
2023-03-09 02:20:12 +00:00
Michal Niewrzal
67ad792d1a satellite/rangedloop: migrate segments verification from segment loop
Segments loop have build-in sanity check to verify if number of segments
processed by loop is roughly fine. We want to have the same verification
for ranged loop.

https://github.com/storj/storj/issues/5544

Change-Id: Ia19edc0fb4aa8dc45993498a8e6a4eb5928485e9
2023-03-08 17:00:11 +00:00
Cameron
f4ea730e69 satellite/console: use custom session duration
If the session_minutes column in user_settings is not null, use this
value to determine the expiration when creating or refreshing a session.

github issue: https://github.com/storj/storj/issues/5487

Change-Id: I1f777f7ffcd4018b0c40f0b26c8a50de85a31909
2023-03-08 14:55:02 +00:00
Erik van Velzen
464ceb1c0e satellite/gc: improve comments
Change-Id: I9e71c9bee3447f78365ba1593e4a4ef55b28356f
2023-03-08 13:15:13 +00:00
Vitalii
a01d47b3ed satellite:{console, web}: remove old project dashboard
Removed old project dashboard along with a feature flag.

Change-Id: I9292b19caf101a84042d8c3006c27bc44db8d6b5
2023-03-07 13:34:59 +02:00
Vitalii
b6721633f1 satellite/payments: return public project ID for project charges
We should return public project id instead of regular one when quering project charges so that it is consistent on a client side.
This is a fix for an issue with displaying project name on a Billing screen.

Issue:
https://github.com/storj/storj/issues/5641

Change-Id: Ic0c544dbe7369aa88f3a7c97a2f5de5bc854e1fd
2023-03-06 22:50:21 +00:00
JT Olio
5b0cada4b3 repairer: monitor non-nil limit amount
Change-Id: I1a7b7a4a6716783449704cd8a7823090109a14de
2023-03-06 20:39:45 +00:00
Michal Niewrzal
bc8f8f62b5 satellite/orders: cleanup after altering primary key
We changed primary key for bucket_bandwidth_rollups table. Now we
need to do some cleanup in places like structs, sorting methods or SQL
queries.

Change-Id: Ida4f874f161356df193379a53507602e04db1668
2023-03-06 16:03:11 +00:00
Egon Elbre
63fa386b0a satellite/payments/stripecoinpayments: avoid mock cross-talk in tests
The tests were using global variables for keeping the mock state, which
was indexed by the satellite ID. However, the satellite ID-s are
deterministic and it's possible for two tests end up using the same
mocks.

Instead make the mock creation not depend on the satellite ID and
instead require it being configured via paymentsconfig.

This fixes TestAutoFreezeChore failure.

Change-Id: I531d3550a934fbb36cff2973be96fd43b7edc44a
2023-03-03 20:20:48 +02:00
Michal Niewrzal
d54ccfa92b satellite/satellitedb: change bucket_bandwidth_rollups primary key
In the past we set incorrect primary key for bucket_bandwidth_rollups
table and bucket name was first column of key instead of project id.
This change fixes this problem.

We will alter primary key manually for production satellites. That's
why migration code is checking if change was already applied.

https://github.com/storj/storj/issues/5332

Change-Id: I7dd555eb0c2f6fda1c90eaed08fa847b7f273bc7
2023-03-03 16:25:35 +00:00
Moby von Briesen
e9628c133d satellite/console: Add comments to generated api
Clarify that this functionality is not currently implemented within the
Satellite UI

Change-Id: I94096ab9b4d8f00195b55f5e875d76930de848e4
2023-03-03 09:29:11 -05:00
Michal Niewrzal
5a9577bb43 satellite/metabase/rangedloop: test loop boundaries
Fixes https://github.com/storj/storj/issues/5620

Change-Id: Ibf23e8866c1a8a8e78281c3d07e81ffc7db30fcf
2023-03-03 09:56:01 +00:00
Jeremy Wharton
3b37a23d7b satellite/console/consoleweb: add pricing package availability endpoint
An endpoint has been added that returns whether a pricing package is
available for a user to purchase. This will be used to conditionally
skip the pricing plan selection step of the onboarding tour.

Change-Id: I8c02a4e474e5f0f80778453b7daf674c8da64306
2023-03-02 23:51:09 +00:00
Cameron
fb471d7d8d satellite/console: delete password recovery token when password is changed
If a user changes password while a password recovery token exists,
delete the token so the reset password email doesn't work.

github issue: https://github.com/storj/storj-private/issues/149

Change-Id: Ibb0c13076e182c46501c1d7ac92d3fecc7d5ee58
2023-03-02 22:47:55 +00:00
Lizzy Thomson
54f0f47839 satellite/satellitedb: add passphrase_prompt column to user_settings
Add passphrase_prompt column to user_settings table to store a boolean
whether a user would like to be prompted to enter a passphrase when
entering a new project.

Issue https://github.com/storj/storj/issues/5613

Change-Id: I5d086df277aff458453343c7c38a379e7b1b8bf9
2023-03-01 14:53:18 -07:00
Michal Niewrzal
b46c0fb78f satellite/orders: don't cancel flushing bandwidth orders
Earlier we made a change to not cancel flushing orders when flushing
was triggered by orders endpoint method but we missed a case
where it can be also triggered (and canceled) by metainfo endpoints
method. This change moves ignoring context cancellation deeper.

Change-Id: Id43176f552efc3167345783f73aab885411ac247
2023-03-01 17:10:05 +00:00
Wilfred Asomani
faeea88265 satellite/{db,analytics,payments}: add chore for auto account freeze
This change adds a new chore that will check for failed invoices and
potentially freeze corresponding accounts.
It makes slight modifications to stripemock.go and invoices.go (adding
stripe CustomerID to the Invoice struct).

Issue: https://github.com/storj/storj-private/issues/140

Change-Id: I161f4037881222003bd231559c75f43360509894
2023-03-01 09:31:27 +00:00
Jeremy Wharton
31ec4fae9e web/satellite: add pricing plan selection to onboarding tour
A pricing plan selection step for users with a recognized partner has
been added to the beginning of the onboarding tour. Once visited, users
have the option of purchasing the pricing plan associated with their
partner or proceeding as a paid or free tier user.

Resolves storj-private#118
Resolves storj-private#126

Change-Id: I3b423194d96deaf87cf9807a766bf4d04fbcf86d
2023-02-28 20:49:59 -06:00
JT Olio
4362761fc7 satellite/audit: fix go1.19 dial timeouts and log more
Change-Id: Ide17c1b8e0ca8c86f305bea1b4ae553cc4cb60d0
2023-02-28 17:09:47 +00:00
Egon Elbre
f13d0f7df0 satellite/satellitedb/dbx: document oauth tables
Change-Id: Ide327699f4b7b35f46c9356d558221ece5bd77a9
2023-02-28 09:07:58 +00:00
Cameron
b24ce98944 satellite/admin: add Groups to config
add new config to the satellite admin: --admin.groups.limit-update.
This can be used as an alternate means of authentication if the request
is coming from the oauth proxy.

Change-Id: Ic2de13862e6414244b060c66a0f2bed72097cbad
2023-02-27 16:07:20 +00:00
Egon Elbre
1cef2334ff satellite/satellitedb/dbx: document billing
Change-Id: Ief90cc0a6bebaa8df9dd9255565eaa42dc0abc9e
2023-02-27 15:13:23 +00:00
Egon Elbre
4aa519a69f satellite/satellitedb/dbx: document node payments
Change-Id: Ie65ec960f1683f5735a2b5959fedbde326007d04
2023-02-27 14:13:35 +00:00
Egon Elbre
1f439f5ae9 satellite/satellitedb/dbx: document audit tables
Change-Id: Iae0f2d7eaddb30cf43a3a7cd678f1dcfb8256a45
2023-02-27 09:56:39 +00:00
Jeremy Wharton
cbbd5ab1ef satellite/payments/stripecoinpayments: undo price override removal
Commit fb59974 disabled usage price overrides because of a failing
test. This change reenables it while resolving the issue that caused
the test to fail.

The previous version of the test passed Gerrit verification and was
merged, but it failed for the primary Jenkins pipeline after merge.

This is due to a difference in how the Jenkins build runs Cockroach
and Postgres for each pipeline.

This commit rewrites the test to be safe for concurrent execution by
ensuring any mutable variables are defined within each test so that
shared state across tests is reduced.

Change-Id: Ia4566c9cd2d698afdb2caa4b7e2808b17e18de4e
2023-02-24 16:54:51 +00:00
paul cannon
20bcdeb8b1 satellite/repair: fix flaky test TestECREpairerGetOffline
It was possible to get into a situation where successfulPieces =
es.RequiredCount(), errorCount < minFailures, and inProgress == 0 (when
the succeeding gets all completed before the failures), whereupon the
last goroutine in the limiter would sit and wait forever for another
goroutine to finish.

This change corrects the handling of that situation.

As an aside, this is really pretty confusing code and we should think
about redoing the whole function.

Change-Id: Ifa3d3ad92bc755e563fd06b2aa01ef6147075a69
2023-02-24 09:05:21 -06:00
Egon Elbre
4a6e34bb2c satellite/satellitedb/dbx: document peer_identity table
Change-Id: I112ef00cb80638a8168dd46a531522b0785b537a
2023-02-24 14:06:46 +00:00
Egon Elbre
cf324c5b6e satellite/satellitedb/dbx: document node tables
Also, fix nodevents iota usage. Type is directly used in the database
hence we should avoid iota, because it's easy to mess-up the constant
values.

Change-Id: I32bf6d0502fbf3d18660dab5b1dd0b5627b444df
2023-02-24 14:05:55 +00:00
Egon Elbre
e4914e9b7a satellite/satellitedb/dbx: document accounting tables
Change-Id: I94478892a37623882d306a811e272f51865e48dd
2023-02-24 16:04:29 +02:00
Márton Elek
dfe3c87b87 satellite/metabase/rangedloop: save observer durations to eventkit
Change-Id: I2d027cd7f7421e2c38dcc1277d640d4183af4cec
2023-02-24 13:30:19 +00:00
Egon Elbre
4e70ef83fb satellite/satellitedb/dbx: split accounting.dbx by project and node
Change-Id: Ia3dee52abf98a41ea62d94af0a155f4f09661224
2023-02-24 09:08:20 +00:00
Cameron
8842985571 satellite/console/consoleweb: create purchase-package endpoint
Add new purchase-package endpoint to Server. The endpoint can be enabled
or disabled by a new config, --console.pricing-packages-enabled.
The purchase-package endpoint applies a coupon and adds and charges a
credit card if user's useragent is a partner with a configured package
plan.

github issue: https://github.com/storj/storj-private/issues/125

Change-Id: I0d6ccccd6874ddba360c45f338fd1c44f95e135a
2023-02-23 15:42:36 -05:00
Egon Elbre
1a2ad602d7 satellite/satellitedb/dbx: document gracefulexit tables
Change-Id: Ia76b67fa85e797c1d75438ae882bdfe83a6ee057
2023-02-23 19:14:51 +02:00
Jeremy Wharton
fb5997484e satellite/payments/stripecoinpayments: remove usage price overriding
Project usage price overriding has been removed because it produces
incorrect results when tested. It should not be re-implemented until
the issues it causes are resolved.

Change-Id: Ic92eff374c9af4fea3bf32782a72303a7978b055
2023-02-23 13:36:59 +00:00