Commit Graph

766 Commits

Author SHA1 Message Date
Michal Niewrzal
98562d06c8 satellite/gc/bloomfilter: add sync observer
Current observer used with ranged loop is using massive amount of
memory because each range is generating separate set of bloom filters.
Each bloom filter can be up to 2MB of memory. That's a lot.

This change is initial change to reduce used memory by sharing bloom
filters between ranges and just synchronize access to them. This
implementation is rather simple and even naive but maybe it will be
enough without doing something more complex.

https://github.com/storj/storj/issues/5803

Change-Id: Ie62d19276aa9023076b1c97f712b788bce963cbe
2023-04-28 07:40:56 +00:00
Michal Niewrzal
1aa24b9f0d satellite/audit: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I9cec0ac454f40f19d52c078a8b1870c4d192bd7a
2023-04-24 15:52:11 +00:00
Michal Niewrzal
6ac5bf0d7c satellite/gracefulexit: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: Ia3e2d7879d91f7f5ffa99b8e8f108380e3b39f31
2023-04-24 15:00:26 +00:00
Michal Niewrzal
6a55682bc6 satellite/accounting/nodetally: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I6176a129ba14cf83fb635048d09e6748276b52a1
2023-04-24 14:25:53 +00:00
Michal Niewrzal
fbfe5aaad7 satellite/metrics: remove code related to segments loop
We are switching completely to ranged loop.

Change-Id: I32120ef496addebec2de088fd10d0c1d02313c68
2023-04-20 13:47:22 +00:00
Jeremy Wharton
6e9dae06d5 satellite/payments: add egress discount ratio to price override config
This change allows for specifying the ratio of free egress per unit
of storage within a price override configuration.

References storj/storj-private#215
References storj/storj-private#224

Change-Id: Ib1c79f77ec8bb11dd5b2f9dace13800b0b3ce942
2023-04-12 14:52:19 +00:00
Michal Niewrzal
31f5e2cb65 satellite/metainfo: limit uploads to the same location
We would like to have ability to limit burst uploads to the single
object (the same location). This change we are limiting such upload to
one per second.

Change-Id: Ib9351df1017cbc07d7fc2f846c2dbdbfcd3a360c
2023-04-06 15:49:11 +00:00
Clement Sam
5ab2e6ac19 go.mod: bump storj.io/private
Updates https://github.com/storj/storj/issues/5349

Change-Id: I5b91a8e9c28d39c6ec432992a352de08aff886d9
2023-04-05 14:46:59 +03:00
Cameron
c2cd213c4f satellite/console: purchase package grants credit
Instead of granting a coupon when purchasing a package, grant credit.
This changes paymentsconfig.PackagePlan to use credit amount rather than
coupon ID. Add additional check to see if a paid invoice with the
description exists. If so, don't create and pay another invoice.

Change-Id: I81df24984c519c773db5fc8e9070bd7797070ec2
2023-03-31 22:29:54 +00:00
Jeremy Wharton
e0bb410192 satellite/payments/stripecoinpayments: parallelize invoice methods
Invoicing-related payment service methods have been modified to send
Stripe API requests in parallel.

Additionally, randomness has been added to the Stripe backend wrapper's
exponential backoff strategy in order to reduce the effects of the
thundering herd problem, which arises when executing many simultaneous
API calls.

Resolves #5156

Change-Id: I568f933284f4229ef41c155377ca0cc33f0eb5a4
2023-03-31 17:14:22 +00:00
Lizzy Thomson
63533c628b Satellite/console: Reduce free account limits
Update free account limits to segments(10k), storage(25gb), and
bandwidth(25gb)

Issue: https://github.com/storj/storj-private/issues/182

Change-Id: I59250fc9f1eebc812bc85ecfbc2924f8c4369064
2023-03-29 15:24:24 +00:00
Michal Niewrzal
54b6e1614a satellite/metainfo: drop MultipleVersions config flag
This flag was in general one time switch to enable versions internally.
New we can remove it as it makes code more complex.

Change-Id: I740b6e8fae80d5fac51d9425793b02678357490e
2023-03-29 13:45:09 +00:00
Wilfred Asomani
e750f6f145 satellite/accountfreeze: change price threshold default
The default price threshold has been changed to 10000 cents. The help
text has also been modified to clarify that this value is in cents.

Change-Id: I3b605056c860a3afef7202950180bc8b863fe16a
2023-03-28 16:05:08 +00:00
Vitalii
e4d9f8686d satellite/{web, console}: remove old access grant and billing flows
Removed old flows along with feature flags.

Issue: https://github.com/storj/storj/issues/5694

Change-Id: Ib3a1cbb12435500bdc2c540bb67615c64ca19e5e
2023-03-27 19:08:35 +00:00
Jeremy Wharton
01f0e602b4 satellite/payments/stripecoinpayments: retry API calls with exp backoff
A Stripe backend implementation has been added that uses an exponential
backoff strategy to retry failed API calls. This behavior can be
configured in the satellite config.

References #5156

Change-Id: I16ff21a39775ea331c442457f976be0c95a7b695
2023-03-22 07:20:27 -05:00
JT Olio
0b4b04900a private/server: debounce noise and tls connections
to support TCP_FAST_OPEN, we're considering just using
two TCP connections in parallel per request, one with
and one without. this allows us to safely fire both
concurrently without stressing out the node too much.

see https://review.dev.storj.io/c/storj/storj/+/9933

Change-Id: I9aa8a0252350db5ace04ee125bfe469203e980ec
2023-03-21 16:51:31 +00:00
Vitalii
6c49cc883e satellite/console: enable new access grant flow by default
Flip feature flag to enable new acces grant flow by default

Change-Id: I66a963ecd368d1b9889b7ee2d88ee9806366efac
2023-03-15 15:36:21 +02:00
Michal Niewrzal
ee720040c9 satellite/metainfo: use project limit cache with limiter
Metainfo needs to know rate and burst limit to be able to limit users
requests. We made cache for per project limiter but to make single
instance we need to know about limits. So far we were doing direct DB
call to get rate/burst limit for project but it's generating lots of
DB requests and can be easily cached as we even have project limit cache.

This change extends project limit cache with rate/burst limit and starts
using this change while creating project limiter instance for metainfo.

Because data size kept in project limit cache is quite small this change
also bumps a bit default capacity of the cache.

Fixes https://github.com/storj/storj/issues/5663

Change-Id: Icb42ec1632bfa0c9f74857b559083dcbd054d071
2023-03-14 08:11:11 +00:00
Michal Niewrzal
38c7671acb satellite: bump capacity for API keys cache
We have lots of direct DB requests to get API keys. It should be handled
by cache but default value is very low at the moment.

Fixes https://github.com/storj/storj/issues/5665

Change-Id: I214ebebd6e397cacff80b2f36dc4a2eea388f93d
2023-03-13 16:32:50 +00:00
paul cannon
97e20bc579 scripts/tests: fix rollingupgrade test even more
This might be pretty awful, but at least it is a complete and non-flaky
solution.

**Only when using the rollingupgrade test** (which implies a throwaway
satellite and also a PostgreSQL backend), create a trigger on the nodes
table which forces last_net to be equal to last_ip_port always.

Change-Id: I8448cf131e46576d96a414d06780270c7b2b1892
2023-03-13 15:49:07 +00:00
paul cannon
fd6ce6b9a5 scripts/tests: fix test-sim-rolling-upgrade.sh
This test involves a satellite with dev defaults (DistinctIP=no) being
upgraded past commit 2522ff09b6, which
means we need to run the dev-defaults-satellite-upgrade migration SQL
to avoid getting DistinctIP=yes behavior (which breaks the tests).

Change-Id: I29fb596d1ffa568dad635d98cfe9abacd3aaa48f
2023-03-09 23:35:36 +00:00
Stefan Benten
4788b9ea3b
scripts/draft-release.sh: update to python3
Change-Id: Iabdf9283f77c1a8518f2373d39157d10e1daab6d
2023-03-09 10:59:31 +01:00
paul cannon
2522ff09b6 satellite/overlay: configurable meaning of last_net
Up to now, we have been implementing the DistinctIP preference with code
in two places:

 1. On check-in, the last_net is determined by taking the /24 or /64
    (in ResolveIPAndNetwork()) and we store it with the node record.
 2. On node selection, a preference parameter defines whether to return
    results that are distinct on last_net.

It can be observed that we have never yet had the need to switch from
DistinctIP to !DistinctIP, or from !DistinctIP to DistinctIP, on the
same satellite, and we will probably never need to do so in an automated
way. It can also be observed that this arrangement makes tests more
complicated, because we often have to arrange for test nodes to have IP
addresses in different /24 networks (a particular pain on macOS).

Those two considerations, plus some pending work on the repair framework
that will make repair take last_net into consideration, motivate this
change.

With this change, in the #2 place, we will _always_ return results that
are distinct on last_net. We implement the DistinctIP preference, then,
by making the #1 place (ResolveIPAndNetwork()) more flexible. When
DistinctIP is enabled, last_net will be calculated as it was before. But
when DistinctIP is _off_, last_net can be the same as address (IP and
port). That will effectively implement !DistinctIP because every
record will have a distinct last_net already.

As a side effect, this flexibility will allow us to change the rules
about last_net construction arbitrarily. We can do tests where last_net
is set to the source IP, or to a /30 prefix, or a /16 prefix, etc., and
be able to exercise the production logic without requiring a virtual
network bridge.

This change should be safe to make without any migration code, because
all known production satellite deployments use DistinctIP, and the
associated last_net values will not change for them. They will only
change for satellites with !DistinctIP, which are mostly test
deployments that can be recreated trivially. For those satellites which
are both permanent and !DistinctIP, node selection will suddenly start
acting as though DistinctIP is enabled, until the operator runs a single
SQL update "UPDATE nodes SET last_net = last_ip_port". That can be done
either before or after deploying software with this change.

I also assert that this will not hurt performance for production
deployments. It's true that adding the distinct requirement to node
selection makes things a little slower, but the distinct requirement is
already present for all production deployments, and they will see no
change.

Refs: https://github.com/storj/storj/issues/5391
Change-Id: I0e7e92498c3da768df5b4d5fb213dcd2d4862924
2023-03-09 02:20:12 +00:00
Michal Niewrzal
67ad792d1a satellite/rangedloop: migrate segments verification from segment loop
Segments loop have build-in sanity check to verify if number of segments
processed by loop is roughly fine. We want to have the same verification
for ranged loop.

https://github.com/storj/storj/issues/5544

Change-Id: Ia19edc0fb4aa8dc45993498a8e6a4eb5928485e9
2023-03-08 17:00:11 +00:00
Erik van Velzen
464ceb1c0e satellite/gc: improve comments
Change-Id: I9e71c9bee3447f78365ba1593e4a4ef55b28356f
2023-03-08 13:15:13 +00:00
Sembeth
0408997e6c
cmd/multinode: Removes dependency on deprecated identity-dir flag, code and documentation. (#5646)
* The Multinode Dashboard no longer requires the identity files.

* Removed deprecated code that used the identity-dir.
2023-03-08 13:56:15 +01:00
Vitalii
a01d47b3ed satellite:{console, web}: remove old project dashboard
Removed old project dashboard along with a feature flag.

Change-Id: I9292b19caf101a84042d8c3006c27bc44db8d6b5
2023-03-07 13:34:59 +02:00
Wilfred Asomani
faeea88265 satellite/{db,analytics,payments}: add chore for auto account freeze
This change adds a new chore that will check for failed invoices and
potentially freeze corresponding accounts.
It makes slight modifications to stripemock.go and invoices.go (adding
stripe CustomerID to the Invoice struct).

Issue: https://github.com/storj/storj-private/issues/140

Change-Id: I161f4037881222003bd231559c75f43360509894
2023-03-01 09:31:27 +00:00
Cameron
b24ce98944 satellite/admin: add Groups to config
add new config to the satellite admin: --admin.groups.limit-update.
This can be used as an alternate means of authentication if the request
is coming from the oauth proxy.

Change-Id: Ic2de13862e6414244b060c66a0f2bed72097cbad
2023-02-27 16:07:20 +00:00
Cameron
8842985571 satellite/console/consoleweb: create purchase-package endpoint
Add new purchase-package endpoint to Server. The endpoint can be enabled
or disabled by a new config, --console.pricing-packages-enabled.
The purchase-package endpoint applies a coupon and adds and charges a
credit card if user's useragent is a partner with a configured package
plan.

github issue: https://github.com/storj/storj-private/issues/125

Change-Id: I0d6ccccd6874ddba360c45f338fd1c44f95e135a
2023-02-23 15:42:36 -05:00
Michal Niewrzal
8aa6110835 tests/backwardcompatibility: use noquic tag for release binaries
Older releases are not compiling with latest Go version if quic is used.
We need to add noquic tag to be able to compile older release with
latest Go version.

Change-Id: Id5768fcaa5c1f7cf3e6fbb633e7ca60309b7a37c
2023-02-21 13:26:46 +01:00
JT Olio
77bf88e916 satellite/overlay: check node difficulty before entering database
closes https://github.com/storj/storj/issues/5568

Change-Id: Id413637c2678e7a7cf8dbf414e082c687c8e8a39
2023-02-15 17:46:25 +00:00
Wilfred Asomani
9138e84fb1 satellite/{web,console}: feature flag for all projects dashboard
Adds a feature flag for the new all projects dashboard. It defaults to false.

Issue: https://github.com/storj/storj/issues/5514

Change-Id: I160904eccae7d30e05b734e69600725702b16aca
2023-02-15 15:53:31 +00:00
Egon Elbre
a8caea8b9c satellite/satellitedb: remove gob migration exceptions
Change-Id: I2958e2145edaa501d510314f9bfd0e5280ea82ec
2023-02-10 15:05:18 +02:00
Vitalii
65e3cfb9c6 satellite/{web, console}: feature flag for new access grant flow
Added a feature flag for new create access grant flow.
Also added some initial setup.

Change-Id: I7f738181c8a83f5a724f9e562427445cae146b6f
2023-02-09 00:35:35 +00:00
paul cannon
d6f8be1ec6 satellite/audit: add ContainmentSyncChore
We will be needing an infrequent chore to check which nodes are in the
reverify queue and synchronize that set with the 'contained' field in
the nodes db, since it is easily possible for them to get out of sync.
(We can't require that the reverification queue table be in the same
database as the nodes table, so maintaining consistency with SQL
transactions is out. Plus, even if they were in the same database, using
such SQL transactions to maintain consistency would be slow and
unwieldy.)

This commit adds the actual chore.

Refs: https://github.com/storj/storj/issues/5431
Change-Id: Id78b40bf69fae1ac39010e3b553315db8a1472bd
2023-02-07 01:18:49 +00:00
Michal Niewrzal
0b519fbd05 satellite/satellitedb: create index for interval_day for project_bandwidth_daily_rollup
To improve deletion of old entries in project_bandwidth_daily_rollup
we need index on `interval_day` column which is used to find those old
entries.

As an addition we are changing interval how often deletion is executed
from 7 to 1 day. We would like to have smaller portion of data to
delete.

Fixes https://github.com/storj/storj/issues/5465

Change-Id: Ie18ebe859887b93d6e4e6065a61fb9214c7ad27a
2023-02-03 10:02:58 +00:00
Egon Elbre
3976a2fd1d go.mod: bump common to latest
Temporarily allow larger wasm size, while we migrate to picobuf.

Change-Id: I55916f47486465e4223cbd79f412ab2876fee042
2023-02-01 12:33:37 +02:00
Vitalii
5377b9c314 satellite/{web, console}: removed old project level passphrase flow
Removed old behavior and bucket creation flow.

Issue:
https://github.com/storj/storj/issues/5521

Change-Id: I1a443911215cd1f611069fd9e6202598066b80b2
2023-01-31 23:02:44 +00:00
JT Olio
2a641b60d3 private/server: support tcp fastopen
we are not enabling it on client-side code yet, but
it will be hard to test this in the wild without
server-side support.

this is phase 2 of tcp fast open support. see
https://forum.storj.io/t/two-new-blueprints-design-drafts-seeking-feedback-replacing-tls-with-noise-and-tcp-fastopen/21053/12
for more details

Change-Id: I20362be4c49abab90afcc9b6572ef9f79816345b
2023-01-30 13:30:43 -05:00
Michal Niewrzal
bb2ac4279a satellite/metainfo: enable multiple versions fix by default
Change-Id: I6cc7ba928e59cc8b8fa50f2ab19ec5418dc76507
2023-01-26 09:35:20 +00:00
Andrew Harding
e7b35381f2 mod: bump common and uplink deps
Change-Id: Ie2d87f517b57a8760f50ac5c9f1b8b0448f5b3b5
2023-01-25 13:34:47 -07:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
Qweder93
d6a948f59d satellite/repair : implemented ranged loop observer
implemented observer and partial, created new structures to keep mon
metrics remain in same way as in segment loop

Change-Id: I209c126096c84b94d4717332e56238266f6cd004
2023-01-23 14:23:03 +00:00
Cameron
7ffa9ef914 satellite/payments: add config for partner package plans
Create a config to specify one-time prices and corresponding coupon
ids for partners.

github issue: https://github.com/storj/storj-private/issues/118

Change-Id: I67b26e7208b12ba8f0e6dc1b164dd9545b09cac0
2023-01-20 02:33:36 +00:00
Vitalii
65a090d111 satellite/console: enable project level passphrase by default
Enable project level passphrase by default.

Issue:
https://github.com/storj/storj/issues/5456

Change-Id: I8c08f2d81a3ce1b5e0134172b7440309b850c5c2
2023-01-18 15:32:29 +02:00
Yaroslav Vorobiov
5644fb1a7e satellite/accounting/nodetally: add ranged loop
Add node tally ranged loop observer and partial.
Add node tally randed observer to range loop peer.
Add config flag to select which loop to use for node tally.
Update satellite core to use segement/ranged loop based on a flag.
Duplicate existing node tally test but using ranged loop.

Change-Id: I6786f1a16933463fab5f79601bf438203a7a5f9e
2023-01-17 13:50:18 +01:00
Qweder93
8c69ee62fc {cmd/storj-sim, satellite/rangedloop}: added rangedloop to storj-sim, removed identity
added in storj-sim rangedloop for each satellite, to verify it works for metrics oveserver,
removed identity from rangedloop peer as we never use it, added logs on service run, added loop
to service instead of endless for loop, interval value to config

Closes: https://github.com/storj/storj/issues/5414

Change-Id: Ibc3b06071b68feda4a35b45da2bbe36e22a02fc8
2023-01-05 11:29:00 +00:00
Andrew Harding
5362dff94b satellite/gc/bloomfilter: implement rangedloop observer
https://github.com/storj/storj/issues/5235

Change-Id: Iffe8f682adfa46e48e47976bf838326e7125ff80
2023-01-03 09:46:02 -07:00
Andrew Harding
590d44301c satellite/audit: implement rangedloop observer
This change implements the ranged loop observer to replace the audit
chore that builds the audit queue.

The strategy employed by this change is to use a collector for each
segment range to  build separate per-node segment reservoirs that are
then merge them during the join step.

In previous observer migrations, there were only a handful of tests so
the strategy was to duplicate them. In this package, there are dozens
of tests that utilize the chore. To reduce code churn and maintenance
burden until the chore is removed, this change introduces a helper that
runs tests under both the chore and observer, providing a pair of
functions that can be used to pause or run the queueing function.

https://github.com/storj/storj/issues/5232

Change-Id: I8bb4b4e55cf98b1aac9f26307e3a9a355cb3f506
2023-01-03 08:52:01 -07:00