Commit Graph

242 Commits

Author SHA1 Message Date
Jeremy Wharton
091ed29935 satellite/payments/stripecoinpayments: make price overrides per-bucket
This change causes the bucket's partner info to be used rather than the
user's when calculating project usage prices. This ensures that users
who own differently-partnered buckets will be charged correctly for
usage based on the specific bucket they are utilizing.
according to the bucket's partner.

Related to storj/storj-private#90

Change-Id: Ieeedfcc5451e254216918dcc9f096758be6a8961
2023-02-01 23:17:25 +00:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
Yaroslav Vorobiov
5644fb1a7e satellite/accounting/nodetally: add ranged loop
Add node tally ranged loop observer and partial.
Add node tally randed observer to range loop peer.
Add config flag to select which loop to use for node tally.
Update satellite core to use segement/ranged loop based on a flag.
Duplicate existing node tally test but using ranged loop.

Change-Id: I6786f1a16933463fab5f79601bf438203a7a5f9e
2023-01-17 13:50:18 +01:00
Michal Niewrzal
a2a9dafa33 satellite/orders: don't store allocated bandwidth in
bucket_bandwidth_rollups table

We have performance problems with updating bucket_bandwidth_rollups. To
improve situation we can stop storing allocated bandwidth in this table.
This should reduce large number of updates which are comming from
metainfo endpoints, repair workers and audit.

Next step will be to drop `allocated` column completely from
bucket_bandwidth_rollups.

Allocated GET bandwidth is all we need and we are keeping it in
bucket_bandwidth_rollups table.

Change-Id: Ifdd26a89ba8262acbca6d794a6c02883ad0c0c9b
2023-01-12 13:21:02 +00:00
Michal Niewrzal
8e030f4e30 satellite/accounting/tally: fix looping over all buckets
We have a bug where if number of buckets in the system will be
multiplication of batch size (2500) then loop that is going over
all buckets can run indefinitely.

Fixes https://github.com/storj/storj/issues/5374

Change-Id: Idd4d97c638db83e46528acb9abf223c98ad46223
2022-12-07 10:32:23 +00:00
Michal Niewrzal
d5eea2db61 satellite/accounting: use custom query for bucket tally by default
We added alternative way to calculate bucket tallies for accounting and
now it's tested and we will enable it by default.

CollectBucketTallies was extended to support overriding current time
to be able to test handling expired objects.

Change-Id: I738b99a33fd2e086245f92d874c1cbb806e834c0
2022-11-22 10:23:40 +00:00
Michal Niewrzal
9253e12c5a satellite/accounting: add missing monitoring
Change-Id: I6ffbc786c99adc2b1a735f4583d2b9f5162fe7a5
2022-11-16 12:02:03 +00:00
Cameron
74ddfab810 satellite/overlay: insert DQ event into node events in overlay.DisqualifyNode
Also, return node email from overlaycache db DisqualifyNode to be used
in node events insertion

Change-Id: I41534cf01351c1690c3966a8055c5fe6fcf0d6a6
2022-11-04 15:18:31 +00:00
Qweder93
5e5d6ecf6c satellite/{accounting, buckets}: added feature that allows to use custom SQL query instead of object loop for buckets accounting
since amount of objects is growing and looping through all of them
starts taking lot of time, we are switching for SQL query to do it
in chunks of tallies per bucket. 2nd part of issue fix.

Closes https://github.com/storj/team-metainfo/issues/125

Change-Id: Ia26bcac0a7e2c6503df9ebbf4817a636841d3284
2022-11-02 09:01:33 +00:00
Michal Niewrzal
c63a401399 satellite/accounting/tally: bring back monkit call
We removed monkit call from "object" method because it was using
too much cpu and was visible on cpu profile but we should first try
optimized version of this call.

Change-Id: Ib76d8a2968a704ce47235c6dac6edad4e40bde48
2022-10-19 02:18:14 +00:00
Michal Niewrzal
bbed10d7d4 satellite/{accounting/tally,metabase}: remove monkit for fast methods
This is another change to remove monkit calls from fast methods. Those
calls are visible in CPU profiles.

Change-Id: Ib3beba0dca6a6d93c3342b0994c580f78bbdd50b
2022-10-05 08:38:48 +00:00
Michal Niewrzal
91c3a45874 satellite/accounting/nodetally: remove monkit call from observer
Benchmark against 'main':
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    5.56µs ± 5%    0.69µs ±12%   -87.57%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8    2.72kB ± 0%    0.00kB           ~     (p=0.079 n=4+5)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      50.0 ± 0%       0.0       -100.00%  (p=0.008 n=5+5)

Change-Id: I20527fb576cd81db667a81929fa95b810ee11b14
2022-10-03 13:32:20 +00:00
Ethan
77357f9877 satellite/{accounting,redis}: Use ARGV in redis scripts
While investigating high memory consumption on a Redis instance, it was found that much of the memory consumed was from cached redis scripts.

Redis caches scripts based on the hash of the script itself.  This change uses ARGV to reduce the number of cached scripts.

Change-Id: Ia878fe81552a3067f09e60c44bb4ace25c6b5f9a
2022-09-28 14:43:00 +00:00
Qweder93
04411e6871 satellite/accounting: use single call to get project limits
segment and usage limits now taken by single call

Closes https://github.com/storj/team-metainfo/issues/124

Change-Id: I55e05e199d2575cb8d985f4d75ba061002c97f0f
2022-09-13 21:50:52 +00:00
Michal Niewrzal
6cc2052f47 satellite: fix segment loop observers metrics
We made optimization for segment loop observers to avoid
heavy monkit initialization on each call. It was applied to very
often executed methods. Unfortunately we used wrong monkit
method to track function times. Instead mon.Task we used
mon.Func().

https://github.com/spacemonkeygo/monkit#how-it-works

Change-Id: I9ca454dbd828c6b43ba09ca75c341991d2fd73a8
2022-08-10 14:13:16 +00:00
Clement Sam
d1d15cbf4f satellite: return interval_end_time in DailyStorageUsage endpoint
Updates https://github.com/storj/storj/issues/4178

Change-Id: If0c6f66c58a258300bbf563eef56caba4f1dc626
2022-07-27 18:24:27 +00:00
Clement Sam
f2f046af7a satellite/satellitedb: add interval_end_time column to accounting_rollups table
Adding an interval_end_time column to the accounting_rollups table
to keep the last interval_end_time for each daily storage tallies.

Updates https://github.com/storj/storj/issues/4178

Change-Id: If7a8210c5e9fe2fc9df84b137a8b6e3db2471c58
2022-07-27 17:53:27 +00:00
Qweder93
2b24632610 {satellite/metainfo, satellite/accounting}: remove ValidateSegmentLimit config value and check
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now

Resolves: https://github.com/storj/storj/issues/4470

Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
2022-07-27 15:36:53 +00:00
Erik van Velzen
b5fc04af89 satellite/metainfo: usage limits for copy
Previously there was no realtime administration of the storage usage
during copies. Now there is.

Closes https://github.com/storj/storj/issues/4719

Change-Id: I0d536bf551d16208116c3aceac89ed590ec473bf
2022-07-25 12:35:27 +02:00
Egon Elbre
82fede2132 satellite/satellitedb: batch delete storage node tallies
Currently we have a significant number of tallies that need to be
deleted together. Add a limit (by default 10k) to how many will
be deleted at the same time.

Change-Id: If530383f19b4d3bb83ed5fe956610a2e52f130a1
2022-07-20 15:42:35 +03:00
Egon Elbre
05e165283f storagenode/console/consoleapi: use fixed time.Now()
It seems the tests relied on time.Now(), which might cause some
discrepancies in calculations. Use a fixed time.Now() rather than
recalculating.

As a sidefix, remove "Test" prefix from t.Run. These are unnecessary.

Change-Id: I1de903fcf0fcf46fc8e3acf2463e17239b8e3cc6
2022-07-01 12:36:01 +03:00
Ivan Fraixedes
6e8e86a0b7 satellite/accounting: Fix typos in doc comments
Change-Id: I497bebe152848e023d927337c61e3eee73b87697
2022-06-22 14:45:21 +00:00
Ivan Fraixedes
f626cd1868 satellite/accounting: Correctly cache project bandwidth
Satellite caches the project bandwidth in Redis when it doesn't have it
because was not set or the key expired, however, it doesn't perform the
check and set if not exists in a transaction. It also uses the increase
function which increases the value if it exists otherwise it sets it.

This provokes that multiple concurrent request to the same project may
increase the total project by multiples of the bandwidth usage
registered in the database rather than setting it because they may check
if the key exists before any other has executed the increase and then
the first one executing it will set the value but the others will
increased causing that Redis has a wrong bandwidth usage value which is
N magnitude of the real one and making the satellite to deny the
downloading if it surpasses the project limit.

This commit changes the "update"" project bandwidth usage by an "insert"
but using a Redis function that only sets the value if the key doesn't
exists for solving the increase issue but also not overriding the value
due to may contain updates of other downloading requests which aren't
already registered in the DB.

Change-Id: I33e2fe462930b2fdb4061fc94002bd3544476f94
2022-06-21 08:31:41 +00:00
Márton Elek
35c022ebf9 satellite/accounting: don't use testplanet in rollup_test if not necessary
TestRollupNoDeletes is very flaky (passes locally but fails in the main branch build).

The exact reason is not clear, but stopping the loop seems to be async, the following lines may not stop the loops immediatelly which is a potential problem:

```
satellitePeer.Accounting.Rollup.Loop.Pause()
satellitePeer.Accounting.Tally.Loop.Pause()
```

Fortunatelly these test check only the database interfaces. Instead of testplanet.Run we can run only satellitedbtest.Run which is faster and more predictable (no background loops).

Other potential problem: comment claims that the default of DeleteTallies is false:

```
// In testplanet the setting config.Rollup.DeleteTallies defaults to false.
```

But it seems to be true (rollup.go):

```
DeleteTallies bool          `help:"option for deleting tallies after they are rolled up" default:"true"`
```

This is also fixed in the patch (as we need set it explicit), but TBH it can be fixed with testplanet, too.

Change-Id: Id7ec80d5c069bed2c556f4d001c71aa23fc5af23
2022-06-09 13:11:45 +00:00
Michał Niewrzał
18ef3d3881 satellite/accounting: add metrics for total pending objects
We don't have metric to track how many pending objects we have in the
system. This change is using tally objects loop to collect pending
objects per bucket and at the end its combining all buckets values
into single metric for all pending objects in a system.

Change-Id: Iac7a6bfb48854f7e70127d275ea8fdd60c4eb8b7
2022-06-06 12:04:46 +00:00
Michał Niewrzał
7a2d2a36ca satellite: use more optimal monkit call for loop observers methods
Recently we applied this optimization to metrics observer and time
used by its method dropped from 12m to 3m for us1 (220m segments).
It looks that it make sense to apply the same code to all observers.

Change-Id: I05898aaacbd9bcdf21babc7be9955da1db57bdf2
2022-05-20 11:03:41 +00:00
Qweder93
50b826e582 satellite/metabase: drop GetProjectSegmentCount and related code
Resolves: https://github.com/storj/storj/issues/4744

Change-Id: I66b12e17de461242745e785dedd397fbe3b386fa
2022-05-17 19:01:51 +00:00
Qweder93
a447b27ffc satellite/accounting: modify tally.go to use returned segments values
modify tally to calculate how we need to update segments in the live accounting
cache with UpdateProjectSegmentUsage method. adjust accounting.ExceedsUploadLimits
to use only cache for segment validation, if cache returns 0 or key not found then
we shouldn't reject such project as its possible that we won't have this value before first object iteration
https://github.com/storj/storj/issues/4744

Change-Id: I32c22d7fb71236e354653ba8719e029fc71f04c7
2022-05-17 17:20:02 +00:00
Michał Niewrzał
7b253075bb satellite/accounting: test if server-side copy affects bandwidth
Doing server-side copy operation should not affect user monthly
bandwidth. This test covers that case.

Fixes https://github.com/storj/storj/issues/4717

Change-Id: I84ffab96b84851f395ea3a34d88f7dba424ec440
2022-05-17 06:24:15 +00:00
Michał Niewrzał
277f8d2ae0 satellite/accounting: don't expire segments usage key
Initialy we wanted to put segment usage into cache values retrieved
directly from metabase but it cause performance issues. Now we
will be collecting segment usage from tally object loop and those
values will be put into a cache but becuse we cannot get those values
on demand we shouldn't expire cache value at all because objects
loop requires sustencial amount of time to be executed.

Part of solution for https://github.com/storj/storj/issues/4744

Change-Id: I3b37e23badeecebed0c95064156e85b38038bfe2
2022-05-16 13:00:16 +00:00
Vitalii
95ae739d9c satellite/projectaccounting: fetch bucket usage since bucket creation instead of project creation
Fix for this customer issue
https://github.com/storj/customer-issues/issues/34

By this change we fetch bucket usage since its creation instead of using project's createdAt timestamp.

Change-Id: Ic0ea5d169056a5bd64ed143d13954d794da6e1d2
2022-05-13 13:27:42 +00:00
Qweder93
a3c9ca653f satellite/accounting: GetAllProjectTotals returns storage and segment totals combined
return storage and segment totals as a single result, instead of returning only storage
and bandwidth and segment values are filtered out, https://github.com/storj/storj/issues/4744

Change-Id: I624d67ed5205ae21ecd5a2f39775f63ed042e629
2022-05-05 13:38:33 +00:00
Michał Niewrzał
6e5a94698e satellite/metabase: add option to override metadata with CommitObject
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.

https://github.com/storj/team-metainfo/issues/105

Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
2022-04-26 08:03:52 +00:00
Qweder93
7e6618cfad satellite/accounting: test tally on copied object flaky fix
Resolves: https://github.com/storj/storj/issues/4732

Change-Id: I2abf50aafc62dd97be12b0c0573db2f95d560796
2022-04-21 10:30:17 +00:00
Yaroslav Vorobiov
3f47d19aa6 satellite/overlay: add disqualification reason
Add disqualification reason to NodeDossier.
Extend DB.DisqualifyNode with disqualification reason.
Extend reputation Service.TestDisqualifyNode with disqualification reason.

Change-Id: I8611b6340c7f42ac1bb8bd0fd7f0648ad650ab2d
2022-04-20 13:29:31 +00:00
Michał Niewrzał
6188c4c0dd satellite/metainfo: enable segment limit by default
Change-Id: I26cc0bbd97d6cc2ffc96213788f261efb749d028
2022-04-14 06:12:09 +00:00
Qweder93
8e0196a6cc satellite/accounting: test accounting/tally with object copies
We implemented server-side copy feature and we would like to
confirm that it is not affecting accounting/tally service.

Resolves https://github.com/storj/storj/issues/4697

Change-Id: I3944ea52c0acc68107ec15c1911750dc7d947501
2022-04-13 09:16:43 +00:00
Vitalii Shpital
1245283637 apigen: new endpoint to get project's single bucket rollup
Added new endpoint to get project's single bucket usage rollup.
Extended generation code to handle service method args.

Change-Id: Ief768632a801c047c66e0617056fbd7b30427b33
2022-03-04 17:33:38 +00:00
Vitalii Shpital
9b5904cd49 satellite/{projectaccounting, console}:query to get single bucket rollup
Added new projectaccounting query to get project's single bucket usage rollup.
Added new service method to call new query.
Added implementation for IsAuthenticated method which is used by new generated API.

Change-Id: I7cde5656d489953b6c7d109f236362eb465fa64a
2022-03-03 12:04:29 +00:00
Vitalii Shpital
150be885b7 satellitedb/projectaccounting, web/satellite: reworked bandwidth chart to show both allocated and settled bandwidth
Extended DB query.
Reworked charts UI functionality to show 2 lines if necessary.

Change-Id: I8ac4e4fa07676fc9fa7e9c078ecdeed62233b8e2
2022-02-08 22:07:59 +00:00
Vitalii Shpital
b3e1be37ff satellite/projectaccounting: query to get daily project usage by date range
Finished implementing queries for both bandwidth and storage using pgx.Batch.
Fixed CSP styling issue.

Change-Id: I5f9e10abe8096be3115b4e1f6ed3b13f1e7232df
2022-02-04 22:32:37 +00:00
Michał Niewrzał
d393f6094c satellite/accounting: adjust tests to expose new uplink errors
Existing tests are checking errors by comparing error message.
We plan to expose storage/segment limit errors as a public uplink
API but before that will happen we need to update tests to handle
both cases as new uplink error will have a bit different error message
than current tested error.

https://github.com/storj/uplink/issues/77

Change-Id: Id2706323c60d050d96752e66e859d4ec051a69b9
2022-01-27 15:15:07 +00:00
Vitalii Shpital
b8c55fdd87 satellite/projectaccounting, satellite/console, web/satellite: implemented backend for bandwidth chart
Implemented endpoint and query to get bandwidth chart data for new project dashboard.
Connected backend with frontend.
Storage chart data is mocked right now.

Change-Id: Ib24d28614dc74bcc31b81ee3b8aa68b9898fa87b
2022-01-26 15:09:22 +00:00
Michał Niewrzał
9ce1a4e25d satellite/accounting: use AOST to get segment usage
For better performance we should use AOST for getting
data where being up to data it not very crucial. We don't
care about small differences for segment limit calculation.

Change-Id: I9b2d4f2bd15ebc9d1c46bc84dd51a2e9d9231506
2022-01-25 11:00:08 +00:00
Michał Niewrzał
913718ef97 satellite/accounting: test bandwidth limit without cache
Additional test case to cover situation when cache value
expired and we need to get this value directly from DB.

Change-Id: I07bda67e19333e09a567104ce70f112fd47a7845
2022-01-24 13:24:50 +00:00
Michał Niewrzał
4be84c3a6c satellite/accounting: fix how we put segments usage into cache
We had two problems here. First was how we were handling
errors message from GetProjectSegmentUsage. We were always
returning error, even for ErrKeyNotFound which should be used
to refresh value in cache and not propagated out of method.
Second, we were using wrong value to update cache. We used
current value from cache which obviously it's not what we intend.

Fixes https://github.com/storj/storj/issues/4389

Change-Id: I4d7ca7a1a0a119587db6a5041b44319102ef64f8
2022-01-24 11:57:50 +00:00
Qweder93
228f465d45 satellite/metainfo: сombine checks for storage and segment limit
We need to combine methods from accounting.Service (ExceedsStorageUsage and ExceedsSegmentUsage)
to run checks concurrently.

Resolves https://github.com/storj/team-metainfo/issues/73

Change-Id: I47831bca92457f16cfda789da89dbd460738ac97
2022-01-19 10:44:04 +00:00
Qweder93
8be655c487 satellite/metainfo: add segment limit validation, caching
We want to be able to limit the number of segments per project for users.
To limit this we need to check limit value associated with project
and value of used segments already in BeginMoveObject, BeginMoveSegment
and increment cache segments usage after each CommitSegment call.

Resolves https://github.com/storj/team-metainfo/issues/1

Change-Id: I6290e67c095a174b9d101c4521802d9bfe0453b8
2022-01-10 14:33:26 +02:00
dlamarmorgan
ab37b65cfc satellite/{accounting,orders,satellitedb}: group bucket bandwidth rollups by time window
Batching of the order submissions can lead to combining the allocated
traffic totals for two completely different time windows, resulting
in incorrect customer accounting. This change will group the batched
order submissions by projectID as well as time window, leading to
distinct updates of a buckets bandwidth rollup based on the hour
window in which the order was created.

Change-Id: Ifb4d67923eec8a533b9758379914f17ff7abea32
2022-01-05 20:24:48 +00:00
Michał Niewrzał
000957f64a satellite/accounting: add functionality to get and cache project segment count
We want to set maximum number of segments per project. This change will
add functionality to get number of segments currently used by project.
To avoid often DB calls segment cound will be cached and refreshed every
few minutes.

Change-Id: I2ecb6484f5afc3875c0e0dfaea360e8872f9d196
2021-12-09 12:30:52 +00:00