Commit Graph

279 Commits

Author SHA1 Message Date
Vitalii
cd9518a3c3 satellite/{accounting, console}: remove unnecessary fields from usage report
Removed MetadataSize, RepairEgress, AuditEgress fields from usage report because they are not relevant.

Issue:
https://github.com/storj/storj/issues/6498

Change-Id: I8ef7d56bd1b01cdb53cec8c67dd47a6a0e7fa184
2023-11-15 22:31:29 +00:00
Vitalii
f6e357be52 satellite/{console, web}: detailed usage report for a single project
Reworked usage report endpoint to return CSV for a single OR all the project user owns.
Added buttons to download usage report CSV for a single project.

Issue:
https://github.com/storj/storj/issues/6154

Change-Id: I55104088180dcf6be49dcde6c9c495f07ba01c5a
2023-11-03 14:58:18 +02:00
Vitalii
e3713fddb8 satellite/{console, web}: added detailed usage report
Allow user to download detailed usage report from Billing -> Overview screen.
Report is a CSV file containing usage data for all the projects user owns.

Issue:
https://github.com/storj/storj/issues/6154

Change-Id: I3109002bf37b1313652a2be3447aaa7bc6204887
2023-10-16 21:54:18 +00:00
Egon Elbre
0335697664 satellite/accounting/live: ensure we don't panic when we get nil
Change-Id: I99b6b94d37a9856e8a705679d117b42d16326e81
2023-08-10 19:41:57 +03:00
Jeremy Wharton
a00ec7af40 satellite/console: create http endpoint for getting bucket usage totals
This change introduces an HTTP endpoint for retrieving bucket usage
totals. In the future, this will replace its GraphQL counterpart.

References #6141

Change-Id: Ic6a0069a7e58b90dc2b6c55f164393f036c6acf4
2023-08-08 21:26:17 +00:00
Michal Niewrzal
7be844351d satellite/metainfo: remove ServerSideCopyDuplicateMetadata
https://github.com/storj/storj/issues/5891

Change-Id: Ib5440169107acca6e832c2280e1ad12dfd380f28
2023-08-08 12:15:10 +00:00
Michal Niewrzal
03f8ad323d satellite/metabase: remove segment_copies support from ListSegments
We don't need to support segment copies with references anymore.
We migrated to copies where all metadata are copied from original
segment to copy.

https://github.com/storj/storj/issues/5891

Change-Id: Ic91dc21b0386ddf5c51aea45530024cd463e8ba9
2023-08-08 11:21:08 +00:00
Vitalii
6d94d6a681 satellite/{console, accounting}: fix project bandwidth calculation which is shown on project dashboard
Another try to fix calculation of used bandwidth which is displayed on Project Dashboard.
This change sums up allocated-dead traffic for the last 3 days and settled traffic for the period which is earlier than 3 days ago.

Issue:
https://github.com/storj/storj-private/issues/293

Change-Id: I91e652eba69f81bd21e0d053ac170e2b926b3cb4
2023-07-26 06:14:48 +00:00
Michal Niewrzal
73a279235a satellite/accounting/live: get project totals in batches
After infrastructure changes redis instance is not neccessay close
to core instance (where tally is calculated) and round trips to get
data from redis can be very costly. From less than hour calculation can take few hours for larger satellite.

This change combines 'segment' and 'storage' usage requests into
batches to reduce latency impact on tally calculation.

https://github.com/storj/storj/issues/5800

Change-Id: I87e57ec09e88fd167060a4ed51dc8b0274a095c5
2023-07-24 14:13:04 +00:00
Egon Elbre
7d1031feda satellite/accounting: fix duplicate code-block
Change-Id: Ic6de7337a5ff9a2fec73aa3d8a74ba5b9076a8c4
2023-07-20 07:58:53 +00:00
Egon Elbre
23631dc8bb satellite/accounting: fix TestProjectSegmentLimit*
Tally ensures that live accounting has the latest information,
however, when there are concurrent updates to live-accounting
it may by off by a few segments. Disable tally for those tests.

Change-Id: I6fa8a1794334bba093e18f29cb76e7b8d1244979
2023-07-18 14:26:05 +00:00
Vitalii
8d8f6734de satellite/{db, accounting}: added functionality to query settled bandwidth for given project
Added functionality to return only settled traffic from project_bandwidth_daily_rollups table for given month.
Updated {projectID}/usage-limits endpoint to return only settled bandwidth used.

This is a possible fix for this issue
https://github.com/storj/storj-private/issues/293

Change-Id: I12516dc898f449c2122e7442b8fbb88309a48ebe
2023-06-30 13:24:16 +03:00
Egon Elbre
049953a7ce go.mod: bump storj.io/uplink
Change-Id: Ib535cb50195c5a0e6197e83e2e05865ba95bcaf8
2023-06-26 13:51:13 +00:00
Michal Niewrzal
2b2bca8e81 satellite/accounting/tally: save tallies in a batches
Because we are saving all tallies as a single SQL statement we finally
reached maximum message size. With this change we will call SaveTallies multiple times in batches.

https://github.com/storj/storj/issues/5977

Change-Id: I0c7dd27779b1743ede66448fb891e65c361aa3b0
2023-06-22 17:02:26 +00:00
Michal Niewrzal
aea3baf6a9 satellite/accounting/tally: calculate pending object count
Change-Id: I4ee6072f4c60fafd809e8184ada9c1abf7edd8aa
2023-06-09 10:15:27 +00:00
Jeremy Wharton
c3d72a269e satellite/{accounting,satellitedb}: create tallies for empty buckets
Tallies are now created for buckets with no objects. Previously, the
bucket tally collector skipped empty buckets since it created tallies
only using information from the objects table. Methods that used
bucket tallies when calculating usage costs would return incorrect
results because of this.

Change-Id: I0b37fe7159a11cc02a51562000dad9258555d9f9
2023-05-25 19:48:59 +00:00
Jeremy Wharton
d8f64326f5 satellite/accounting/tally: remove objects loop
This change removes the use of the objects loop for calculating bucket
tallies. It has been superseded by a custom query.

Change-Id: I9ea4633006c9af3ea14d7de40871639e7b687c22
2023-05-22 15:18:03 -05:00
Michal Niewrzal
4bdbb25d83 satellite/metabase/rangedloop: move Segment definition
We will remove segments loop soon so we need first to move
Segment definition to rangedloop package.

https://github.com/storj/storj/issues/5237

Change-Id: Ibe6aad316ffb7073cc4de166f1f17b87aac07363
2023-05-16 12:37:17 +00:00
Michal Niewrzal
36e046375c satellite/repair/checker: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I8583549973cd36aa0e0c482c20d7a75cb7568ab3
2023-05-08 12:19:13 +00:00
Michal Niewrzal
1aa24b9f0d satellite/audit: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I9cec0ac454f40f19d52c078a8b1870c4d192bd7a
2023-04-24 15:52:11 +00:00
Michal Niewrzal
6a55682bc6 satellite/accounting/nodetally: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I6176a129ba14cf83fb635048d09e6748276b52a1
2023-04-24 14:25:53 +00:00
Egon Elbre
f40a0cb7ba satellite/*: use typed lrucache and ReadCache
Change-Id: Ieee535dd8735a95dd196a77413e4a25a6a72342c
2023-04-21 10:49:08 +00:00
Jeff Wendling
df9cc4d10f all: bump storj.io/{common,uplink,drpc}
Change-Id: Ie98a7eacc11a492a20c76636f8044c81b7ed580d
2023-04-17 19:59:30 +00:00
Michal Niewrzal
c4c391e154 satellite/accounting/live: replace address parsing with redis util
With this change we are replacing parsing code with existing go-redis
util.

We also switch redis client to version 9.

Change-Id: Ie4a651e3ae6960e68958c690873925d319b70e10
2023-04-05 13:20:11 +00:00
Michal Niewrzal
4c05293d8b satellite: set names for lrucache metrics
Fixes https://github.com/storj/storj/issues/5685

Change-Id: I9c19b7644b0cd3297448982b920c97abcdce9119
2023-04-04 11:55:20 +00:00
Michal Niewrzal
4db3d33d3e satellite/accounting: check bucket existance with GetProjectObjectsSegments
Check if project have any buckets before trying to get stats from
bucket_storage_tallies table.

Change-Id: If87159e29236a1816c3fb99c2d889ff735737100
2023-04-03 20:06:11 +00:00
Lizzy Thomson
d8a4ca62b8 satellite/accounting: update GetProjectSegmentLimit to fix bug
Update GetProjectSegmentLimit to reflect segment limit instead of
usage limit.

Fixes https://github.com/storj/storj/issues/5713

Change-Id: If4dfc5b45227595f3ae2b3ac7d267ea541fd748a
2023-03-31 11:47:21 +00:00
paul cannon
ae5947327b satellite/accounting: Use metabase.AliasPiece with tally observer
We want to eliminate usages of LoopSegmentEntry.Pieces, because it is
costing a lot of cpu time to look up node IDs with every piece of every
segment we read.

In this change, we are eliminating use of LoopSegmentEntry.Pieces in the
node tally observer (both the ranged loop and segments loop variants).
It is not necessary to have a fully resolved nodeID until it is time to
store totals in the database. We can use NodeAliases as the map key
instead, and resolve NodeIDs just before storing totals.

Refs: https://github.com/storj/storj/issues/5622
Change-Id: Iec12aa393072436d7c22cc5a4ae1b63966cbcc18
2023-03-29 12:24:05 +00:00
Michal Niewrzal
ee720040c9 satellite/metainfo: use project limit cache with limiter
Metainfo needs to know rate and burst limit to be able to limit users
requests. We made cache for per project limiter but to make single
instance we need to know about limits. So far we were doing direct DB
call to get rate/burst limit for project but it's generating lots of
DB requests and can be easily cached as we even have project limit cache.

This change extends project limit cache with rate/burst limit and starts
using this change while creating project limiter instance for metainfo.

Because data size kept in project limit cache is quite small this change
also bumps a bit default capacity of the cache.

Fixes https://github.com/storj/storj/issues/5663

Change-Id: Icb42ec1632bfa0c9f74857b559083dcbd054d071
2023-03-14 08:11:11 +00:00
Michal Niewrzal
e181f4b90e satellite/accounting: test that project limit cache is really used
We just fixed case were project limit cache was not used properly. This
is test case to cover that fix.

Change-Id: Iee467f0a46836860a14ab6238a9842ffbf54ed4c
2023-03-13 17:40:55 +00:00
Michal Niewrzal
0696256769 satellite/accounting: fix project limit cache usage
It looks that at some point we broke how project limits cache is used
and we were missing cache in most critical paths (upload/download).
This is fix for this issue.

I also adjusted cache methods naming.

Change-Id: Ic98372779a39365d0920fe3943f1f7a68b064173
2023-03-10 14:42:38 +00:00
Jeremy Wharton
cbbd5ab1ef satellite/payments/stripecoinpayments: undo price override removal
Commit fb59974 disabled usage price overrides because of a failing
test. This change reenables it while resolving the issue that caused
the test to fail.

The previous version of the test passed Gerrit verification and was
merged, but it failed for the primary Jenkins pipeline after merge.

This is due to a difference in how the Jenkins build runs Cockroach
and Postgres for each pipeline.

This commit rewrites the test to be safe for concurrent execution by
ensuring any mutable variables are defined within each test so that
shared state across tests is reduced.

Change-Id: Ia4566c9cd2d698afdb2caa4b7e2808b17e18de4e
2023-02-24 16:54:51 +00:00
Jeremy Wharton
fb5997484e satellite/payments/stripecoinpayments: remove usage price overriding
Project usage price overriding has been removed because it produces
incorrect results when tested. It should not be re-implemented until
the issues it causes are resolved.

Change-Id: Ic92eff374c9af4fea3bf32782a72303a7978b055
2023-02-23 13:36:59 +00:00
Michal Niewrzal
16b7901fde satellite/metabase: add piece size calculation to segment
This code is essentially replacement for eestream.CalcPieceSize. To call
eestream.CalcPieceSize we need eestream.RedundancyStrategy which is not
trivial to get as it requires infectious.FEC. For example infectious.FEC
creation is visible on GE loop observer CPU profile because we were
doing this for each segment in DB.

New method was added to storj.Redundancy and here we are just wiring it
with metabase Segment.

BenchmarkSegmentPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize-8         	    5822	    189189 ns/op	    9776 B/op	       8 allocs/op
BenchmarkSegmentPieceSize/segment.PieceSize
BenchmarkSegmentPieceSize/segment.PieceSize-8              	94721329	        11.49 ns/op	       0 B/op	       0 allocs/op

Change-Id: I5a8b4237aedd1424c54ed0af448061a236b00295
2023-02-22 11:04:02 +00:00
Michal Niewrzal
aba2f14595 satellite/metabase/rangedloop: few additions for monitoring
Additional elements added:
* monkit metric for observers methods like Start/Fork/Join/Finish to
be able to check how much time those methods are taking
* few more logs e.g. entries with processed range
* segmentsProcessed metric to be able to check loop progress

Change-Id: I65dd51f7f5c4bdbb4014fbf04e5b6b10bdb035ec
2023-02-17 08:46:00 +00:00
Lizzy Thomson
155b927c5e satellite/console: add notifications for free account limits
Add notifications for free account limits for segment usage
and update to follow the figma designs.

Issue: https://github.com/storj/storj/issues/5482

Change-Id: I8a2fe38d609d53e09bf5074484cedc343223bffd
2023-02-16 13:31:01 -07:00
Michal Niewrzal
0b519fbd05 satellite/satellitedb: create index for interval_day for project_bandwidth_daily_rollup
To improve deletion of old entries in project_bandwidth_daily_rollup
we need index on `interval_day` column which is used to find those old
entries.

As an addition we are changing interval how often deletion is executed
from 7 to 1 day. We would like to have smaller portion of data to
delete.

Fixes https://github.com/storj/storj/issues/5465

Change-Id: Ie18ebe859887b93d6e4e6065a61fb9214c7ad27a
2023-02-03 10:02:58 +00:00
Jeremy Wharton
091ed29935 satellite/payments/stripecoinpayments: make price overrides per-bucket
This change causes the bucket's partner info to be used rather than the
user's when calculating project usage prices. This ensures that users
who own differently-partnered buckets will be charged correctly for
usage based on the specific bucket they are utilizing.
according to the bucket's partner.

Related to storj/storj-private#90

Change-Id: Ieeedfcc5451e254216918dcc9f096758be6a8961
2023-02-01 23:17:25 +00:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
Yaroslav Vorobiov
5644fb1a7e satellite/accounting/nodetally: add ranged loop
Add node tally ranged loop observer and partial.
Add node tally randed observer to range loop peer.
Add config flag to select which loop to use for node tally.
Update satellite core to use segement/ranged loop based on a flag.
Duplicate existing node tally test but using ranged loop.

Change-Id: I6786f1a16933463fab5f79601bf438203a7a5f9e
2023-01-17 13:50:18 +01:00
Michal Niewrzal
a2a9dafa33 satellite/orders: don't store allocated bandwidth in
bucket_bandwidth_rollups table

We have performance problems with updating bucket_bandwidth_rollups. To
improve situation we can stop storing allocated bandwidth in this table.
This should reduce large number of updates which are comming from
metainfo endpoints, repair workers and audit.

Next step will be to drop `allocated` column completely from
bucket_bandwidth_rollups.

Allocated GET bandwidth is all we need and we are keeping it in
bucket_bandwidth_rollups table.

Change-Id: Ifdd26a89ba8262acbca6d794a6c02883ad0c0c9b
2023-01-12 13:21:02 +00:00
Michal Niewrzal
8e030f4e30 satellite/accounting/tally: fix looping over all buckets
We have a bug where if number of buckets in the system will be
multiplication of batch size (2500) then loop that is going over
all buckets can run indefinitely.

Fixes https://github.com/storj/storj/issues/5374

Change-Id: Idd4d97c638db83e46528acb9abf223c98ad46223
2022-12-07 10:32:23 +00:00
Michal Niewrzal
d5eea2db61 satellite/accounting: use custom query for bucket tally by default
We added alternative way to calculate bucket tallies for accounting and
now it's tested and we will enable it by default.

CollectBucketTallies was extended to support overriding current time
to be able to test handling expired objects.

Change-Id: I738b99a33fd2e086245f92d874c1cbb806e834c0
2022-11-22 10:23:40 +00:00
Michal Niewrzal
9253e12c5a satellite/accounting: add missing monitoring
Change-Id: I6ffbc786c99adc2b1a735f4583d2b9f5162fe7a5
2022-11-16 12:02:03 +00:00
Cameron
74ddfab810 satellite/overlay: insert DQ event into node events in overlay.DisqualifyNode
Also, return node email from overlaycache db DisqualifyNode to be used
in node events insertion

Change-Id: I41534cf01351c1690c3966a8055c5fe6fcf0d6a6
2022-11-04 15:18:31 +00:00
Qweder93
5e5d6ecf6c satellite/{accounting, buckets}: added feature that allows to use custom SQL query instead of object loop for buckets accounting
since amount of objects is growing and looping through all of them
starts taking lot of time, we are switching for SQL query to do it
in chunks of tallies per bucket. 2nd part of issue fix.

Closes https://github.com/storj/team-metainfo/issues/125

Change-Id: Ia26bcac0a7e2c6503df9ebbf4817a636841d3284
2022-11-02 09:01:33 +00:00
Michal Niewrzal
c63a401399 satellite/accounting/tally: bring back monkit call
We removed monkit call from "object" method because it was using
too much cpu and was visible on cpu profile but we should first try
optimized version of this call.

Change-Id: Ib76d8a2968a704ce47235c6dac6edad4e40bde48
2022-10-19 02:18:14 +00:00
Michal Niewrzal
bbed10d7d4 satellite/{accounting/tally,metabase}: remove monkit for fast methods
This is another change to remove monkit calls from fast methods. Those
calls are visible in CPU profiles.

Change-Id: Ib3beba0dca6a6d93c3342b0994c580f78bbdd50b
2022-10-05 08:38:48 +00:00
Michal Niewrzal
91c3a45874 satellite/accounting/nodetally: remove monkit call from observer
Benchmark against 'main':
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    5.56µs ± 5%    0.69µs ±12%   -87.57%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8    2.72kB ± 0%    0.00kB           ~     (p=0.079 n=4+5)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      50.0 ± 0%       0.0       -100.00%  (p=0.008 n=5+5)

Change-Id: I20527fb576cd81db667a81929fa95b810ee11b14
2022-10-03 13:32:20 +00:00
Ethan
77357f9877 satellite/{accounting,redis}: Use ARGV in redis scripts
While investigating high memory consumption on a Redis instance, it was found that much of the memory consumed was from cached redis scripts.

Redis caches scripts based on the hash of the script itself.  This change uses ARGV to reduce the number of cached scripts.

Change-Id: Ia878fe81552a3067f09e60c44bb4ace25c6b5f9a
2022-09-28 14:43:00 +00:00