Commit Graph

81 Commits

Author SHA1 Message Date
Jeremy Wharton
bc4cc4974d satellite/satellitedb: exclude end date when calculating project totals
This change modifies the method responsible for returning project
usage summaries such that the end date of the given time period
is excluded to prevent overlap.

Change-Id: If06155efff5c6fce3865f5f6e4344873abe3e432
2022-11-14 11:24:05 +00:00
Márton Elek
9ddd20b72e satellite/projectaccounting: fix project usage right after the genesis
After you create a brand new cluster (with storj-up, for example) the project usage fails during the first 5 minutes.

The problem is the usage of `AS OF SYSTEM TIME` which points to a time where the master database didn't exist.

In this specific case the database not found error can be ignored to avoid such messages. (if the database is really missing, we will have problems way more earlier, eg. at the login)

Change-Id: I51ee78994d91fc2a14b56646402faaaa8154c934
2022-10-04 14:41:30 +00:00
wilfredasomani
d8b010f7bd satellite/{web,satellitedb}: fix project daily usage error
The new dashboard currently gets stuck on loading and displays an error when
it fails to get usage data. Failure happens on satelliteDb due to a cockroach transaction error
caused by reading data before using AS OF SYSTEM TIME in the same transaction.
This change reverses the order of daily usage queries to avoid this error.
And hides the loaders on the dashboard if/when an error occurs.

see: https://github.com/storj/storj/issues/5012

Change-Id: I06b6ee434f72242f9b7d21dec7aaf39d1d622f1e
2022-08-10 19:18:28 +00:00
paul cannon
37a4edbaff all: reformat comments as required by gofmt 1.19
I don't know why the go people thought this was a good idea, because
this automatic reformatting is bound to do the wrong thing sometimes,
which is very annoying. But I don't see a way to turn it off, so best to
get this change out of the way.

Change-Id: Ib5dbbca6a6f6fc944d76c9b511b8c904f796e4f3
2022-08-10 18:24:55 +00:00
Vitalii
a61f0f6be3 satellite/projectaccounting: make daily usage query return correct allocated traffic
Updated daily project usage query to return correct allocated traffic.
If allocated egress has expired then we return settled egress.
If not then we return allocated egress - dead egress.

Fix for this issue
https://github.com/storj/storj/issues/4563

Change-Id: Ia15a50d3bb8d8cb1106936e17dbe0f1f5a40fa87
2022-06-01 15:24:52 +00:00
Vitalii
07e65cd338 satellite/projectaccounting: sum up bucket usages for daily usage query
Fixed daily usage query returning single bucket usage.
We sum up bucket usages now.
Also fixed https://github.com/storj/storj/issues/4559.

Change-Id: I2eb6299f1ef500d68150879195011b6fbb5f37ed
2022-06-01 12:50:10 +00:00
Vitalii
95ae739d9c satellite/projectaccounting: fetch bucket usage since bucket creation instead of project creation
Fix for this customer issue
https://github.com/storj/customer-issues/issues/34

By this change we fetch bucket usage since its creation instead of using project's createdAt timestamp.

Change-Id: Ic0ea5d169056a5bd64ed143d13954d794da6e1d2
2022-05-13 13:27:42 +00:00
Qweder93
7e6618cfad satellite/accounting: test tally on copied object flaky fix
Resolves: https://github.com/storj/storj/issues/4732

Change-Id: I2abf50aafc62dd97be12b0c0573db2f95d560796
2022-04-21 10:30:17 +00:00
Vitalii Shpital
9b5904cd49 satellite/{projectaccounting, console}:query to get single bucket rollup
Added new projectaccounting query to get project's single bucket usage rollup.
Added new service method to call new query.
Added implementation for IsAuthenticated method which is used by new generated API.

Change-Id: I7cde5656d489953b6c7d109f236362eb465fa64a
2022-03-03 12:04:29 +00:00
Vitalii Shpital
150be885b7 satellitedb/projectaccounting, web/satellite: reworked bandwidth chart to show both allocated and settled bandwidth
Extended DB query.
Reworked charts UI functionality to show 2 lines if necessary.

Change-Id: I8ac4e4fa07676fc9fa7e9c078ecdeed62233b8e2
2022-02-08 22:07:59 +00:00
Vitalii Shpital
b3e1be37ff satellite/projectaccounting: query to get daily project usage by date range
Finished implementing queries for both bandwidth and storage using pgx.Batch.
Fixed CSP styling issue.

Change-Id: I5f9e10abe8096be3115b4e1f6ed3b13f1e7232df
2022-02-04 22:32:37 +00:00
Vitalii Shpital
b8c55fdd87 satellite/projectaccounting, satellite/console, web/satellite: implemented backend for bandwidth chart
Implemented endpoint and query to get bandwidth chart data for new project dashboard.
Connected backend with frontend.
Storage chart data is mocked right now.

Change-Id: Ib24d28614dc74bcc31b81ee3b8aa68b9898fa87b
2022-01-26 15:09:22 +00:00
Michał Niewrzał
d94d8d1775 satellite/accounting: expose project segment limit
Exposes functionality to get and update project segment
limit. It will be used to limit number of segments per project
while uploading object.

Change-Id: I971d48eebb4e7db8b01535c3091829e73437f48d
2021-12-08 11:27:21 +00:00
Vitalii Shpital
b78f65e83b satellite/console, web/satellite: added object and segment count per project info to new project dashboard
Added new query to get project object and segment count.
Added appropriate object and segment count view for new project dashboard.

Change-Id: I69a2e55442f318c51dc365c0c578b964f2f06c7f
2021-12-01 17:46:07 +02:00
Yingrong Zhao
774ae017e3 satellite/{satellitedb, web}: display object count in satellite UI
Even though we want to start charging segment fee instead of object fee,
it's hard for users to understand what a segment is. This PR adds the
object count back in the UI alongside with segment count to help address
the issue.

Change-Id: I92eb42c769d350eba68a72443deffec5c278359c
2021-11-01 18:26:13 +00:00
Yingrong Zhao
d441c8da15 satellite: use segment count for billing
Change-Id: Iafaf41cb6cde4cb2abdac3f75ce0fead838cb758
2021-10-27 19:21:50 +00:00
Fadila Khadar
f77f61532a satellite/orders: use egress_dead for calculating allocated bandwidth
Change-Id: I8240d99d0cdaad4c5e059565e88ee9619b62526e
2021-10-11 14:58:26 +02:00
Fadila Khadar
fb0d055a41 satellites/orders: populate egress_dead in project_bandwidth_daily_rollups
Populate the egress_dead column for taking into account allocated bandwidth that can be removed because orders have been sent by the storage nodes. The bandwidth not used in these orders can be allocated again.

Change-Id: I78c333a03945cd7330aec052edd3562ec671118e
2021-10-06 16:54:49 +00:00
Michał Niewrzał
27a714e8b0 satellite/accounting/tally: use objects iterator instead metaloop
Bucket tally calculation will be removed from metaloop and will
use metabase objects iterator directly.

At the moment only bucket tally needs objects so it make no sense
to implement separate objects loop.

Change-Id: Iee60059fc8b9a1bf64d01cafe9659b69b0e27eb1
2021-07-20 15:52:18 +00:00
Egon Elbre
bf5194d134 satellite/satellitedb: optimize ProjectAccounting.ArchiveRollupsBefore
The previous query was making a full table scan. This modifies code to
do the queries separately on each action. It will probably be slower on
a small table, however there should be a several magnitude boost on
large tables.

Change-Id: Ib8885024d8a5a0102bbab4ce09bd6af9047930c9
2021-07-02 17:06:58 +00:00
Michał Niewrzał
0ca7583282 satellite/accounting: add total for bytes and segments to tallies
We want to calculate bucket tally only from iterating objects.
Object currently has an info about totals for bytes and segments.
We need to adjust tallies to keep those totals. Older entries will
be untouched and code will use totals only if available. Change
is adding columns for totals to bucket_storage_tally table and
is adding general handling for them.

Next step is to start using total columns instead of inline/remote.
This will be done with next change.

Change-Id: I37fed1b327789efcf1d0570318aee3045db17fad
2021-07-01 08:52:32 +00:00
Michał Niewrzał
aa7fd8c9cd satellite/satellitedb: fix ProjectAccounting project bandwidth calculation
Period end was calculated
incorrectly as it was still in current month but
should be the first day of next month.

Change-Id: I37451d29a9b901b69e6c3c401b333c58b3376d61
2021-06-30 14:54:12 +02:00
Michał Niewrzał
4267a958d4 satellite/satellitedb: use AS OF SYSTEM TIME for GetProjectBandwidth query
Adding AS OF SYSTEM TIME to query that is calculating project bandiwdth.
As an addition method for setting interval is added as test doesn't
work well with default interval.

Change-Id: Id1e15be4f6afff13b9dc2b7f595e2edb6de28db9
2021-06-28 15:19:47 +00:00
Fadila Khadar
3de9655b68 satellite/redis: used bandwidth key depends on day
The redis key associated to bandwidth usage depended on the current month.
This change makes it depends also on the day, so that we update bandwidth usage
daily to take into account changes associated to expired but not used allocated bandwidth.

It also add a test to to check that the allocated but not settled bandwidth is not counted after 2 days.

Change-Id: Iee9dbe3517cc3b9825438360b276a07a43dfbc64
2021-06-05 17:14:52 +00:00
Fadila Khadar
aa49c8c44d satellite/orders: fix TestProjectUsageBandwidth
Change-Id: I3e9e81ea1ac2b0d7ef0936e4480a6b13c36bfcc5
2021-06-01 10:41:59 +02:00
Fadila Khadar
63cfc8fbe0 satellite/orders: use project daily bandwidth rollups
Replace GetProjectAllocatedBandwidth by GetProjectBandwidth which calculates
used bandwidth from allocated and settled bandwidth recorded in the
project_bandwidth_daily_rollups table.
For each day in the month, if the allocated bandwidth is expired, it uses the
settled bandwidth for computing used bandwidth.

Change-Id: Ife723c6d5275338f470619631acb25930d39ac3c
2021-05-25 18:28:58 +00:00
Michał Niewrzał
59eabcca24 satellite/orders: populate
project_bandwidth_daily_rollups table

We want to calculate used bandwidth better so we need to calculate it
from allocated and settled bandwidth. To do this we need first populate
this new table.

https://storjlabs.atlassian.net/browse/PG-56

Change-Id: I308b737bf08ee48ce4e46a3605697ab2095f7257
2021-05-25 18:07:22 +00:00
Egon Elbre
0858c3797a satellite/{metabase,satellitedb}: deduplicate AS OF SYSTEM TIME code
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.

As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.

Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.

Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
2021-05-11 12:40:36 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00
Jeff Wendling
a65aecfd98 compensation: always generate invoices for every node
instead of only generating invoices for nodes that had some
activity, we generate it for every node so that we can find
and pay terminal nodes that did not meet thresholds before
we recognized them as terminal.

Change-Id: Ibb3433e1b35f1ddcfbe292c034238c9fa1b66c44
2021-03-29 14:15:45 +00:00
Ethan
9506b67ca2 satellite/projectaccounting: Improve performance of ProjectAccounting.getBuckets
Limit bucket name lookup to date range of the calling methods since we only need distinct bucket names for that time period.
Adds new index and removes an index specific to project ID since it is no longer needed.

Change-Id: Ic07bbfb1c32280e0c0e39f8da020b284e1e5d974
2021-02-03 14:05:12 +00:00
Natalie Villasana
91bd4191dd satellite/accounting: add rollup archiver chore
The rollup archiver chore moves bucket bandwidth rollups and
storagenode rollups that are older than a given duration
to two new archive tables.

Change-Id: I1626a3742ad4271bc744fbcefa6355a29d49c6a5
2021-02-01 09:29:54 -05:00
Egon Elbre
7183dca6cb all: fix defers in loop
defer should not be called in a loop.

Change-Id: Ifa5a25a56402814b974bcdfb0c2fce56df8e7e59
2020-11-02 15:06:38 +02:00
Stefan Benten
0b43b93259 satellite/satellitedb: make limits per default NULL
This change completes the column migration of
5f6fccc6e8 and
2f648fd981.
It resets every users project limits who are below or equal to our
current production defaults.

Change-Id: Ie041d08bb67b62844f6023190fc00bc2dad5b1cb
2020-10-14 20:28:16 +00:00
Egon Elbre
c23a8e3b81 go.mod: update pgx to v4.9.0
Fix query to use TextArray instead of VarcharArray.
Fix queries to use the correct type.

Change-Id: Ibb7e55adba277d05778118d81ca697470e72c374
2020-09-29 19:03:08 +00:00
Jessica Grebenschikov
4a2c66fa06 satellite/accounting: add cache for getting project storage and bw limits
This PR adds the following items:
1) an in-memory read-only cache thats stores project limit info for projectIDs

This cache is stored in-memory since this is expected to be a small amount of data. In this implementation we are only storing in the cache projects that have been accessed. Currently for the largest Satellite (eu-west) there is about 4500 total projects. So storing the storage limit (int64) and the bandwidth limit (int64), this would end up being about 200kb (including the 32 byte project ID) if all 4500 projectIDs were in the cache. So this all fits in memory for the time being. At some point it may not as usage grows, but that seems years out.

The cache is a read only cache. When requests come in to upload/download a file, we will read from the cache what the current limits are for that project. If the cache does not contain the projectID, it will get the info from the database (satellitedb project table), then add it to the cache.

The only time the values in the cache are modified is when either a) the project ID is not in the cache, or b) the item in the cache has expired (default 10mins), then the data gets refreshed out of the database. This occurs by default every 10 mins. This means that if we update the usage limits in the database, that change might not show up in the cache for 10 mins which mean it will not be reflected to limit end users uploading/downloading files for that time period..

Change-Id: I3fd7056cf963676009834fcbcf9c4a0922ca4a8f
2020-09-25 16:28:49 +00:00
Stefan Benten
2f648fd981 satellite: make limits be nullable
Currently we have no way to actually set one
of the following limits to 0 (meaning not usable):

- maxBuckets
- usageLimit
- bandwidthLimit

With having the field nullable,
NULL corresponds to the global default,
0 now actually 0 and
a set value determines a custom limit.

Change-Id: I92bb77529dcbd0881ae8368921be9d246eb0919e
2020-09-21 19:34:19 +00:00
Michal Niewrzal
0604a672c1 satellite/metainfo: use metabase in loop
Change-Id: I1bb0c6fe0a762895fde950690b06f7dd9d77e178
2020-09-01 10:06:16 +00:00
Ethan
cfca021839 satellite/accounting: Add chore to cleanup old project bandwidth rollups data
Removes old project_bandwidth_rollups records that are no longer used.

Uses a retain months configuration to determine how many months to save.  Current month cannot be removed.
Tests retainMonths=-1, 0, 2

Change-Id: Ia4be2546cdb28802427acf41ecd85ad66df3e62c
2020-07-22 18:56:49 +00:00
Egon Elbre
d8dcae3075 all: fix error checking
Change-Id: Ia0da1bbd6ce695139922f94096c2419281905e32
2020-07-16 19:13:14 +03:00
stefanbenten
257855b5de all: replace == comparison with errors.Is
Change-Id: I05d9a369c7c6f144b94a4c524e8aea18eb9cb714
2020-07-14 15:50:25 +00:00
stefanbenten
f768302c91 satellite/admin: harden project deletion requirements
Change-Id: Ia7ea469f87469b16e464dc22af24b98a6ef1873d
2020-07-14 11:36:29 +00:00
paul cannon
bbdb351e5e all: use jackc/pgx in place of lib/pq
What:

Use the github.com/jackc/pgx postgresql driver in place of
github.com/lib/pq.

Why:

github.com/lib/pq has some problems with error handling and context
cancellations (i.e. it might even issue queries or DML statements more
than once! see https://github.com/lib/pq/issues/939). The
github.com/jackx/pgx library appears not to have these problems, and
also appears to be better engineered and implemented (in particular, it
doesn't use "exceptions by panic"). It should also give us some
performance improvements in some cases, and even more so if we can use
it directly instead of going through the database/sql layer.

Change-Id: Ia696d220f340a097dee9550a312d37de14ed2044
2020-07-13 15:54:41 +00:00
Cameron Ayer
6a60e1e96b satellite/satellitedb: inclusive interval_start in GetAllocatedBandwidthTotal
The DB query in GetAllocatedBandwidthTotal uses an exclusive range:

'WHERE interval_start > ?'

The value that is used for this condition is the first day of current the month,
00:00:00 UTC.

By using the exclusive '>', we exclude the entire first hour of the month from the
result set.

Change-Id: I3ed300f5230c7514dc9495a85e8166213cd0842e
2020-06-02 13:06:45 -04:00
Stefan Benten
e23bd806b4
satellite/accounting: separate usage and bandwidth limit (#3878) 2020-05-12 15:01:15 +02:00
Michal Niewrzal
22fbe804e3 satellite/accounting: test if project bandwidth limits reset with
billing cycle

https://storjlabs.atlassian.net/browse/USR-287

Change-Id: I4dc5f6342417b6af3384da32d3d2ed8592904406
2020-05-11 15:11:53 +00:00
Ethan
acf53bea4d satellite/orders;accounting: Add monthly project download bandwidth rollup
See https://storjlabs.atlassian.net/browse/SM-776

Change-Id: Ifd5cccea43c556fd59822d17344f399cfe9a7164
2020-05-04 15:49:57 +00:00
Stefan Benten
d73630fd4a
satellite/satellitedb: Ensure we just return bucket usage for buckets that exist (#3863) 2020-04-24 22:25:16 +02:00
Egon Elbre
8f73fb7a32 all: simplify uuid usage
uuid.UUID implements driver.Value so it can be directly used as a
scannable result.

Replace uses of dbutil.BytesToUUID with uuid.FromBytes.

Change-Id: I51a670185ceb3cc2199d5aa2b76bc3fc191ca8fe
2020-04-02 05:48:58 +00:00