Commit Graph

126 Commits

Author SHA1 Message Date
Clement Sam
c64f3f3132 {storagenode/console,web/storagenode}: fetch pricing model from storagenode API
Instead of the hardcoded payout rates that is assumed for all satellites,
this change adds a new endpoint for fetching the pricing model for
each satellite.
The pricing model is then displayed on the Info & Estimation table
on the dashboard

Updates https://github.com/storj/storj-private/issues/245

Change-Id: Iac7669e3e6eb690bbaad6e64bbbe42dfd775f078
2023-05-09 13:25:44 +00:00
Clement Sam
3d3f9d133a storagenode: fix B*h to bytes disk usage conversion
The used space graph values are correct when a single satellite is
selected but wrong for 'All satellites'. This is related to the
queries for getting the individual disk usages for all satellites
per day and the summary and average for all satellites per day:

1. dividing the sum of at_rest_total by the total_hours is wrong.
Simply put, we were assuming that, for example (4/2)+(6/3) equals
to (4+6)/(2+3), assuming we had 4 and 6 at_rest_total values with
2 and 3 respective hours.

2. To get the average, we need to first find the sum of the
at_rest_total_bytes for each timestamp across all satellites
before taking the average of the sums instead of just taking the
average from the individual satellite values.

Closes https://github.com/storj/storj/issues/5519

Change-Id: Ib1314e238b695a6c1ecd9f9171ee86dd56bb3b24
2023-02-06 18:50:31 +00:00
Clement Sam
7461ffe148 {storagenode,web/multinode}: fix storage usage db/cache retrieval queries
The query changes we did while fixing the usage graph led to wrong
payout calculations directly linked to disk space.

This change:

- avoids converting from Bh to B directly in the query
- returns the at_rest_total in the original bytes*hour value
- returns at_rest_total_bytes as the calculated disk spaced used in bytes
- uses the at_rest_total_bytes only for the disk space graph
- return summary_bytes as the average disk space used within the specified date
- updates the disk space graph header to "average disk space used this month"

The total disk used in the month is also displayed in B not B*day

Resolves https://github.com/storj/storj/issues/5355

Change-Id: I2cfefb0fe711f9c59de2adb547c4ab50b05c7cbb
2022-12-09 11:07:33 +00:00
Clement Sam
59b37db670 storagenode: overhaul QUIC check implementation
The current implementation blocks the the startup until one or none
of the trusted satellites is able to reach the node via QUIC.
This can cause delayed startup. Also, the quic check is done
once during startup, and if there is a misconfiguration later,
snos would have to restart to node.

In this change, we reuse the contact service which pings the satellite
periodically for node checkin. During checkin the satellite tries
pinging the node back via both TCP and QUIC and reports both statuses.
WIth this, we are able to get a periodic update of the QUIC status
without restarting the node.

Also adds the time the node was last pinged via QUIC to the tooltip
on the QUIC status tab.

Resolves https://github.com/storj/storj/issues/4398

Change-Id: I18aa2a8e8d44e8187f8f2eb51f398fa6073882a4
2022-11-09 03:15:57 +00:00
Clement Sam
d625eb85fd storagenode: use bytes instead of bytes*hour unit for used space graph
Closes https://github.com/storj/storj/issues/5146

Change-Id: I1b135da81a68193b5b50c761088d79471ca3a2fe
2022-10-28 18:42:45 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
Clement Sam
9a539c4830 storagenode/storageusage: add interval_end_time, rename interval_start to timestamp
The satellite now returns the last interval_end_time for each
daily storage usage.
We need to store the interval_end_time in the storage usage cache.
Also, renamed interval_start to timestamp to avoid ambiguity since
the interval_start only stores just the date/day returned by the
satellite.

Updates https://github.com/storj/storj/issues/4178

Change-Id: I94138ba8a506eeedd6703787ee03ab3e072efa32
2022-08-10 01:03:00 +00:00
Qweder93
98b8c7be06 storagenode/console: consoleAPI EstimatedPayout flacky test fixed
In rare cases time frame between creating time.Now() variable and calling
service method that receives it (after API method call) was big enough to distort
current month expectations and make test fail with last digit difference in 1

Change-Id: Ib811492d62f6598a5c40a09de6a87bffeaa0a78e
2022-08-01 15:13:15 +00:00
Egon Elbre
05e165283f storagenode/console/consoleapi: use fixed time.Now()
It seems the tests relied on time.Now(), which might cause some
discrepancies in calculations. Use a fixed time.Now() rather than
recalculating.

As a sidefix, remove "Test" prefix from t.Run. These are unnecessary.

Change-Id: I1de903fcf0fcf46fc8e3acf2463e17239b8e3cc6
2022-07-01 12:36:01 +03:00
Egon Elbre
dc0f7b5f77 storagenode,web/storagenode: use go:embed for assets
Go can now directly embed files without relying on external tools.
This makes code use go:embed and avoid the external tooling.

go:embed requires files to be present in the embedded directory,
hence we need to add .keep to "dist" folder. We also add one to
public/.keep, such that it won't be deleted when building storagenode.

Change-Id: I8bef81236be6829ed37ed4c16ef693677b93a631
2022-03-11 16:01:28 +02:00
Egon Elbre
5f7ea1358d private/web: make caching headers reusable
Move storagnode/console caching headers to private/web. Also,
start using them in multinode/console/server.

Change-Id: I1f0f3c9833a183476009737cece515ae7537fb83
2022-03-11 11:19:11 +02:00
Clement Sam
8b40a071a0 storagenode: check if QUIC is properly configured
This change adds a check to confirm if UDP port if properly configured for QUIC

Resolves https://github.com/storj/storj/issues/4332
Partly resolves https://github.com/storj/storj/issues/4358

Change-Id: I9a66f26a115e48b4fcd168f50a7d0b4d81712f4e
2022-01-20 12:04:04 +00:00
Clement Sam
589c82f23e storagenode: display quic status on SNO dashboard
This change includes storagenode QUIC status on SNO dashboard.
If disabled, it displays warning for SNOs to foward their
UDP port for quic.

Change-Id: I8d28c9c0f5f1e90d80b7c18b9e1e7b78c5e45609
2021-12-23 12:22:24 +00:00
Qweder93
4d0fe39235 storagenode/satellites: address added, caching satellite's addresses from trust
Change-Id: Ica3eea5b8d81b176c6a4385fea803730b08ece16
2021-07-08 15:38:23 +00:00
Egon Elbre
10372afbe4 ci: fix lint errors
Change-Id: Ib5893440807811f77175ccd347aa3f8ca9cccbdf
2021-05-17 13:37:31 +00:00
Qweder93
a11698f370 multinode/payouts: estimated payouts added
estimated payouts for specific/all satellites added.

Change-Id: I2530c9f1775593588e2a8f6c087ce6b4f9e354c4
2021-05-11 11:33:32 +00:00
Egon Elbre
961e841bd7 all: fix error naming
errs.Class should not contain "error" in the name, since that causes a
lot of stutter in the error logs. As an example a log line could end up
looking like:

    ERROR node stats service error: satellitedbs error: node stats database error: no rows

Whereas something like:

    ERROR nodestats service: satellitedbs: nodestatsdb: no rows

Would contain all the necessary information without the stutter.

Change-Id: I7b7cb7e592ebab4bcfadc1eef11122584d2b20e0
2021-04-29 15:38:21 +03:00
Qweder93
866a3e3ad3 storagenode/console: estimated payout tests fix
Added check if days paste <1 and node joined in current month -> days paste = 1

Change-Id: Ice1ccd6c869f629a87da585850b9b4f3729ec65c
2021-03-24 18:05:37 +02:00
NickolaiYurchenko
034eb2c214 storagenode: wallet features
WHAT: updated dashboard type to return wallet features from storagenode api

Change-Id: Ifdbdeae8b9ae239eaebe32d0a093557154f777ad
2021-03-24 09:08:36 +00:00
Qweder93
0b060c8297 storagenode/console: estimated payout flaky test skip
Change-Id: Ibb20b5716df94f9430026ed697d57b6be9cc137d
2021-03-15 15:26:47 +02:00
Qweder93
487ad84b23 storagenode:consoleAPI flacky tests for estimatedPayout fixed
joinedAt date now is 2 months before time.Now which should fix calculations on early dates.

Change-Id: I6250ec2463c2dd609e811b74d908a02a3e64f19a
2021-03-13 20:33:25 +02:00
Moby von Briesen
0370d1553e storagenode/console: Fix TestStorageNodeApi flakiness
TestStorageNodeApi is failing due to very slight differences in
float values. This change rounds these values to 3 decimal places before
comparing them.

Change-Id: Ic7fae3a5e0a0a942c03d982bfa7b19357f2e3d2e
2021-02-26 15:12:35 -05:00
Qweder93
f2be856864 storagenode/payouts: estimated payouts calculations upgraded
Added recalculations of estimated income in current month in case
node joined in current month

Change-Id: Iac925fd8bcf2be8f1089e5f001a01448ba7650be
2021-02-25 14:11:44 +00:00
NickolaiYurchenko
badd6d12d4 web/satellite: distributed amount added to tables, content changed
Change-Id: Ib0eb3fa3f4f9d451f4b4048a6409d19968a41956
2021-02-24 16:01:40 +00:00
Jeff Wendling
3eb321fd33 storagenode/console/consoleapi: fix flaky test
Change-Id: I1af928953a19155e3c2e9f8c63e7162978e7b475
2021-02-17 19:48:55 +00:00
Egon Elbre
5d895fb404 storagenode/{payouts,console}: use same time for all calculations
When using calling time.Now() multiple times, they can cross
month boundary causing errors in calculations.

Change-Id: I66b5be7598f3bf475b4b5fe0dcce82eee55b3134
2021-02-01 13:59:50 +02:00
Qweder93
6ba8f6c8a9 storanode, satellite: payout renamed to payouts, expected estimation payouts added, console api for audits reworked
Change-Id: I4aa5e99bffaa87d0a800a429a4c83aa498ad4b7b
2021-01-18 10:56:03 +00:00
Moby von Briesen
710b86849c storagenode/{console, reputation}: Add audit history to storagenode API
Change-Id: Id680b07feea380824e56496d545f46a94fdcc5d5
2021-01-04 17:37:45 +00:00
Qweder93
2fd7809e54 storagenode/payout: stefanbenten satellite name added to payout history, satellites with no held history removed from list
Change-Id: I96861058ccb9c8ce52698796c91b999eaec1f6e6
2020-12-17 11:01:28 +00:00
Qweder93
12144a600b storagenode/console: payout tests and heldhistory joined_at rounding added
Change-Id: I1d43620fbafbf7ed92588b84cb9c6b8ced8832ef
2020-12-14 19:35:04 +02:00
Qweder93
2f62cdf491 storagenode/console: diskSpaceInfo extended with overused diskspace, getDashboardData updated.
Change-Id: I44db26661a8dfb45b5d8e9fcb7511f63deb88cad
2020-12-08 14:55:55 +00:00
Qweder93
8dc10e32ad stefan benten satellited added to historical payout data
Change-Id: I1177b2d2ef10d514f7d401e29891fa7dd964e9ac
2020-11-09 15:43:41 +00:00
Moby von Briesen
aa86c0889c storagenode/console: Add current storage used per satellite to storagenode api
Right now, the best way for a storage node operator to get the current
space used for each satellite is to run the `storagenode exit-satellite`
command for graceful exit, and cancel at the second confirmation prompt.
This is convoluted and the data is readily available from the Blobs
Usage Cache.

This change adds the current space used by each satellite to the
endpoints `/api/sno` and `/api/sno/satellite/<Satellite ID>`

Change-Id: I2173005bb016fc76db96fd598d26b485e5b2aa0b
2020-10-14 21:30:28 +00:00
Qweder93
245986d528 negative space calculations fix removed
Change-Id: I342c61856fce6d02dc99fd27fd3d563540f22b64
2020-09-30 14:08:24 +00:00
Michal Niewrzal
cd2a5484f3 storagenode/console: ignore untrusted satellite while returning
dashboard data and calculating satellites data

Change-Id: I71d596891477e0839863e007689b6e2e6e420a22
2020-09-29 18:27:49 +00:00
Qweder93
8182fdad0b storagenode: heldamount renamed to payouts, renamed some methods and structs to more meaningful names. grouped estimated payout with pathouts
satellite: heldamount renamed to SNOpayouts.

Change-Id: I244b4d2454e0621f4b8e22d3c0d3e602c0bbcb02
2020-09-16 14:57:35 +00:00
Qweder93
ac29d80495 storagenode: heldamount GetPaystub refactored, estimationPayouts logic separated form console to separate service, storagenodeapi tests fixed.
Change-Id: I902823ef40a62861ce32799e9fb7a67a1e14710d
2020-09-09 15:31:16 +00:00
Qweder93
7d9897b7af storagenode/nodestats: online_score added
Change-Id: I84b50a6cace306e5f10d53a2073fe8810d4d2960
2020-09-02 17:45:01 +03:00
Qweder93
c4a4745dd8 storagenode/console: audit per satellite now uses satelliteName instead of satelliteID
Change-Id: I8221ec840f654a62aedfb62a4194616db890f539
2020-08-25 12:52:47 +00:00
Qweder93
f16cf5cccf storagenode/console & /inspector: added recalculation of disk space info
Change-Id: Id003d031a6464ec095c31290fd6a756ead644261
2020-08-25 14:19:10 +03:00
Qweder93
4ee1b2d45a storagenode/console: added list of all audits per satellite to sno dashboard/satellites
Change-Id: I52e58748d6467f372d9a308347fc77e400d137e2
2020-08-10 12:55:07 +00:00
Qweder93
b4c9badab1 storagenode/console: estimation payout fix
Change-Id: I5d9f11fffd74978f3ca684fd08aac44a27a83c71
2020-07-27 21:41:07 +03:00
Qweder93
f531bc8638 storagenode/heldamount payout-history rout fix, usage_at_rest in estimation payout calculations fixed
Change-Id: I6f819a404a45b2a96c1aae33c67ebea1ab83aef0
2020-07-24 15:19:45 +00:00
Qweder93
0949731caa storagenode/console: estimation payout held split from total payout, calculations fixed
Change-Id: I064f473ffeb3a3051c9228d1dd84fe0fc86dd3ef
2020-07-21 15:31:51 +03:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
Qweder93
62fec25104 storagenode/heldamount: returns usage_at_rest in tbm instead of tbh
Change-Id: I183a56460ea76a53680ca6861d02cecebe3576ec
2020-07-15 15:46:13 +03:00
Qweder93
7b4a8c4d6d storagenode/heldamount: payoutHistory added
Change-Id: I93dd3d024085d19ecff76075e52bf66796207fd6
2020-07-14 17:35:03 +03:00
Egon Elbre
262da14359 storagenode/console/consoleapi: disable flaky TestStorageNodeApi
Change-Id: I076c9a46fece86d34eae117ab84f94f99e7e64e0
2020-07-13 18:35:38 +03:00
Qweder93
e17243fcd7 storagenode/console: estimation payour for current and previous month reworked
Change-Id: I937d5d8f7c17949b539dcd6e36af27400a5043e2
2020-07-10 12:18:53 +00:00
Bill Thorp
a3c902ab84 storagenode/pieces: hours in a month should be 720
Per https://documentation.tardigrade.io/pricing/billing-and-payment:
"The calculation of per object fees is based on a standard 720-hour month."

On most years, the average value is 730 (365*24/12), except leap years.
However, we want to have ours be 720 (30*24) so its lines up with days.

Change-Id: Ifb9691878f1a7ea81ed36c92b37985493295fe31
2020-07-07 15:26:15 -04:00