Commit Graph

737 Commits

Author SHA1 Message Date
Clement Sam
589c82f23e storagenode: display quic status on SNO dashboard
This change includes storagenode QUIC status on SNO dashboard.
If disabled, it displays warning for SNOs to foward their
UDP port for quic.

Change-Id: I8d28c9c0f5f1e90d80b7c18b9e1e7b78c5e45609
2021-12-23 12:22:24 +00:00
Z
4762493e0f
storagenode/reputation: fix missing space in suspension message (#4309)
Fix missing space in the suspended message

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-13 19:30:33 +00:00
littleskunk
07fad75912
storagenode/piecestore: upload and download metrics for Grafana alerts (#4280)
* storagenode/piecestore: Add upload and download metrics for Grafana alerts

* storagenode/piecestore: group download metrics by piece action

Change-Id: Ib2a42b60c56c3f581915d512f4907c8db71e4624

Co-authored-by: Clement Sam <clement@storj.io>
2021-11-18 12:50:39 +00:00
TungHoang
f9b630b0f4
Fix monthly earning estimation (#4282) 2021-11-17 18:26:21 -05:00
Egon Elbre
96058be1a7 storagenode/storagenodedb: fix error message
Change-Id: Ibc98aa6539199f8dc8117f21a6ae645cea419403
2021-11-10 17:46:30 +00:00
Egon Elbre
af5b90ed32 storagenode/storagenodedb: include dbname in error
When something happens during opening or closing then it wasn't clear
which database had the issue.

Fixes storj/storj#4271

Change-Id: I34c8bae79a5b41ccd9b40aa8d836805f8c1a573c
2021-11-10 16:25:34 +00:00
Yingrong Zhao
2df41028a3 storagenode/piecestore: add logs for restore trash endpoint
When a satellite operator runs restore trash command, it's hard for node
operator to know whether their node has received the request. Adding
logs so this process is visible from node operators perspective.

Change-Id: I08c670b1b0c5971e6b73e038a0935cb0caaca63d
2021-10-20 15:37:28 +00:00
Clement Sam
29599dd7cd testsuite/ui/multinode: add first node test
Change-Id: Ibeaee92e6d1a41e408659a5045f52e1908f73089
2021-10-18 16:07:01 +00:00
Clement Sam
0d58172c38 storagenode: add doc.go files for sno packages
Change-Id: I23d4b8b462e1b03718d0c4801cc2aaff520e7356
2021-09-29 08:24:56 +00:00
Egon Elbre
71eb184ef3 storagenode/piecestore: simplify TestTooManyRequests
Change-Id: I1452735bef206bc8cf9bde72fc503fec3ae5b067
2021-09-21 17:18:37 +03:00
Fadila Khadar
c00ecae75c satellite/gracefulexit: stop using gracefulexit_transfer_queue
Remove the logic associated to the old transfer queue.
A new transfer queue (gracefulexit_segment_transfer_queue) has been created for migration to segmentLoop.
Transfers from the old queue were not moved to the new queue.
Instead, it was still used for nodes which have initiated graceful exit before migration.
There is no such node left, so we can remove all this logic.
In a next step, we will drop the table.

Change-Id: I3aa9bc29b76065d34b57a73f6e9c9d0297587f54
2021-09-14 11:52:34 +00:00
Egon Elbre
1aec831d98 satellite/audit,storage: increase sleep delay in TestMaxVerifyCount
Currently TextMaxVerifyCount flakes in some tests, try increasing the
sleep time to ensure that things are slow enough to trigger the error
condition.

Also pass ctx to all the funcs so we can handle sleep better.

Change-Id: I605b6ea8b14a0a66d81a605ce3251f57a1669c00
2021-09-10 15:30:37 +00:00
Michał Niewrzał
c258f4bbac private/testplanet: move Metabase outside Metainfo for satellite
At some point we moved metabase package outside Metainfo
but we didn't do that for satellite structure. This change
refactors only tests.
When uplink will be adjusted we can remove old entries in
Metainfo struct.

Change-Id: I2b66ed29f539b0ec0f490cad42c72840e0351bcb
2021-09-09 07:15:51 +00:00
igor gaidaienko
92deef4f34 Revert "storagenode/payouts: historical payouts use satellitesDB instead of trustPool"
This reverts commit 4a98dd40e2.
2021-08-02 13:55:21 +03:00
Fadila Khadar
c4202b9451 satellite/gracefulexit: use graceful_exit_segment_transfer_queue
For being able to use the segment metainfo loop, graceful exit transfers have to include the segment stream_id/position instead of the path. For this, we created a new table graceful_exit_segment_transfer_queue that will replace the graceful_exit_transfer_queue. The table has been created in a previous migration and made accessible through graceful exit db in another one.
This changes makes graceful exit enqueue transfer items for new exiting nodes in the new table.

Change-Id: I7bd00de13e749be521d63ef3b80c168df66b9433
2021-07-21 14:02:20 +00:00
Fadila Khadar
b0d98b1c1a satellite/gracefulexit: allow use of graceful_exit_segment_transfer_queue
For being able to use the segment metainfo loop, graceful exit transfers have to include the segment stream_id/position instead of the path. For this, we created a new table graceful_exit_segment_transfer_queue that will replace the graceful_exit_transfer_queue. The table has been created in a previous migration.
This change gives access to this table.
Graceful Exit doesn't use the table yet, this will be done in a next change.

Change-Id: I6c09cff4cc45f0529813a8898ddb2d14aadb2cb8
2021-07-21 12:34:44 +00:00
Qweder93
4a98dd40e2 storagenode/payouts: historical payouts use satellitesDB instead of trustPool
Change-Id: I39f4215f4ebf91bd1b38fbcb5c58e6ba53ceff1b
2021-07-15 16:19:18 +03:00
Qweder93
4d0fe39235 storagenode/satellites: address added, caching satellite's addresses from trust
Change-Id: Ica3eea5b8d81b176c6a4385fea803730b08ece16
2021-07-08 15:38:23 +00:00
Yaroslav Vorobiov
cbb4cd3fc3 multinode/reputation: add vetted at timestamp
Change-Id: Id35cb6cfdabf4bf2762e4a162cf3157afb0ff170
2021-07-07 18:11:54 +03:00
Yaroslav Vorobiov
a5fd903177 storagenode/reputation: add vetted at timestamp
Change-Id: I02d59414b6b172cf7f7bfc92df222cf4a5574e0e
2021-07-07 18:11:54 +03:00
Yaroslav Vorobiov
818f6c6ea6 multinode/console: add summary to storage usage API
Change-Id: Ia8a1e598d667f25461f73f1626da22113cb7caeb
2021-07-07 15:00:05 +03:00
Qweder93
6c02258925 storagenode/peer: register multinode Payouts endpoint
Change-Id: Iaaba0157e605bb51a73244517986d6b49a59d8d5
2021-07-06 10:09:24 +00:00
Yaroslav Vorobiov
68627e7d80 multinode/console: add reputation satellite api
Change-Id: I7cef6c1c271607f7485f604d5b61587558a31878
2021-07-05 15:32:22 +00:00
TungHoang
e1379bea0f
storagenode/piecestore: allow rejecting slow clients
Estimate speed of the uploads by calculating the time a client has been in a non-congested state. Storage node is uncongested when it has used up over 80% of its maximum concurrent requests capacity.

Currently it's disabled by default.
2021-07-01 09:59:04 +03:00
Qweder93
b9d7874dc2 multinode: old drpc api returned
drpc api with old names added to prevent breaking
backwards compatibility.

Change-Id: I1b8a41f2c1e0bd11ac83c86f0b1bfbfc1f07378f
2021-06-22 23:02:47 +00:00
Jeremy Wharton
8a070e7c25 satellite/overlay: Ignore unnecessary check-ins
This prevents the database from being contacted unnecessarily,
reducing load.

Change-Id: Ib2420f68a20636ec35eb3dd3df8e02bd5341b419
2021-06-22 09:00:41 +00:00
JT Olio
8fec48f310 storagenode: use storj.io/dcs-satellites now
Change-Id: I4c791b37778757c77872f1c25744efa848a5fed3
2021-06-21 15:11:30 +00:00
Yingrong Zhao
81dd7b2f37 storagenode/internalpb: fix protobuf generation for newer protobuf
version

Change-Id: I24cc5568b5f5c36f8c63062f79a4cb8d1a8f164e
2021-06-11 17:20:42 +00:00
Qweder93
5d70b6abef multinode/bandwidth: added monthly bandwidth summaries
montly bandwidth summaries for single/all nodes, single/all satellites added.

Change-Id: Ic384886c10622df74a4bd0645e2d7f2a85477644
2021-06-11 16:31:59 +00:00
Yaroslav Vorobiov
157d3d980e multinode/console: storage usage and total storage usage
Change-Id: I4970275daf4a4a9c5d02aea6a205891869dd4eff
2021-06-10 16:01:41 +00:00
Yaroslav Vorobiov
c9cfb5ed0c storagenode/retain: add more verbose monkit monitoring
Change-Id: Ibb9804268751b4b1842eb729bc510dba83e9b28b
2021-06-04 20:20:11 +00:00
Qweder93
d8c11a79b9 multinode/payouts: paystub extended with disposed
added Disposed to paystub struct to count NetTotal value.

Change-Id: Iaae1f98a69b82166ba5594e8589c7d2a540bd07f
2021-06-01 16:28:20 +00:00
crawter
6d9b91d435 private/multinodepb: drpc operators controller added
Change-Id: Ie9bad9d5dba3e508d4cb8165aaca88d98ef1304d
2021-06-01 11:00:31 +00:00
Qweder93
91b7e24d55 multinode/payouts: naming refactoring
Renamed methods, error messages.

Change-Id: I7d7b6b092c05bbc5bf1322855efc5ccb9b312671
2021-05-31 19:47:12 +00:00
Yaroslav Vorobiov
23f9beb635 multinode/console: held amount summary
Change-Id: Ia800748343e363d930ce0a0b9ab286b5abdc96af
2021-05-28 20:05:16 +03:00
Egon Elbre
92226d8ddb storagenode/storagenodedb: fix fillInBlobAccess
fillInBlobAccess was using a non-pointer receiver so the receiver wasn't
being modified. Luckily, this seems to be only being used in tests.

Change-Id: Ice01419933295562d558d48ba314d476660b67bd
2021-05-25 16:41:10 +00:00
Qweder93
099a31c69f multinode/payouts: paystubs for single period added
paystubs for single/all satellites for specific period added.

Change-Id: Ibb822aa11380bc70f4db8c81fa0b9b6f7a739078
2021-05-25 18:59:05 +03:00
Qweder93
1bc4f8fccd multinode/payouts: paystubs for all time added
paystubs for single/all satellites for all time added.

Change-Id: Ia665c69994f769cb0071363a617eaf87ef1a05f2
2021-05-25 18:19:23 +03:00
Qweder93
79172777bd mnd/payouts: estimations replaced with expectations
Added expectations endpoint (estimations and distributed), added
coalesce to db query, so in case of empty payouts db 0 will be returned instead of error.

Change-Id: I535f14ef097876448d8949bc302895b25da2b6e7
2021-05-24 18:11:30 +00:00
Qweder93
f0dd7f3739 mnd/payouts: undistributed payouts added
Change-Id: Id61ac58fceca13b9a9e12a7e03c3d9624cce3fea
2021-05-24 18:20:08 +03:00
Egon Elbre
10372afbe4 ci: fix lint errors
Change-Id: Ib5893440807811f77175ccd347aa3f8ca9cccbdf
2021-05-17 13:37:31 +00:00
Qweder93
f2812d76cd multinode/payouts: satellite period/allTime summaries added
payout summaries for specific satellite period/allTime added

Change-Id: I144138304f01f23d5c4b10931988eaaced656aaf
2021-05-14 18:49:31 +00:00
Yingrong Zhao
59f443e71a storagenode/contact: add authentication for PingNode endpoint
Currently, if a node has untrusted a satellite, the satellite can still
successfully ping the node. If a node decide to untrust a satellite, the
satellite should also mark it as conact failed

Change-Id: Idf80fa00d9849205533dd3e5b3b775b5b9686705
2021-05-12 18:15:32 +00:00
Qweder93
19561698ba multinode/payouts: all satellites summaries added.
payout summaries for specific/all periods added.

Change-Id: I92087edec548c0418a0f543d643e59f5c7df9621
2021-05-12 18:43:47 +03:00
Qweder93
a11698f370 multinode/payouts: estimated payouts added
estimated payouts for specific/all satellites added.

Change-Id: I2530c9f1775593588e2a8f6c087ce6b4f9e354c4
2021-05-11 11:33:32 +00:00
Egon Elbre
69b149a66f mod: bump uplink
uplink stopped using zap, hence some of the private methods needed to be
changed.

Change-Id: Iac1fae45a40cd3f1649b9f672bf8c250344986d5
2021-05-06 14:48:36 +00:00
Egon Elbre
dae9464ee8 storagenode/trust: missed one error test
Change-Id: I3aff2e6b57ecc1b42f5c8eaca4dc762899cd53c5
2021-04-30 10:52:44 +03:00
Egon Elbre
2c657f594e satellite/metainfo: don't rely on exact error name
Change-Id: I7975a00c32891a43b4f1e6dc4f5847201844f2ec
2021-04-29 18:35:56 +00:00
Egon Elbre
961e841bd7 all: fix error naming
errs.Class should not contain "error" in the name, since that causes a
lot of stutter in the error logs. As an example a log line could end up
looking like:

    ERROR node stats service error: satellitedbs error: node stats database error: no rows

Whereas something like:

    ERROR nodestats service: satellitedbs: nodestatsdb: no rows

Would contain all the necessary information without the stutter.

Change-Id: I7b7cb7e592ebab4bcfadc1eef11122584d2b20e0
2021-04-29 15:38:21 +03:00
Yingrong Zhao
307886ffe8 storagenode/piecestore: fix error handling in TestDownload
If a read error is returned, we want to preserve it instead of
overriding it with close error

Change-Id: Ie253a453c7b6b62598b89dffd7635b8266357074
2021-04-23 18:43:45 +00:00