Commit Graph

516 Commits

Author SHA1 Message Date
Qweder93
0521435e08 storagenode/gracefulexit: added deletion of all files left in storage/blobs/satellite after successful GE
https://storjlabs.atlassian.net/browse/SG-368

Change-Id: I29a978fe0d0153aedf2be91dc7f45b4ef386d447
2020-07-08 14:38:31 +03:00
Bill Thorp
a3c902ab84 storagenode/pieces: hours in a month should be 720
Per https://documentation.tardigrade.io/pricing/billing-and-payment:
"The calculation of per object fees is based on a standard 720-hour month."

On most years, the average value is 730 (365*24/12), except leap years.
However, we want to have ours be 720 (30*24) so its lines up with days.

Change-Id: Ifb9691878f1a7ea81ed36c92b37985493295fe31
2020-07-07 15:26:15 -04:00
Moby von Briesen
e9dd5b2845 storagenode/piecestore: Properly log/send metrics for all successful pieces
When an uplink or repair work finishes uploading a piece to a
storagenode, it has no reason to wait another round trip after the piece
is committed to gracefully close the connection - in many (most?) cases,
the connection is simply canceled once the upload is complete. This has
the unintended side effect of producing a lot of "piece canceled" logs
and metrics on the storagenode side, when the reality is that the piece
uploads were successful, and not really canceled. This commit fixes
that.

Change-Id: Icbc1f7857d380134560219c1c19c186df2783cd0
2020-07-07 15:19:17 +00:00
Qweder93
ac716e1514 storagenode/heldamount: payment receipt added to monthly paystub, heldamount.service separated for service and endpoint
Change-Id: Id759586c6362edbef34c230d4f0d2585c11c9b47
2020-07-06 09:51:52 +00:00
Cameron Ayer
35b709ba18 storagenode/storagenodedb: check if db is nil before closing
In the event of an error in storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases
the caller will attempt to close all databases. However, the error prevents the DB from
being opened and set in the proper place. Attempting to close results in a nil pointer
dereference

https://forum.storj.io/t/node-wont-start-after-update-to-v1-6-4-runtime-error-invalid-memory-address-or-nil-pointer-dereference/7889

Change-Id: Ibfe6f3e13c36d9d15a0cb46e384f0120afdab60b
2020-07-02 15:02:38 +00:00
Qweder93
577f72cb92 storagenode/version: notifications added
Change-Id: Ib9720d8124d8e078354a292b644e2db1f5fffe67
2020-07-01 19:35:46 +03:00
NickolaiYurchenko
b878fcc4b2 storagenode/heldamount: id removed from satellite name
Change-Id: Ic524a40930a5fe7673ccce817d6f68c3538e5208
2020-07-01 15:38:05 +03:00
Qweder93
9b90712aa0 storagenode/heldamount: payents added to db
Change-Id: Ib6c486251ca08d34003c35379d10314127edf103
2020-06-30 17:24:35 +03:00
Qweder93
9a02149654 storagenode/heladamount: held history extended with joined_at date, total_held and total_disposed amounts
Change-Id: I41fe9ab8c5667aa988257a94848ea70225305d79
2020-06-30 13:33:25 +00:00
Yingrong Zhao
51dfc6bf4f storagenode/gracefulexit: make minimum transfer speed to be 5KiB
with 128B/sec, a satellite with 10min default timeout could already
closed its connection to a node even though the node was able to
compelete the transfer.

Change-Id: I6173d6473a62c6d0b0e0a8765c1ae0a5e57b0a08
2020-06-23 21:14:18 +00:00
Ivan Fraixedes
98d477effb
storagenode/collector: Fix comment doc
Change-Id: I703dce7d1b7d7653bbea901c798266a0108b9eec
2020-06-19 13:51:23 +02:00
Jeff Wendling
3842118bab storagenode/heldamount: use correct field for repair usage
Change-Id: I1e0d0bd4c416a21d6900fb723185599f58391d8a
2020-06-12 10:55:35 -06:00
Qweder93
2c3fe5597d storagenode/nodestats: unknown_audit_score added to Service
Change-Id: I1f97f6f0eace9858e466a53d4d4eeabe8059e4eb
2020-06-12 14:00:41 +00:00
Qweder93
0826c8d87f satellite/heldamount: fix dimension of usage_at_rest
Change-Id: If1518ad41736912d15fb2c882c9e236c16f85a51
2020-06-11 13:07:51 +00:00
Moby von Briesen
be59727790 storagenode/orders: Add archival functionality to orders filestore
* Allow orders to be archived after being settled successfully with the
satellite.
* Allow for cleanup of orders that were archived before a certain time.
* Rewrite other parts of the orders file store to work better with new
design.

Change-Id: I39bea96d80e66a324ec522745169bd6d8b351751
2020-06-11 08:47:37 +00:00
Qweder93
e52809d53e cmd/storagenode: add check if satellites available to gracefulexit
Change-Id: I8747507593d810bbdec0d140de0600ee147011c3
2020-06-10 13:38:36 +00:00
Moby von Briesen
0b109c32e4 storagenode/piecestore/usedserials: add monkit metric for serials that are randomly deleted
This will give storagenode operators a better idea of whether the memory
allocated to the usedserials store is sufficient.

Change-Id: I5c30f2e39473a573f43409511ad9e2e32680479c
2020-06-09 17:04:37 -04:00
Yaroslav Vorobiov
09ca382abf storagenode/db: preflight improve index discovery
Change-Id: I876b321f6cd4e91dfced87aa4d39f2cf9a8e63d0
2020-06-05 14:03:25 +03:00
Qweder93
7f8e553022 console/dashboard: added pieces headers size to calculations
Change-Id: I0ee8d6bcb9ce9f69d49ebac2b95579166389668e
2020-06-04 16:39:02 +00:00
Moby von Briesen
c8c0a42269 storagenode/orders: begin implementation of file store for order limits
* Will replace order limits database.
* This change adds functionality for storing and listing unsent orders.
* The next change will add functionality for order archival after
submission.

Change-Id: Ic5e2abc63991513245b6851a968ff2f2e18ce48d
2020-06-03 22:47:04 +00:00
Egon Elbre
4d0cb1af7e storagenode/piecestore: verify before checking free disk
Change-Id: I3fe0383f9b13b99ef9d63ff235616ff204cf6d76
2020-06-02 17:49:14 +03:00
Qweder93
89c9672ce0 storagenode/piecestore: available storage check added in Upload
Change-Id: I71e9e5f335d4320d5de8b374fe747fec43179f78
2020-06-01 16:55:22 +00:00
Egon Elbre
07050eea26 all: use common/storj
Change-Id: Id1e36d52f9807b5ffbb72ce73f4b60cb21b68a78
2020-05-29 11:57:32 +03:00
Moby von Briesen
dc57640d9c storagenode/piecestore: switch usedserials db for in-memory usedserials store
Part 2 of moving usedserials in memory
* Drop usedserials table in storagenodedb
* Use in-memory usedserials store in place of db for order limit
verification
* Update order limit grace period to be only one hour - this means
uplinks must send their order limits to storagenodes within an hour of
receiving them

Change-Id: I37a0e1d2ca6cb80854a3ef495af2d1d1f92e9f03
2020-05-28 12:52:52 -04:00
Moby von Briesen
909d6d9668 storagenode/piecestore/usedserials: add in-memory store for used serials
Implement an in-memory store for keeping track of order limit serial
numbers. It automatically deletes items if its size exceeds a configured
limit.
This change is part 1 - it creates the store
In part 2, the in-memory store will replace the usedserials database

Change-Id: I36f540ed809f034a27c1d7cede8a0a8b080af818
2020-05-28 12:52:52 -04:00
paul cannon
7395dd1e6e storagenode/gracefulexit: revalidate existing pieces
..before they are transferred to another node and submitted to the
satellite as successful piece transfers, because if we submit an invalid
signature, the node will be marked as a cheater and disqualified
immediately.

These signatures should have been validated when the piece was
originally stored, but bitrot does happen and needn't be cause for an
immediate DQ.

Change-Id: I8b0ebd5812ea8a2e60766005b7251fbb74ef7857
2020-05-28 09:50:14 -05:00
Qweder93
73214c6d1c storagenode/heldamount: heldhistory reworked to all satellites
Change-Id: I8d7707fddfbdc52d29951a8a002978c7fbb07049
2020-05-28 11:44:26 +00:00
Qweder93
f2a0c64425 storage/filestore: log potential disk corruption
In walkNamespaceWithPrefix log in case of "lstat" error, because this may indicate an underlying disk corruption.

SG-50

Change-Id: I867c3ffc47cfac325ae90658ec4780d213ff3e63
2020-05-27 12:12:55 +00:00
Qweder93
8db848791f storagenode/console: added estimated payout for current month and estimated pay stub for previous month (until there's real data in satellite's table) + heldback percentage rate for previous month.
Change-Id: I9346f6d22ed6fbb7e5346b102fc898467678f384
2020-05-27 14:51:23 +03:00
littleskunk
2fbb34c3ea
nodeselection: Increase minimum free space to 500MB (#3898) 2020-05-25 12:13:28 +02:00
crawter
f5ac678b0a storagenode/satellitesdb: added FK constraint to satelliteID
Change-Id: If5adf2b92627fcf80850670ba672b346320ddd87
2020-05-21 13:01:20 +00:00
Egon Elbre
bef84a5f9d storagenode: remove dependency to overlay.NodeDossier
This is the last dependency from storage node to satellite.

Change-Id: I12f7abb91e84f823ba5af126c6e2979519838612
2020-05-21 08:37:13 +03:00
crawter
2c9afe7f17 storagenode/console/api/helamount: periods with heldamount data endpoint added
Change-Id: Ie893f56f02c7a76bcfc21c32c10bd1f1d05660e7
2020-05-20 11:45:06 +00:00
Qweder93
49ad90dcd8 storagenode/reputation: unknown_score (unknown_alpha / unknown_alpha+unknown_beta) added to reputation stats, https://storjlabs.atlassian.net/browse/SG-326
Change-Id: I0b29ad736f7a11c7e57a846b6891f4b40755aa48
2020-05-20 11:25:14 +00:00
Egon Elbre
941d10cbc3 private/testplanet: remove Peer.Local()
Currently storagenode depends on overlay.NodeDossier, this is the first
step in removing it.

Change-Id: I034a3f1601835f8349bd41752455022e19bcc707
2020-05-20 11:05:34 +00:00
Egon Elbre
94b2b315f7 storagenode/trust: refactor GetAddress to GetNodeURL
Most places now need the NodeURL rather than the ID and Address
separately. This simplifies code in multiple places.

Change-Id: I52621d8ca52296a8b5bf7afbc1001cf8bfb44239
2020-05-20 11:05:15 +00:00
Egon Elbre
ed627144ed all: use DialNodeURL throughout the codebase
Change-Id: Iaf9ae3aeef7305c937f2660c929744db2d88776c
2020-05-20 10:36:30 +00:00
littleskunk
ef2671927d
storagenode/piecestore: move queue size defaults (#3881) 2020-05-15 19:10:26 +02:00
Qweder93
ef87192120 storagenode/notifications: tests fixed: added time interval between inserts so created_at fields are different when running tests on Windows
Change-Id: I26ba059ab58d0216122ab2f49ae85f7ce7cfced4
2020-05-14 05:26:18 +00:00
Ethan
159df8b2e4 Add logging listener for retrieving and setting log levels
See https://storjlabs.atlassian.net/browse/SM-752

These changes allow us to change the log level at runtime through a handler off of the debug endpoint.

Examples of changing the log level on storj-sim

To get the current level for the satellite api process:
curl -XGET 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain'

To change the log level:
curl -XPUT 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"error"}'

Change-Id: I05d164b290929fa06b6d78c01075ee41f8238044
2020-05-12 16:38:06 -04:00
Kaloyan Raev
e0cf9ae888 storagenode/orders: set devDefault sender interval to 30s
Change-Id: Ic2877da12b5985b73d1565e8ccd65bc817a76448
2020-05-11 18:55:56 +03:00
Egon Elbre
ec589a8289 all: fix comments about grpc
Change-Id: Id830fbe2d44f083c88765561b6c07c5689afe5bd
2020-05-11 13:05:34 +03:00
Egon Elbre
7d29f2e0d3 all: remove drpc wrappers
Change-Id: I45016f7d2a771dc00776196c1f531f3343e93b40
2020-05-11 08:20:34 +03:00
Egon Elbre
e6d5ce6b77 all: remove grpc
It seems everyone has migrated to drpc.

Change-Id: Ica6b2d0bdef68c6603083f2963458843eca71e9e
2020-05-10 06:36:09 +00:00
Yaroslav Vorobiov
d8bdc60e19 storagenode/console: remove last ping id and address
Change-Id: I001e7d31d50aee7b9a07608158f8bd5c407c9c45
2020-05-07 13:34:29 +03:00
Jeff Wendling
efee3563a8 storagenode/storagenodedb: fix joined_at migration
Change-Id: I9091741bbec6e7a2fc6addc23249a40176ec040a
2020-05-05 15:01:07 -06:00
Qweder93
b754806b23 storagenode/reputation: unknown_audit_reputation_alpha and beta added to db, and reputation endpoint
Change-Id: I5d92268851bb9a202cc5d6b4d403467e6f692726
2020-05-05 15:48:04 +00:00
Qweder93
8a443bc9f5 storagenode/console: api notifications tests added
Change-Id: I472abc22746ea95841dcb0ae72dacee06fd98a2e
2020-05-05 17:53:19 +03:00
Qweder93
16cd9b06ec storagenode/heldamount: added api for heldamount history separated by periods
Change-Id: I170010364269822848bc6cd051e0e0fb3df95d91
2020-05-05 12:29:44 +00:00
Qweder93
7a83473f00 storagenode/console: API /dashboard disk-space data separated for UsedForPieces, Trash, Available
Change-Id: I7411da2c92c72a24af98e007efe476d8a023db82
2020-05-04 12:36:18 +00:00