Commit Graph

727 Commits

Author SHA1 Message Date
Egon Elbre
12055e7864 all: minor cleanups
Change-Id: I4248dbe36a62a223b06135254b32851485a2eec1
2020-12-16 10:47:46 +00:00
Qweder93
12144a600b storagenode/console: payout tests and heldhistory joined_at rounding added
Change-Id: I1d43620fbafbf7ed92588b84cb9c6b8ced8832ef
2020-12-14 19:35:04 +02:00
Qweder93
2f62cdf491 storagenode/console: diskSpaceInfo extended with overused diskspace, getDashboardData updated.
Change-Id: I44db26661a8dfb45b5d8e9fcb7511f63deb88cad
2020-12-08 14:55:55 +00:00
Stefan Benten
494bd5db81
all: golangci-lint v1.33.0 fixes (#3985) 2020-12-05 17:01:42 +01:00
Jessica Grebenschikov
b261110352 satellite/orders: get bucketID from encrypted metadata in order instead of serial_numbers table
We want to stop using the serial_numbers table in satelliteDB. One of the last places using the serial_numbers table is when storagenodes settle orders, we look up the bucket name and project ID from the serial number from the serial_numbers table.

Now that we have support to add encrypted metadata into the OrderLimit, this PR makes use of that and now attempts to read the project ID and bucket name from the encrypted orderLimit metadata instead of from the serial_numbers table. For backwards compatibility and to ensure no errors, we will still fallback to the old way of getting that info from the serial_numbers table, but this will be removed in the next release as long as there are no errors.

All processes that create orderLimits must have an orders.encryption-keys set. The services that create orderLimits (and thus need to encrypt the order metadata) are the satellite apiProcess, the repair process, audit service (core process), and graceful exit (core process). Only the satellite api process decrypts the order metadata when storagenodes settle orders. This means that the same encryption key needs to be provided in the config for the satellite api process, repair process, and the core process like so:
orders.include-encrypted-metadata=true
orders.encryption-keys="<"encryptionKeyID>=<encryptionKey>"

Change-Id: Ie2c037971713d6fbf69d697bfad7f8b672eedd66
2020-12-01 15:29:32 +00:00
Egon Elbre
aeb801604e {satellite,storagenode}/orders: fix flaky tests
Before manipulating order information on storagenodes we need to wait
for the orders to propagate to the database. Some of that happens
async with uplink.

Change-Id: Iaacfd7db0909ab5d2831d06388e5fb27b6d4778f
2020-11-18 13:44:02 +00:00
Moby von Briesen
41d86c0985 storagenode/orders/ordersfile: Add reasonable size caps for orders/limits when detecting file corruption.
Define constants of 32 KiB as the upper limit of the marshalled order
and limit protobuf sizes. This value gives lots of buffer in case the
protobufs ever change, but is not as extreme as what we were doing
before in V0 files, which was to use the Uint32 max value.

Change-Id: I0914d17dde3b044b2611af33f931d46d55f81e98
2020-11-18 12:33:26 +00:00
Qweder93
a17cd9aa3e storageode/apikey: added service, CLI issue api key
Change-Id: I840cd0fdbd8dca884eefbd111f21fd3990c11e68
2020-11-18 10:40:17 +00:00
Ivan Fraixedes
fa95c6bbb9
storagenode/orders/ordersfile: Fix error message wrong var
Fix the error message reported by a wrong order size due to passing the
wrong variable to the interpolation pattern.

Change-Id: Ic0059615c60cfa33a26d4aeb0ebda5e586f0df05
2020-11-17 15:22:27 +01:00
Ivan Fraixedes
9740da6508 storagenode/orders: Don't panic if size is over MaxInt32
`make` built function to build a new slice with a negative
length panics.
`make` length parameter is of `int` type.

These changes avoid that `make` panics on 32 bits architecture due to
the fact that `int` type is a `int32` an uint32 value can be over the
maximum `int32`, and when that happens the length parameter value
becomes negative and makes `make` to panic.

Change-Id: Ife9ab5993916d6dcf5584b37c208272269cb2b45
2020-11-17 10:35:21 +00:00
Qweder93
c409194d43 storagenode/payouts: estimation payout heldamount rounding removed
Change-Id: I9fdc7cda15de0df8875436b0b376f0e6479d3aeb
2020-11-17 10:06:11 +00:00
Cameron Ayer
48d8114b3f satellite/contact: treat pingback failure as error
If the satellite fails to pingback the storage node during CheckIn
an error message is returned to the node in the response, but the actual
error value returned is nil. We are only checking the error. This means
the node has no feedback about the failure, and the node also does not
attempt to retry the connection.

Change-Id: Iaed00e422ba91af573e72255cc6671ea97928eae
2020-11-16 18:26:37 +00:00
Moby von Briesen
db480e6e1b storagenode/orders: Improve performance of handling corrupt orders.
This change fixes two things which can make reading from a corrupted
orders file inefficient.
* When a corrupted order is detected, but the underlying error is an
UnexpectedEOF (as opposed to a pb.Unmarshal error, for instance), there
is no point in attempting to read from the file another time to find an
additional uncorrupted order - we will continue to get UnexpectedEOF
errors until we seek to the very end of the file and get a normal EOF.
Instead, when UnexpectedEOF occurs, log and send metrics as with other
types of corruption, but do not attempt to read again.
* When a corrupted order is detected, instead of seeking forward only
one byte for the next attempt, seek forward by the size of entryHeader.
This cuts down on the number of iterations needed to find an uncorrupted
order after detecting a corrupted one.

Change-Id: Ie1a613127e29d29318584ec7f60e8f7554f73487
2020-11-16 14:08:36 +00:00
Cameron Ayer
5a337c48ec {cmd,private,storagenode}: create storage dir verification during setup
Previously, we created a new file to use for directory verification
every time the storage node starts. This is not helpful if the storage node
points to the wrong directory when restarting. Now we will only create the file
on setup. Now the file should be created only once and will be verified at
runtime.

Change-Id: Id529f681469138d368e5ea3c63159befe62b1a5b
2020-11-11 11:01:36 -05:00
Egon Elbre
b892a00143 mod: bump dependencies and reenable test
We shouldn't have any EOF issues with recent drpc fix, let's reenable
and see whether it's still flaky.

Change-Id: I0de312bcb087c7f70ec9d3281d73d86f971845d5
2020-11-10 10:32:21 +00:00
Moby von Briesen
db6bc6503d satellite/metainfo: Update metainfo RS config to more easily support multiple RS schemes.
Make metainfo.RSConfig a valid pflag config value. This allows us to
configure the RSConfig as a string like k/m/o/n-shareSize, which makes
having multiple supported RS schemes easier in the future.

RS-related config values that are no longer needed have been removed
(MinTotalThreshold, MaxTotalThreshold, MaxBufferMem, Verify).

Change-Id: I0178ae467dcf4375c504e7202f31443d627c15e1
2020-11-09 22:16:13 +00:00
Qweder93
8dc10e32ad stefan benten satellited added to historical payout data
Change-Id: I1177b2d2ef10d514f7d401e29891fa7dd964e9ac
2020-11-09 15:43:41 +00:00
Egon Elbre
7183dca6cb all: fix defers in loop
defer should not be called in a loop.

Change-Id: Ifa5a25a56402814b974bcdfb0c2fce56df8e7e59
2020-11-02 15:06:38 +02:00
Egon Elbre
fd8e697ab2 {satellite,storagenode}/internalpb: use specific package name
Ensure we don't register types with the same name into protobuf.

Change-Id: I53d025863fff8c91a067ca5819befa87eb5e35bb
2020-10-30 17:31:08 +02:00
Egon Elbre
1903b15474 storagenode/internalpb: move gracefulexit.proto
Change-Id: Ia3614846ed49a39c8f39331516d16d45a695240b
2020-10-30 15:24:56 +02:00
Egon Elbre
cda67a659a storagenode/internalpb: move inspector.proto
Change-Id: I951379c3b2ff00d1bc09d6a49c026a7e723432d6
2020-10-30 14:51:26 +02:00
Qweder93
f5ba8b8009 storagenode/suspensions: added offline-suspension notificatio chore + tests
Change-Id: I2521cd2e7d08a1dd379e717a554a026c7508c18f
2020-10-29 19:44:22 +02:00
Egon Elbre
e0dca4042d all: add pprof labels for debugger
By using pprof.Labels debugger is able to show service/peer names in
goroutine names.

Change-Id: I5f55253470f7cc7e556f8e8b87f746394e41675f
2020-10-29 15:10:07 +00:00
Qweder93
624255e8ba storagenode/secret: db tests added, small renaming fixes added
Change-Id: I7eae1a9a64c20a39c97e81fa741cfc9b9e1e615a
2020-10-29 14:23:04 +02:00
Egon Elbre
caefde6b32 private/{dbutil,tagsql}: pass ctx to database opening
Database opening usually dial and hence we should pass ctx to them.

Change-Id: Iaa2875981570d83e65be3710f841cf30349f807b
2020-10-29 10:51:29 +00:00
Egon Elbre
89ce1fe626 storagenode/storagenodedb: add ctx to OpenNew and OpenExisting
Database opening usually dial and hence we should pass ctx to them.

Change-Id: I9160ae95829f22f347bd525904898a47279a7427
2020-10-29 09:52:37 +02:00
Egon Elbre
76f4619a9c {satellite,storagenode}/gracefulexit: ensure client is closed
Change-Id: I576a955a5578caf7fcbee832beca28cef2b0c83e
2020-10-27 23:27:07 +02:00
Moby von Briesen
2fbb4095b2 storagenode/orders/ordersfile: Handle remaining pb.Unmarshal errors
Missed one case of Unmarshal in the previous commit for V0 files (0f4e4969b7)
In V1, unmarshalling was being attempted before the checksum was
verified, so this commit moves those calls to the end of the V1 ReadOne
function.

Change-Id: Ic0b49f0bbc91fb61fb28af6003060994d0af22ed
2020-10-26 20:27:05 +00:00
Moby von Briesen
53ba01b1f1 storagenode/orders/ordersfile/v0.go: Return ErrEntryCorrupt on pb.Unmarshal failure
In V0 orders files, unexpected EOF is correctly treated as a file
corruption, but pb.Unmarshal can also fail, and this is not treated as a
file corruption. This commit fixes that.

Change-Id: I6b446a10f4b1a5a44e832cbcc9bf8b2548cfcfeb
2020-10-26 17:38:22 +00:00
Jessica Grebenschikov
f5880f6833 satellite/orders: rollout phase3 of SettlementWithWindow endpoint
Change-Id: Id19fae4f444c83157ce58c933a18be1898430ad0
2020-10-26 14:56:28 +00:00
paul cannon
76d4977b6a storagenode/gracefulexit: logic moved from worker to service
Change-Id: I8b12606a96b712050bf40d587664fb1b2c578fbc
2020-10-22 23:19:30 +00:00
Jessica Grebenschikov
89bdb20a62 storagenodedb/orders: select unsent satellite with expiration
In production we are seeing ~115 storage nodes (out of ~6,500) are not using the new SettlementWithWindow endpoint (but they are upgraded to > v1.12).

We analyzed data being reported by monkit for the nodes who were above version 1.11 but were not successfully submitting orders to the new endpoint.
The nodes fell into a few categories:
1. Always fail to list orders from the db; never get to try sending orders from the filestore
2. Successfully list/send orders from the db; never get to calling satellite endpoint for submitting filestore orders
3. Successfully list/send orders from the db; successfully list filestore orders, but satellite endpoint fails (with "unauthenticated" drpc error)

The code change here add the following to address these issues:
- modify the query for ordersDB.listUnsentBySatellite so that we no longer select expired orders from the unsent_orders table
- always process any orders that are in the ordersDB and also any orders stored in the filestore
- add monkit monitoring to filestore.ListUnsentBySatellite so that we can see the failures/successes

Change-Id: I0b473e5d75252e7ab5fa6b5c204ed260ab5094ec
2020-10-21 15:02:23 +00:00
littleskunk
77d54ff0ac
storagenode/bandwidthdb: Use existing indexes (#3949)
* storagenode/bandwidthdb: Use existing indexes
2020-10-20 22:48:40 +02:00
Qweder93
9df74338a8 storagenode: secret db and service added
Change-Id: I91257e5adc4fc6711653f30c118e476ed1c95b6b
2020-10-16 13:24:33 +00:00
NickolaiYurchenko
7c275830a1 web/storagenode: gross total added to historical data, with surge moved
WHAT:
changed estimation table row order.

WHY:
to show gross total for selected period to avoid misunderstanding
when held amount is bigger than paid multiple times.

Change-Id: I03881c8af682372139a378030acf04f199d3260b
2020-10-16 13:26:28 +03:00
Yaroslav Vorobiov
139a7ee959 private/migrate: add ablity to create dbs during migration
Use tagsql.DB pointer as step database, to propagate changes
back and forth between actual database and migration.
Adds CreateDB operation to the migration step to be able to
create new dbs before executing migration action.
Adjusts storagenode database migration to use inner tagsql.DB
pointer of each database as step.DB.
Adjusts satellite dabase migration, adds proxy migrationDB field
to satellite db that wraps itself as tagsql.DB, pointer of which
is used as step.DB.

Change-Id: Ifed4de5b01a356cf7b37db64d2eaeb7b61982c5c
2020-10-15 15:28:04 +03:00
Moby von Briesen
aa86c0889c storagenode/console: Add current storage used per satellite to storagenode api
Right now, the best way for a storage node operator to get the current
space used for each satellite is to run the `storagenode exit-satellite`
command for graceful exit, and cancel at the second confirmation prompt.
This is convoluted and the data is readily available from the Blobs
Usage Cache.

This change adds the current space used by each satellite to the
endpoints `/api/sno` and `/api/sno/satellite/<Satellite ID>`

Change-Id: I2173005bb016fc76db96fd598d26b485e5b2aa0b
2020-10-14 21:30:28 +00:00
Moby von Briesen
02cbf1e72a storagenode/orders: Add V1 orders file
V1 allows the storagenode to continue reading orders from an
unsent/archived orders file, even if orders in the middle are corrupted.

Change-Id: Iea4117d55c05ceeb77f47d5c973e5ba95da46c66
2020-10-14 15:04:33 +00:00
Egon Elbre
cf2dd76db7 cmd/satellite: proper log usage
log.Fatal immediately terminates the program without running any defers.
We should properly close all the services and databases.

Change-Id: I5e959cef3eafedeacb3a2062e3da47e8d04e8e75
2020-10-13 16:56:35 +03:00
Egon Elbre
2268cc1df3 all: fix linter complaints
Change-Id: Ia01404dbb6bdd19a146fa10ff7302e08f87a8c95
2020-10-13 15:59:01 +03:00
Egon Elbre
0bdb952269 all: use keyed special comment
Change-Id: I57f6af053382c638026b64c5ff77b169bd3c6c8b
2020-10-13 15:13:41 +03:00
Stefan Benten
c1ca470e7e storagenode/orders: fix import and cleanup go.mod and go.sum
Accidentally we imported the wrong monkit package with a previous
commit and made our go.mod and go.sum file unclean.
This should fix it.

Change-Id: I4c3c8b696f59cfd06dc2d5436bb7aea2805936ce
2020-10-09 00:04:57 +02:00
Moby von Briesen
3209effeb6 storagenode/orders: Increase order sending interval from 5m to 1h
Since storage nodes check to see if any order files can be sent every 5
minutes, every storage node attempts to send orders to the satellite
within 5 minutes of each hour since this is when the files become
"available" to send. It is placing a lot of load on our satellite and
storage nodes are not being paid out properly due to timeouts during
order sending due to the increased satellite load.

Change-Id: I44d991b5884b8c11e8a3856d39aee8323f086b51
2020-10-08 12:51:21 -04:00
Moby von Briesen
fbf2c0b242 storagenode/orders: Refactor orders store
Abstract details of writing and reading data to/from orders files so
that adding V1 and future maintenance are easier.

Change-Id: I85f4a91761293de1a782e197bc9e09db228933c9
2020-10-06 15:28:07 -04:00
Qweder93
664b8f6821 storagenode/payout: estimation payout values switched from int64 to float64 to avoid incorrect rounding.
float64 values rounding to 2nd sign after dot.

Change-Id: Ice49f6a0944231ea6adb3343545bf1a62ff6dbc1
2020-10-02 11:33:43 +00:00
Qweder93
245986d528 negative space calculations fix removed
Change-Id: I342c61856fce6d02dc99fd27fd3d563540f22b64
2020-09-30 14:08:24 +00:00
Yaroslav Vorobiov
a840cb71e7 storagenode: check db version before run
Change-Id: I912f63fd62f2bff10341346c28dfb92fcd683806
2020-09-30 10:58:09 +00:00
Michal Niewrzal
cd2a5484f3 storagenode/console: ignore untrusted satellite while returning
dashboard data and calculating satellites data

Change-Id: I71d596891477e0839863e007689b6e2e6e420a22
2020-09-29 18:27:49 +00:00
Yaroslav Vorobiov
8786e55a78 storagenode/storagenodedb: allow existing dbs on setup
Allow existing storagenode dbs on setup to be able to reinstall
the node with existing data.

Change-Id: Ib42ab585432e61dfecc10640b6cd755ce83f0c46
2020-09-28 16:31:48 +03:00
nerdatwork
870abd8676
storagenode/pieces: tidying trash log 2020-09-24 11:55:06 +03:00
Moby von Briesen
8287e3a32d storagenode/orders/store.go: combine writeLimit/writeOrder operations
Combine store.writeLimit and store.writeOrder into
store.writeLimitAndOrder, which only requires a single call to
file.Write(). This simplifies code, but it also reduces the likelihood
of multiple calls to Write() increasing the likelihood of file
corruption.

Also combine the corresponding readLimit/readOrder functions for
consistency.

Change-Id: I62ed406fa2c02708465a678d18293f510f666440
2020-09-22 17:53:12 +00:00
nerdatwork
54dd430048
storagenode/pieces: fix typo for satellite id and piece id 2020-09-22 08:19:12 +03:00
nerdatwork
96ec44ff1b
storagenode/pieces: make log more legible 2020-09-18 15:10:13 +03:00
Qweder93
8182fdad0b storagenode: heldamount renamed to payouts, renamed some methods and structs to more meaningful names. grouped estimated payout with pathouts
satellite: heldamount renamed to SNOpayouts.

Change-Id: I244b4d2454e0621f4b8e22d3c0d3e602c0bbcb02
2020-09-16 14:57:35 +00:00
Moby von Briesen
7db5794c16 storagenode/orders/store: Do not lock order enqueues for entire duration of ListUnsentBySatellite
We only need to lock aquire mutexes inside ListUnsentBySatellite when we
want to determine whether a file has an active enqueue in progress.
On some nodes, ListUnsentBySatellite can take a particularly long time, having
undesired side-effects, so if we can minimize locking time, those nodes
will be better off.

Also, lock archive mu during ListUnsentBySatellite so files cannot be
archived and listed at the same time.

Change-Id: Ieb7e2a759c20c724a74dd8315728c873ccab14a3
2020-09-15 15:15:30 +00:00
Qweder93
528aa76ae6 storagenode/payouts: payoutHistoryMonthly surge reworked, empty receipt now won't return error
Change-Id: If99f8aec102550cd30e5906f986a4417903100be
2020-09-14 18:19:17 +03:00
Moby von Briesen
789b07e226 storagenode/orders/store.go: Do not return error from ListUnsentBySatellite when order files are corrupted.
If we see an UnexpectedEOF error when attempting to read orders, return
the orders we have been able to read successfully and do not return an
error. This behavior ensures that the storagenode orders service
attempts to archive corrupted files and does not retry them repeatedly
and get stuck.

Change-Id: I0d00d1e174f968af6e99ca861eddad190f1339e2
2020-09-10 23:36:05 +00:00
Qweder93
ac29d80495 storagenode: heldamount GetPaystub refactored, estimationPayouts logic separated form console to separate service, storagenodeapi tests fixed.
Change-Id: I902823ef40a62861ce32799e9fb7a67a1e14710d
2020-09-09 15:31:16 +00:00
Stefan Benten
179b5adad4 storagenode/orders: add missing mon.Task parameter
Change-Id: If98cf347a81f29698a6bdb0907520d60f71db433
2020-09-06 00:05:53 +00:00
Jennifer Johnson
4e2413a99d satellite/satellitedb: uses vetted_at field to select for reputable nodes
Additionally, this PR changes NewNodeFraction devDefault and testplanet config from 0.05 to 1.
This is because many tests relied on selecting nodes that were reputable based on audit and uptime
counts of 0, in effect, selecting new nodes as reputable ones.
However, since reputation is now indicated by a vetted_at db field that is explicitly set
rather than implied by audit and uptime counts, it would be more complicated to try to
update all of the nodes' reputations before selecting nodes for tests.
Now we just allow all test nodes to be new if needed.

Change-Id: Ib9531be77408662315b948fd029cee925ed2ca1d
2020-09-04 16:45:32 +00:00
Michal Niewrzal
aa47e70f03 satellite/metainfo: use metabase.SegmentKey with metainfo.Service
Instead of using string or []byte we will be using dedicated type
SegmentKey.

Change-Id: I6ca8039f0741f6f9837c69a6d070228ed10f2220
2020-09-03 15:11:32 +00:00
Qweder93
36d752e92d storagenode/reputation: offline_under_review_at added
Change-Id: Ia7ec79b2d6f20fe29de0c36223f9485380d2845c
2020-09-02 18:48:28 +03:00
Qweder93
7d9897b7af storagenode/nodestats: online_score added
Change-Id: I84b50a6cace306e5f10d53a2073fe8810d4d2960
2020-09-02 17:45:01 +03:00
JT Olio
1f711523d5 satellite/repair: switch to piecestore.UploadReader part 2
Change-Id: I5a91d2960b037c7a3c96d01bc40404316ba028e3
2020-09-01 12:40:54 -06:00
JT Olio
b872fe52a1 satellite/repair: switch to piecestore.UploadReader
Change-Id: Ia99ad2cf5422e6ba1d98b32946740f9cadba7b6d
2020-09-01 09:26:54 -06:00
Cameron Ayer
ca0c1a5f0c storagenode/{monitor,pieces}, storage/filestore: add loop to check storage directory writability
periodically create and delete a temp file in the storage directory
to verify writability. If this check fails, shut the node down.

Change-Id: I433e3a8d1d775fc779ae78e7cf3144a05ffd0574
2020-08-31 21:20:49 +00:00
nerdatwork
e072febbcc
Fixed typo in log for allocated space (#3934) 2020-08-29 16:36:37 +02:00
Egon Elbre
c86c732fc0 satellite: simplify tests
satellite.DB.Console().Projects().GetAll database query
can be replaced with planet.Uplinks[0].Projects[0].ID

Change-Id: I73b82b91afb2dde7b690917345b798f9d81f6831
2020-08-28 22:28:04 +00:00
Egon Elbre
3ca405aa97 satellite/orders: use metabase types as arguments
Change-Id: I7ddaad207c20572a5ea762667531770a56fd54ef
2020-08-28 15:52:37 +03:00
Qweder93
c4a4745dd8 storagenode/console: audit per satellite now uses satelliteName instead of satelliteID
Change-Id: I8221ec840f654a62aedfb62a4194616db890f539
2020-08-25 12:52:47 +00:00
Qweder93
f16cf5cccf storagenode/console & /inspector: added recalculation of disk space info
Change-Id: Id003d031a6464ec095c31290fd6a756ead644261
2020-08-25 14:19:10 +03:00
Egon Elbre
f0ef01de5b storagenode/gracefulexit: retry workers faster
Change-Id: Ica20a691ff117a2b36a6362ee1fed21ce49a9ac1
2020-08-24 12:27:27 +03:00
Egon Elbre
e6bea41083 Revert "gracefulexit: reconnect added"
This reverts commit cff44fbd19.

Change-Id: I6590f483493e308b8244151e1df7570fd32ca2f8
2020-08-23 18:11:24 +03:00
Qweder93
cff44fbd19 gracefulexit: reconnect added
Change-Id: I236689af944effe3e79ef92e852ae264d3b372e5
2020-08-22 14:59:46 +03:00
Moby von Briesen
68b67c83a7 storagenode/{orders,piecestore}: Always unlock unsent orders file, even with an empty order.
When we call ordersStore.BeginEnqueue, the unsent orders file for that
satellite and hour is prevented from being sent. It is freed when the
commit callback returned by BeginEnqueue is used. This change ensures
that we always call the commit callback, even when we have an empty
order or an order with Amount <= 0.

Change-Id: Ic4678f7eaa1e6957dd77d4bb5a23bb35d25b1e93
2020-08-21 11:35:31 -04:00
littleskunk
db57d76ee9
storagenode/gracefulexit: fix wrong error handling for corrupted pieces (#3930) 2020-08-21 11:35:03 +02:00
Jeff Wendling
91698207cf storagenode: live tracking of order window usage
This change accomplishes multiple things:

1. Instead of having a max in flight time, which means
   we effectively have a minimum bandwidth for uploads
   and downloads, we keep track of what windows have
   active requests happening in them.

2. We don't double check when we save the order to see if it
   is too old: by then, it's too late. A malicious uplink
   could just submit orders outside of the grace window and
   receive all the data, but the node would just not commit
   it, so the uplink gets free traffic. Because the endpoints
   also check for the order being too old, this would be a
   very tight race that depends on knowledge of the node system
   clock, but best to not have the race exist. Instead, we piggy
   back off of the in flight tracking and do the check when
   we start to handle the order, and commit at the end.

3. Change the functions that send orders and list unsent
   orders to accept a time at which that operation is
   happening. This way, in tests, we can pretend we're
   listing or sending far into the future after the windows
   are available to send, rather than exposing test functions
   to modify internal state about the grace period to get
   the desired effect. This brings tests closer to actual
   usage in production.

4. Change the calculation for if an order is allowed to be
   enqueued due to the grace period to just look at the
   order creation time, rather than some computation involving
   the window it will be in. In this way, you can easily
   answer the question of "will this order be accepted?" by
   asking "is it older than X?" where X is the grace period.

5. Increases the frequency we check to send up orders to once
   every 5 minutes instead of once every hour because we already
   have hour-long buffering due to the windows. This decreases
   the maximum latency that an order will be reported back to
   the satellite by 55 minutes.

Change-Id: Ie08b90d139d45ee89b82347e191a2f8db1b88036
2020-08-19 19:42:33 +00:00
Cameron Ayer
0155c21b44 private/testplanet, storagenode/{monitor,pieces}: write storage dir verification file on run and verify on loop
On run, write the storage directory verification file.

Every time the node runs it will write the file even if it already exists.
The reason we do this is because if the verification file is missing, the SN
doesn't know whether it is an incorrect directory, or it simply hasn't written
the file yet, and we want to keep nodes running without needing operator intervention.

Once this change has been a part of the minimum version for several releases,
we will move the file creation from the run command to the setup
command. Run will only verify its existence.

Change-Id: Ib7d20e78e711c63817db0ab3036a50af0e8f49cb
2020-08-19 19:12:21 +00:00
Cameron Ayer
586e6f2f13 private/testblobs, storage, storage/filestore: add storage dir verification to filestore
Sometimes SNOs fail to properly configure or lose connection to their storage directory
which can result in DQ. This causes unnecessary repair and is unfortunate for all parties.

This change introduces the creation of a special file in the storage directory at runtime
containing the node ID. While the storage node runs, it periodically verifies that it can
find said file with the correct contents in the correct location. If not, the node will
shut down with an error message.

This change will solve the issue of nodes losing access to the storage directory, but it will not
solve the issue of nodes pointing to the wrong directory, as the identifying file is created each
time the node starts up. After this change has been the minimum version for a few releases, we will
remove the creation of the directory-identifying file from the storage node run command and add it
to the setup command.

Change-Id: Ib7b10e96ac07373219835e39239e93957e7667a4
2020-08-19 17:18:14 +00:00
Moby von Briesen
708cb48aa6 storagenode/orders: implement orders filestore on storagenode
* Add all new orders to the orders filestore instead of the database.
* Submit orders from the filestore to the new satellite SettleWindow
endpoint.

The orders filestore will eventually replace the orders DB completely.
For now, we will still be checking the orders DB and submitting those
orders if they exist. In a later release, we will completely remove the
orders DB, but we need both the DB and filestore for the transitionary
period.

Change-Id: Iac8780fd5ab770296181bbd313e1d335f072d4dc
2020-08-19 15:00:35 +00:00
Ethan
5445d595c0 storagenode/gracefulexit: Wait for the worker delete and transfer goroutines to finish before completing the exit
A failed test showed the same piece being deleted twice. This happens if the graceful exit completes before a previous piece deletion finishes. This change adds a "wait" on the limiter before executing the delete all step when GE is done.

Change-Id: I1c8c49d1e501c2728c80d4224a4854e742be27da
2020-08-19 14:20:26 +00:00
Egon Elbre
be3fd0147e storagenode/storagenodedb: database name in all preflight errors
Shorten the error strings and include database name in all potential
preflight errors.

Change-Id: Ic92ca1ec6e14ffbddb0a0cf89e357eec9532d27e
2020-08-18 16:31:19 +03:00
NickolaiYurchenko
4cdba365ef web/storagenode: payout history table
Change-Id: I448ea8424baf31400d9868ef9ca2b8002caa7bbd
2020-08-13 12:05:56 +00:00
Egon Elbre
94a09ce20b all: add missing dots
Change-Id: I93b86c9fb3398c5d3c9121b8859dad1c615fa23a
2020-08-11 17:50:01 +03:00
Qweder93
4ee1b2d45a storagenode/console: added list of all audits per satellite to sno dashboard/satellites
Change-Id: I52e58748d6467f372d9a308347fc77e400d137e2
2020-08-10 12:55:07 +00:00
Qweder93
373934efb2 storagenode/heldamount: payout history: removed extra doubling with surge percent, added held percent
Change-Id: Idd3927c3130bff771e5437b9b18b4a4907f787e4
2020-08-10 15:29:34 +03:00
Qweder93
f804f03b1f storagenode/heldamount: payuout history updated
Change-Id: I6dc91e9eed51f9b81af3e47a45168c43d254356a
2020-08-05 09:58:34 +00:00
Qweder93
53a5d18e1a storagenode: fixed logging about piece being moved to trash, and added logging when piece was actually deleted
Change-Id: I46f6a141b27033c2087b5c4681506d80b90f4a18
2020-08-02 20:00:05 +03:00
Qweder93
b4c9badab1 storagenode/console: estimation payout fix
Change-Id: I5d9f11fffd74978f3ca684fd08aac44a27a83c71
2020-07-27 21:41:07 +03:00
Qweder93
5988ad6646 storagenode/heldamount: payout history extended with satellite's id, url, surge percent
Change-Id: I669b7b6073ded48fd5686c587357b6c86a970fc7
2020-07-25 14:30:07 +03:00
Qweder93
123aebd79f storagenode/version: version chore test fix
Change-Id: I61537ea325779cefbb1f8d7c5d373dc4bf80a7aa
2020-07-24 20:17:35 +03:00
Qweder93
f531bc8638 storagenode/heldamount payout-history rout fix, usage_at_rest in estimation payout calculations fixed
Change-Id: I6f819a404a45b2a96c1aae33c67ebea1ab83aef0
2020-07-24 15:19:45 +00:00
Yaroslav Vorobiov
4d2a505788 storagenode/db: explicitly open and create dbs
To prevent storagenode from implicitly recreating missing dbs and storage,
as such behaviour leads to audit failures. Do not allow storagenode to
start if any of dbs or storage is missing, corrupted, or dedicated storage disk is
unmounted, to get downtime instead.

Change-Id: Ic64e1f0ff4d8ef5b2fddbe7a7e53df4f4bd8652e
2020-07-24 14:08:47 +03:00
Qweder93
92efffb48a storagenode/version: notification flow now based on cursor, chore_test added, versioncontrol added to reconfigure.
Change-Id: I70713def8d585228270ec5a8c586ecc5b4d510c4
2020-07-23 14:13:24 +00:00
Cameron Ayer
1f5d5235a6 storagenode/{monitor,piecestore}: if free disk < expected available space, return free disk
We only sync free disk and available space, if necessary, on startup. If the
SNs disk fills up with non-storj data, we will not know about it when reporting
available space to the satellite.

Solution: whenever we check the node's capacity, double check free disk.
If free disk < than expected available space, return free disk.

Change-Id: I66265c16e03be45b6e1f5817c70df7eac0a76455
2020-07-22 15:08:37 +00:00
Qweder93
aa6afc3879 error handling in heldamount cash and collector delete fixed
Change-Id: I8fe58c50f844a6b819eacc14a40bc5c67268ed5c
2020-07-22 12:26:13 +00:00
Qweder93
0949731caa storagenode/console: estimation payout held split from total payout, calculations fixed
Change-Id: I064f473ffeb3a3051c9228d1dd84fe0fc86dd3ef
2020-07-21 15:31:51 +03:00
Egon Elbre
d8dcae3075 all: fix error checking
Change-Id: Ia0da1bbd6ce695139922f94096c2419281905e32
2020-07-16 19:13:14 +03:00
Egon Elbre
e70da5cd4e all: fix comments
Change-Id: I2d2307e3fab87de47a72b3595d051e2c95ff4f8a
2020-07-16 19:13:14 +03:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
Qweder93
dfdf73282d storagenode/heldamount: db tests updated with payout.Receipt
Change-Id: I17699b923c5a4d7decbd446c382f0c886c36d5e1
2020-07-16 12:24:22 +00:00
Moby von Briesen
1b807761bd storagenode/orders: Update orders filestore to be compatible with new satellite endpoint
* Instead of archiving a list of orders and deleting an "unsent" file in
separate steps, archival simply moves the old unsent file to a new
archived file
* Add maxInFlightTime to be used along with grace period for sending
buffer
* Create unsent/archival directories in constructor
* Code cleanup

Change-Id: Ia3bc2aaf60cced6c6d413465423d78c7d5151188
2020-07-15 14:21:56 -04:00
Qweder93
62fec25104 storagenode/heldamount: returns usage_at_rest in tbm instead of tbh
Change-Id: I183a56460ea76a53680ca6861d02cecebe3576ec
2020-07-15 15:46:13 +03:00
stefanbenten
257855b5de all: replace == comparison with errors.Is
Change-Id: I05d9a369c7c6f144b94a4c524e8aea18eb9cb714
2020-07-14 15:50:25 +00:00
Qweder93
7b4a8c4d6d storagenode/heldamount: payoutHistory added
Change-Id: I93dd3d024085d19ecff76075e52bf66796207fd6
2020-07-14 17:35:03 +03:00
Qweder93
7d6973b5a2 satellite: heldamount and nodestats not returning error node not found by rpc
Change-Id: Ifb00b16a4a04603251de60da6a6612fd5e98d597
2020-07-14 16:31:02 +03:00
Egon Elbre
262da14359 storagenode/console/consoleapi: disable flaky TestStorageNodeApi
Change-Id: I076c9a46fece86d34eae117ab84f94f99e7e64e0
2020-07-13 18:35:38 +03:00
Isaac Hess
78f5755d46 storagenode/nodestats: Add sat to heldamount error
Nodes are receiving an error that heldamount rpc doesn't exist on a
satellite. This simply adds which satellite to the error.

Change-Id: I7708e0511b55fdd2425969db2a545645339bad81
2020-07-13 14:29:09 +00:00
Qweder93
facde770de storagenode/heldadmount: removed logging errors node not found during getAllPaystubs/Payments from trusted satellites
Change-Id: I87f6c697d98546812450fcfb090623c76dec4bbc
2020-07-13 16:45:23 +03:00
Qweder93
f73e92c268 storagenode/gracefulexit: added blobs clean
on node's start checks if any of trusted satellites has GE status "Exited successfully"
if so - trying to delete blobs/satellite folder, so no trash left on SNO.

Change-Id: I566266c84f2a872df54cd01bc2f15a9934f138ed
2020-07-13 11:49:18 +00:00
Qweder93
e17243fcd7 storagenode/console: estimation payour for current and previous month reworked
Change-Id: I937d5d8f7c17949b539dcd6e36af27400a5043e2
2020-07-10 12:18:53 +00:00
Qweder93
0521435e08 storagenode/gracefulexit: added deletion of all files left in storage/blobs/satellite after successful GE
https://storjlabs.atlassian.net/browse/SG-368

Change-Id: I29a978fe0d0153aedf2be91dc7f45b4ef386d447
2020-07-08 14:38:31 +03:00
Bill Thorp
a3c902ab84 storagenode/pieces: hours in a month should be 720
Per https://documentation.tardigrade.io/pricing/billing-and-payment:
"The calculation of per object fees is based on a standard 720-hour month."

On most years, the average value is 730 (365*24/12), except leap years.
However, we want to have ours be 720 (30*24) so its lines up with days.

Change-Id: Ifb9691878f1a7ea81ed36c92b37985493295fe31
2020-07-07 15:26:15 -04:00
Moby von Briesen
e9dd5b2845 storagenode/piecestore: Properly log/send metrics for all successful pieces
When an uplink or repair work finishes uploading a piece to a
storagenode, it has no reason to wait another round trip after the piece
is committed to gracefully close the connection - in many (most?) cases,
the connection is simply canceled once the upload is complete. This has
the unintended side effect of producing a lot of "piece canceled" logs
and metrics on the storagenode side, when the reality is that the piece
uploads were successful, and not really canceled. This commit fixes
that.

Change-Id: Icbc1f7857d380134560219c1c19c186df2783cd0
2020-07-07 15:19:17 +00:00
Qweder93
ac716e1514 storagenode/heldamount: payment receipt added to monthly paystub, heldamount.service separated for service and endpoint
Change-Id: Id759586c6362edbef34c230d4f0d2585c11c9b47
2020-07-06 09:51:52 +00:00
Cameron Ayer
35b709ba18 storagenode/storagenodedb: check if db is nil before closing
In the event of an error in storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases
the caller will attempt to close all databases. However, the error prevents the DB from
being opened and set in the proper place. Attempting to close results in a nil pointer
dereference

https://forum.storj.io/t/node-wont-start-after-update-to-v1-6-4-runtime-error-invalid-memory-address-or-nil-pointer-dereference/7889

Change-Id: Ibfe6f3e13c36d9d15a0cb46e384f0120afdab60b
2020-07-02 15:02:38 +00:00
Qweder93
577f72cb92 storagenode/version: notifications added
Change-Id: Ib9720d8124d8e078354a292b644e2db1f5fffe67
2020-07-01 19:35:46 +03:00
NickolaiYurchenko
b878fcc4b2 storagenode/heldamount: id removed from satellite name
Change-Id: Ic524a40930a5fe7673ccce817d6f68c3538e5208
2020-07-01 15:38:05 +03:00
Qweder93
9b90712aa0 storagenode/heldamount: payents added to db
Change-Id: Ib6c486251ca08d34003c35379d10314127edf103
2020-06-30 17:24:35 +03:00
Qweder93
9a02149654 storagenode/heladamount: held history extended with joined_at date, total_held and total_disposed amounts
Change-Id: I41fe9ab8c5667aa988257a94848ea70225305d79
2020-06-30 13:33:25 +00:00
Yingrong Zhao
51dfc6bf4f storagenode/gracefulexit: make minimum transfer speed to be 5KiB
with 128B/sec, a satellite with 10min default timeout could already
closed its connection to a node even though the node was able to
compelete the transfer.

Change-Id: I6173d6473a62c6d0b0e0a8765c1ae0a5e57b0a08
2020-06-23 21:14:18 +00:00
Ivan Fraixedes
98d477effb
storagenode/collector: Fix comment doc
Change-Id: I703dce7d1b7d7653bbea901c798266a0108b9eec
2020-06-19 13:51:23 +02:00
Jeff Wendling
3842118bab storagenode/heldamount: use correct field for repair usage
Change-Id: I1e0d0bd4c416a21d6900fb723185599f58391d8a
2020-06-12 10:55:35 -06:00
Qweder93
2c3fe5597d storagenode/nodestats: unknown_audit_score added to Service
Change-Id: I1f97f6f0eace9858e466a53d4d4eeabe8059e4eb
2020-06-12 14:00:41 +00:00
Qweder93
0826c8d87f satellite/heldamount: fix dimension of usage_at_rest
Change-Id: If1518ad41736912d15fb2c882c9e236c16f85a51
2020-06-11 13:07:51 +00:00
Moby von Briesen
be59727790 storagenode/orders: Add archival functionality to orders filestore
* Allow orders to be archived after being settled successfully with the
satellite.
* Allow for cleanup of orders that were archived before a certain time.
* Rewrite other parts of the orders file store to work better with new
design.

Change-Id: I39bea96d80e66a324ec522745169bd6d8b351751
2020-06-11 08:47:37 +00:00
Qweder93
e52809d53e cmd/storagenode: add check if satellites available to gracefulexit
Change-Id: I8747507593d810bbdec0d140de0600ee147011c3
2020-06-10 13:38:36 +00:00
Moby von Briesen
0b109c32e4 storagenode/piecestore/usedserials: add monkit metric for serials that are randomly deleted
This will give storagenode operators a better idea of whether the memory
allocated to the usedserials store is sufficient.

Change-Id: I5c30f2e39473a573f43409511ad9e2e32680479c
2020-06-09 17:04:37 -04:00
Yaroslav Vorobiov
09ca382abf storagenode/db: preflight improve index discovery
Change-Id: I876b321f6cd4e91dfced87aa4d39f2cf9a8e63d0
2020-06-05 14:03:25 +03:00
Qweder93
7f8e553022 console/dashboard: added pieces headers size to calculations
Change-Id: I0ee8d6bcb9ce9f69d49ebac2b95579166389668e
2020-06-04 16:39:02 +00:00
Moby von Briesen
c8c0a42269 storagenode/orders: begin implementation of file store for order limits
* Will replace order limits database.
* This change adds functionality for storing and listing unsent orders.
* The next change will add functionality for order archival after
submission.

Change-Id: Ic5e2abc63991513245b6851a968ff2f2e18ce48d
2020-06-03 22:47:04 +00:00
Egon Elbre
4d0cb1af7e storagenode/piecestore: verify before checking free disk
Change-Id: I3fe0383f9b13b99ef9d63ff235616ff204cf6d76
2020-06-02 17:49:14 +03:00
Qweder93
89c9672ce0 storagenode/piecestore: available storage check added in Upload
Change-Id: I71e9e5f335d4320d5de8b374fe747fec43179f78
2020-06-01 16:55:22 +00:00
Egon Elbre
07050eea26 all: use common/storj
Change-Id: Id1e36d52f9807b5ffbb72ce73f4b60cb21b68a78
2020-05-29 11:57:32 +03:00
Moby von Briesen
dc57640d9c storagenode/piecestore: switch usedserials db for in-memory usedserials store
Part 2 of moving usedserials in memory
* Drop usedserials table in storagenodedb
* Use in-memory usedserials store in place of db for order limit
verification
* Update order limit grace period to be only one hour - this means
uplinks must send their order limits to storagenodes within an hour of
receiving them

Change-Id: I37a0e1d2ca6cb80854a3ef495af2d1d1f92e9f03
2020-05-28 12:52:52 -04:00
Moby von Briesen
909d6d9668 storagenode/piecestore/usedserials: add in-memory store for used serials
Implement an in-memory store for keeping track of order limit serial
numbers. It automatically deletes items if its size exceeds a configured
limit.
This change is part 1 - it creates the store
In part 2, the in-memory store will replace the usedserials database

Change-Id: I36f540ed809f034a27c1d7cede8a0a8b080af818
2020-05-28 12:52:52 -04:00
paul cannon
7395dd1e6e storagenode/gracefulexit: revalidate existing pieces
..before they are transferred to another node and submitted to the
satellite as successful piece transfers, because if we submit an invalid
signature, the node will be marked as a cheater and disqualified
immediately.

These signatures should have been validated when the piece was
originally stored, but bitrot does happen and needn't be cause for an
immediate DQ.

Change-Id: I8b0ebd5812ea8a2e60766005b7251fbb74ef7857
2020-05-28 09:50:14 -05:00
Qweder93
73214c6d1c storagenode/heldamount: heldhistory reworked to all satellites
Change-Id: I8d7707fddfbdc52d29951a8a002978c7fbb07049
2020-05-28 11:44:26 +00:00
Qweder93
f2a0c64425 storage/filestore: log potential disk corruption
In walkNamespaceWithPrefix log in case of "lstat" error, because this may indicate an underlying disk corruption.

SG-50

Change-Id: I867c3ffc47cfac325ae90658ec4780d213ff3e63
2020-05-27 12:12:55 +00:00
Qweder93
8db848791f storagenode/console: added estimated payout for current month and estimated pay stub for previous month (until there's real data in satellite's table) + heldback percentage rate for previous month.
Change-Id: I9346f6d22ed6fbb7e5346b102fc898467678f384
2020-05-27 14:51:23 +03:00
littleskunk
2fbb34c3ea
nodeselection: Increase minimum free space to 500MB (#3898) 2020-05-25 12:13:28 +02:00
crawter
f5ac678b0a storagenode/satellitesdb: added FK constraint to satelliteID
Change-Id: If5adf2b92627fcf80850670ba672b346320ddd87
2020-05-21 13:01:20 +00:00
Egon Elbre
bef84a5f9d storagenode: remove dependency to overlay.NodeDossier
This is the last dependency from storage node to satellite.

Change-Id: I12f7abb91e84f823ba5af126c6e2979519838612
2020-05-21 08:37:13 +03:00
crawter
2c9afe7f17 storagenode/console/api/helamount: periods with heldamount data endpoint added
Change-Id: Ie893f56f02c7a76bcfc21c32c10bd1f1d05660e7
2020-05-20 11:45:06 +00:00
Qweder93
49ad90dcd8 storagenode/reputation: unknown_score (unknown_alpha / unknown_alpha+unknown_beta) added to reputation stats, https://storjlabs.atlassian.net/browse/SG-326
Change-Id: I0b29ad736f7a11c7e57a846b6891f4b40755aa48
2020-05-20 11:25:14 +00:00
Egon Elbre
941d10cbc3 private/testplanet: remove Peer.Local()
Currently storagenode depends on overlay.NodeDossier, this is the first
step in removing it.

Change-Id: I034a3f1601835f8349bd41752455022e19bcc707
2020-05-20 11:05:34 +00:00
Egon Elbre
94b2b315f7 storagenode/trust: refactor GetAddress to GetNodeURL
Most places now need the NodeURL rather than the ID and Address
separately. This simplifies code in multiple places.

Change-Id: I52621d8ca52296a8b5bf7afbc1001cf8bfb44239
2020-05-20 11:05:15 +00:00
Egon Elbre
ed627144ed all: use DialNodeURL throughout the codebase
Change-Id: Iaf9ae3aeef7305c937f2660c929744db2d88776c
2020-05-20 10:36:30 +00:00
littleskunk
ef2671927d
storagenode/piecestore: move queue size defaults (#3881) 2020-05-15 19:10:26 +02:00
Qweder93
ef87192120 storagenode/notifications: tests fixed: added time interval between inserts so created_at fields are different when running tests on Windows
Change-Id: I26ba059ab58d0216122ab2f49ae85f7ce7cfced4
2020-05-14 05:26:18 +00:00