Commit Graph

671 Commits

Author SHA1 Message Date
Michał Niewrzał
67e26aafcd Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I9b183323cb470185be22f7c648bb76917d2e6fca
2021-03-10 08:53:38 +01:00
Moby von Briesen
0370d1553e storagenode/console: Fix TestStorageNodeApi flakiness
TestStorageNodeApi is failing due to very slight differences in
float values. This change rounds these values to 3 decimal places before
comparing them.

Change-Id: Ic7fae3a5e0a0a942c03d982bfa7b19357f2e3d2e
2021-02-26 15:12:35 -05:00
Qweder93
f2be856864 storagenode/payouts: estimated payouts calculations upgraded
Added recalculations of estimated income in current month in case
node joined in current month

Change-Id: Iac925fd8bcf2be8f1089e5f001a01448ba7650be
2021-02-25 14:11:44 +00:00
NickolaiYurchenko
badd6d12d4 web/satellite: distributed amount added to tables, content changed
Change-Id: Ib0eb3fa3f4f9d451f4b4048a6409d19968a41956
2021-02-24 16:01:40 +00:00
Michał Niewrzał
d995fb497f Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I367da03351ab80f7343332420490dde9282aa47a
2021-02-23 12:31:31 +01:00
Jeff Wendling
3eb321fd33 storagenode/console/consoleapi: fix flaky test
Change-Id: I1af928953a19155e3c2e9f8c63e7162978e7b475
2021-02-17 19:48:55 +00:00
Jeff Wendling
9e7e753faf sn/sndb/payouts: migrate distributed == paid for periods < 2020-12
while satellites have also run this logic, old satellites that
no longer exist cannot and so the node cannot get the updated
data. this locally migrates it so that the calculations for
the undistributed amounts are correct.

there's also some tab/space whitespace and gofmt fixes.

Change-Id: I470879703314fe6541eaba5f21b47849781894f8
2021-02-17 13:07:10 -05:00
Qweder93
dcf7b884ca storagenode/payouts: estimated payouts calculations fixed
Change-Id: I9d5d180aea1606350e0d9214fbf352bf9f76c184
2021-02-17 12:58:41 +00:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Qweder93
a86bd27513 storagenode/storagenodedb: paystubs distributed not null added
Change-Id: I021ba1c4ce4865f2575466d9b4d702775a5654cf
2021-02-10 12:51:09 +00:00
Yaroslav Vorobiov
966535e9de {storagenode,satellite}/nodeoperator: add wallet features
Change-Id: Iac7eb40a52b8fddcc573aebaad2e3a30a10cded9
2021-02-08 22:09:45 +02:00
Qweder93
dc6a8cbcf3 storagenode/suspension: fixed timings of downtime suspension notifications
Change-Id: Ie5b4f369efc0e66910b2f939aa8c5b11dc92946c
2021-02-08 15:24:29 +02:00
Qweder93
cdf58a1ffd multinode/payouts: EarnedPerSatellite added
private/multinodepb: multinode.proto updated

Change-Id: Iaeffd31ae124ac5b2716ca1b2d05e764372e2d94
2021-02-04 18:10:17 +02:00
Egon Elbre
615586a471 {satellite,storagnode}/internalpb: bump gogo to v1.3.2
Change-Id: I585a3197630b9b675a4b540cf25612e1ed39b599
2021-02-04 11:09:13 +00:00
Qweder93
b92a86e524 storagenode/payouts: earned amount at satellite and list of paying satellites IDs added to DB
Change-Id: I66e34d6d80e37481d6eaae4c7bd017a7e096a751
2021-02-04 10:19:08 +00:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Ivan Fraixedes
9c9f481469 satellite/orders: Remove deprecated endpoint
Remove the orders Settlement endpoint because it isn't used and it was
already always returning an error.

Change-Id: I81486fbe7044a1444182173bc0693698ee7cfe7e
2021-02-03 23:47:07 +00:00
Yingrong Zhao
7e80badaf9 pkg/server,pkg/quic: accept an existing conn to create quic listener and
allow disabling tcp/quic

In order to have more control of a server so that we can
simulate connection failures in `testplanet`, this PR changes
quic.Listener to accept an existing UDPConn instead of relying on the
quic-go library to create the UDPConn.
This PR also adds two flags on the `server.Config` struct to allow
enabling/disabling tcp/tls listener and quic listener. By default, they
are both set to true.
    - `DisableTCPTLS`: internal flag, disables tcp/tls listener.
    - `DisableQUIC`: hidden flag, disables quic listener
By making the `DisableQUIC` a hidden flag, it allows storagenode operators to
have the ability to disable quic traffic in case their set up can't work
with udp traffic.

Change-Id: I853b12435d988b9c41ad9b873fd57480d792e378
2021-02-03 12:04:29 -05:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Qweder93
e383722812 storagenode/payouts: Distributed added to paystubs
Change-Id: I02ba1f681be84765a0a5dae7af17d2a6674e7cca
2021-02-02 13:51:43 +00:00
Ivan Fraixedes
d93944c57b satellite/orders: Delete unused methods & DB tables
Delete satellite order methods and DB tables which aren't used anymore
after we have done a refactoring on the orders to stuck bucket
information in the orders' encrypted metadata.

There are also configuration parameters and a satellite chore that
aren't needed anymore after the orders refactoring.

Change-Id: Ida3682b95921df70792284b42c96d2508bf8ca9c
2021-02-01 18:01:29 +00:00
paul cannon
c489a70e62 storagenode/gracefulexit: omit finished exits from ListPendingExits
From the name of the function and from the way it is used (only called
in one place, from "storj.io/storagenode/gracefulexit".(*Chore).Run()),
it should not return graceful exits that have already completed.

In particular, this causes a problem in the case that a node has already
completed a graceful exit from one satellite, after which the satellite
was decommissioned and no longer in the "trusted" list. This causes an
error message to show up in the node logs every single minute like
"failed to get satellite address ... satellite \"X\" is untrusted".

https://forum.storj.io/t/error-gracefulexit-service-failed-to-get-satellite-address/11372

This change causes ListPendingExits to list pending exits only, not all
exits.

Correspondingly, the check for whether an exit is already completed, in
(*Chore).Run(), becomes unnecessary and is here removed.

Change-Id: Ia3e9bb3e92be4a32ebcbda0321e3fe61d77deaa8
2021-02-01 15:28:50 +00:00
nerdatwork
74e293693e
storagenode/gracefulexit: improve error message 2021-02-01 15:09:18 +02:00
Kaloyan Raev
d0612199f0 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/config.go
	satellite/metainfo/metainfo_test.go

Change-Id: I95cf3c1d020a7918795b5eec63f36112fdb86749
2021-02-01 14:32:12 +02:00
Egon Elbre
5d895fb404 storagenode/{payouts,console}: use same time for all calculations
When using calling time.Now() multiple times, they can cross
month boundary causing errors in calculations.

Change-Id: I66b5be7598f3bf475b4b5fe0dcce82eee55b3134
2021-02-01 13:59:50 +02:00
Egon Elbre
8a3db08f68 storagenode/payouts/estimatedpayouts: fix calculations
Change-Id: Iaa01ebd06a32c19d3ddc46b52524020e51212a7b
2021-02-01 13:32:56 +02:00
Cameron Ayer
c5ecca1e1d web/storagenode: remove uptime columns and references
Full scope:
storagenode/{console,nodestats,notifications,reputation,storagenodedb},
web/storagenode

These columns are deprecated. They used to be for the uptime reputation
system which has been replaced by downtime tracking with audits.

Change-Id: I151d6569577d89733ac97af21a1d885323522b21
2021-01-29 15:36:59 -05:00
Qweder93
c139cbd76b storagenode/payouts: fix CurrentMonthExpectations timezone handling. Estimations based on node's join date.
On servers with non-UTC it would have calculated a different month boundary.
If node joined in current month calculations will be related on amount of days node've been working.

Change-Id: Ie572b197f50c6cdff5a044a53dfb5b9138f82f24
2021-01-25 19:03:30 +02:00
Kaloyan Raev
c24ada7114 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum

Change-Id: Icf7c029e9d800e5f6a9fdd208c36f28e05468690
2021-01-20 17:35:57 +02:00
Qweder93
6ba8f6c8a9 storanode, satellite: payout renamed to payouts, expected estimation payouts added, console api for audits reworked
Change-Id: I4aa5e99bffaa87d0a800a429a4c83aa498ad4b7b
2021-01-18 10:56:03 +00:00
Kaloyan Raev
6dff40f5c5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/metainfo.go

Change-Id: Ib5c49f3c911c58319855a171f9ce73657da976d9
2021-01-14 14:33:59 +02:00
Qweder93
a3b1059fe0 multinode/console: list node updated with total earned
Change-Id: I53a6ebd98acb7741f60d61178451575e7677c730
2021-01-12 23:01:49 +00:00
Qweder93
cf56071bb4 storagenode/payout: error message from empty receipt removed
Change-Id: Iff77ca3a839404b541c687f52dd3ffd891723968
2021-01-11 16:02:07 +02:00
Michał Niewrzał
ec88d21a3c Merge 'main' branch.
Change-Id: I6e8162d1a6caf75e89c9f9c9f9522730aebf83ae
2021-01-11 10:26:58 +01:00
Yaroslav Vorobiov
6507f3ebc6 multinode/console: trusted satellites list api
Change-Id: I97bb9efb1d6cb7d456df0b86e66417c31018b762
2021-01-08 14:50:12 +02:00
Yaroslav Vorobiov
5a43c86b68 multinode/console: list node satellite infos
Change-Id: Ic6cb8d1a6fd7637fdb7bf49e040c43ac30ab1bbf
2021-01-05 14:49:58 +00:00
Yaroslav Vorobiov
fb00d099cf multinode/console: list node infos
Change-Id: I5cac49feff2bac6fbd7ac61dfccffd672da8e8c0
2021-01-05 14:49:11 +00:00
Moby von Briesen
710b86849c storagenode/{console, reputation}: Add audit history to storagenode API
Change-Id: Id680b07feea380824e56496d545f46a94fdcc5d5
2021-01-04 17:37:45 +00:00
Michał Niewrzał
ad3e3a38c5 Merge 'main' branch
Change-Id: Ia0db1b1f9ef3e0671d3f2208881b0abc3064e200
2021-01-04 12:13:45 +01:00
Moby von Briesen
39260b6fdc storagenode/storagenodedb: Add audit_history to storagenode reputationdb
This change will give us the ability to make audit windows visible via
the storagenode api.

Change-Id: Ifb3d044ce9b456c16f8ea3b3b162ba894426477a
2020-12-31 11:00:42 -05:00
Moby von Briesen
edbee53888 satellite,storagenode: Pass audit history over GetStats endpoint
Full prefix: satellite/{overlay,nodestats},storagenode/{reputation,nodestats}

Allow the storagenode to receive its audit history data from the
satellite via the satellite's GetStats endpoint.

The storagenode does not save this data for use in the API yet.

Change-Id: I9488f4d7a4ccb4ccf8336b8e4aeb3e5beee54979
2020-12-30 19:13:26 +00:00
Ivan Fraixedes
2223877439 storgenode/piecestore: Log upload size on Upload
Add the upload size to the log lines of the storagenode Upload endpoint
to provides the information to Storage node operators.

Change-Id: Ife661d28be72c2bf02579093e21fa811566ac8dd
2020-12-28 10:28:09 +00:00
Michal Niewrzal
b3aa28cc02 satellite/gracefulexit: migrate to metabase
Change-Id: I8be9cc68894124427e4a30d7631126b3afb1f281
2020-12-18 10:57:39 +00:00
Qweder93
2fd7809e54 storagenode/payout: stefanbenten satellite name added to payout history, satellites with no held history removed from list
Change-Id: I96861058ccb9c8ce52698796c91b999eaec1f6e6
2020-12-17 11:01:28 +00:00
Egon Elbre
12055e7864 all: minor cleanups
Change-Id: I4248dbe36a62a223b06135254b32851485a2eec1
2020-12-16 10:47:46 +00:00
Qweder93
12144a600b storagenode/console: payout tests and heldhistory joined_at rounding added
Change-Id: I1d43620fbafbf7ed92588b84cb9c6b8ced8832ef
2020-12-14 19:35:04 +02:00
Qweder93
2f62cdf491 storagenode/console: diskSpaceInfo extended with overused diskspace, getDashboardData updated.
Change-Id: I44db26661a8dfb45b5d8e9fcb7511f63deb88cad
2020-12-08 14:55:55 +00:00
Stefan Benten
494bd5db81
all: golangci-lint v1.33.0 fixes (#3985) 2020-12-05 17:01:42 +01:00
Jessica Grebenschikov
b261110352 satellite/orders: get bucketID from encrypted metadata in order instead of serial_numbers table
We want to stop using the serial_numbers table in satelliteDB. One of the last places using the serial_numbers table is when storagenodes settle orders, we look up the bucket name and project ID from the serial number from the serial_numbers table.

Now that we have support to add encrypted metadata into the OrderLimit, this PR makes use of that and now attempts to read the project ID and bucket name from the encrypted orderLimit metadata instead of from the serial_numbers table. For backwards compatibility and to ensure no errors, we will still fallback to the old way of getting that info from the serial_numbers table, but this will be removed in the next release as long as there are no errors.

All processes that create orderLimits must have an orders.encryption-keys set. The services that create orderLimits (and thus need to encrypt the order metadata) are the satellite apiProcess, the repair process, audit service (core process), and graceful exit (core process). Only the satellite api process decrypts the order metadata when storagenodes settle orders. This means that the same encryption key needs to be provided in the config for the satellite api process, repair process, and the core process like so:
orders.include-encrypted-metadata=true
orders.encryption-keys="<"encryptionKeyID>=<encryptionKey>"

Change-Id: Ie2c037971713d6fbf69d697bfad7f8b672eedd66
2020-12-01 15:29:32 +00:00
Egon Elbre
aeb801604e {satellite,storagenode}/orders: fix flaky tests
Before manipulating order information on storagenodes we need to wait
for the orders to propagate to the database. Some of that happens
async with uplink.

Change-Id: Iaacfd7db0909ab5d2831d06388e5fb27b6d4778f
2020-11-18 13:44:02 +00:00