Commit Graph

35 Commits

Author SHA1 Message Date
Egon Elbre
e1a443b04a private/testplanet: allow modifying created database
Instead of providing the database from outside to testplanet create it
inside and then allow wrapping and modifying it. This is more convenient
to use.

Change-Id: I9b8f69e6e0a19ff984b4e2bfe927c9100c77bc6c
2020-03-27 19:14:48 +00:00
Egon Elbre
e8f18a2cfe private/testplanet: expose storagenode and satellite Config
Change-Id: I80fe7ed8ef7356948879afcc6ecb984c5d1a6b9d
2020-03-27 17:01:25 +02:00
Yingrong Zhao
b7b19289d1 bump storj.io/common to latest
Change-Id: I16e337660ce8e1ef332cc842dbf4cfa067b9b98b
2020-03-25 09:08:40 -04:00
Michal Niewrzal
426c8eb31a private/testplanet: add DeleteBucket method for uplink
New method added to be able to delete easily bucket during tests.

Change-Id: Iaae89618cc676ddbbbd4b0df2eeacd143ea6f3c2
2020-02-11 15:58:13 +00:00
Michal Niewrzal
6502454947 satellite/metainfo: move RS configuration to satellite
With this change RS configuration will be set on satellite. Uplink with
get RS values with BeginObject request and will use it. For backward
compatibility and to avoid super large change redundancy scheme stored
with bucket is not touched. This can be done in future.

Change-Id: Ia5f76fc10c37e2c44e4f7b8754f28eafe1f97eff
2020-01-22 09:33:53 +00:00
Yingrong Zhao
76ee8a1b4c satellite: remove UptimeReputation configs from codebase
With the new storage node downtime tracking feature, we need remove current uptime reputation configs: UptimeReputationAlpha, UptimeReputationBeta, and
UptimeReputationDQ. This is the first step of removing the uptime
reputation columns from satellitedb

Change-Id: Ie8fab13295dbf545e33aeda0c4306cda4ba54e36
2020-01-08 18:54:15 +00:00
Natalie Ventura Villasana
1cb0f80a8d satellite/gracefulexit: dq node on exit fail
Disqualifies a node when the node fails to complete a graceful
exit.

Adds a new DisqualifyNode method to the overlay cache, since there
wasn't an existing method to disqualify a node but do nothing else
to its stats.

Adds checks to existing tests to make sure that a storage node that
fails a graceful exit is marked as disqualified in the overlay
cache.

https: //storjlabs.atlassian.net/browse/V3-3342
Change-Id: I4d554a519ab59db31ad3b8e28764c8683a6e3888
2020-01-06 19:16:26 -05:00
Egon Elbre
2680bae88c private/testplanet: remove dependency to uplink
Remove direct dependency on uplink.RSConfig, this simplifies
moving the config file without introducing weird dependencies.

Change-Id: I7fd2a145401e0205d7047631df9d2810241efeec
2020-01-02 09:40:46 +00:00
Natalie Ventura Villasana
aa3e183c2e
satellite/gracefulexit: add ge eligibility check
Adds check to see if storage nodes are eligible to initiate
graceful exit, by checking their CreatedAt date and seeing if
their "age" is greater than the new config value:
NodeMinAgeInMonths
The default for this value is 6 months for now.

https://storjlabs.atlassian.net/browse/V3-3357

Change-Id: Ib807ab8987ddb5a38a27a83886490f73fe8c5816
2019-12-31 09:31:58 -05:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Egon Elbre
d55288cf68 pkg/rpc: replace methods with direct calls to pb
Change-Id: I8bd015d8d316a2c12c1daceca1d9fd257f6f57bc
2019-12-22 17:12:43 +02:00
Ethan
b959ccbae6 satellite/gracefulexit: Use proper rpc status codes for disqualified nodes and too many connections
Change-Id: I41380026175e7678c7cd3d44211de8eb86ce4d0f
2019-12-20 19:05:28 +00:00
Egon Elbre
afe05edff2 {storagenode,satellite}/gracefulexit: ensure workers finish their work
Fixes a data race caused by not waiting for workers to finish
before shutting down. Currently this ended up failing logging
because it was closed when test tried to write to it.

Change-Id: I074045cd83bbf49e658f51353aa7901e9a5d074b
2019-12-17 17:21:52 +02:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Natalie Villasana
1a9757a7f2 satellite/gracefulexit: add count for order limits sent from satellite to exiting node (#3544) 2019-11-13 09:54:50 -05:00
Yingrong Zhao
6331f839ae
satellite/gracefulexit: not allow disqualified node to graceful exit (#3493) 2019-11-07 12:19:34 -05:00
Ethan Adams
f3dccb56b1
satellite/gracefulexit: Check if pointer has been overwritten or deleted before sending transfer message. (#3481) 2019-11-07 11:13:05 -05:00
Egon Elbre
cc032d3151 satellite/metainfo: fix some uses of metainfo.Delete (#3513)
* satellite/metainfo: rename Delete to UnsynchronizedDelete

* fix deletes

* make db private

* fix typos

* also verify on commit object
2019-11-06 18:02:14 +01:00
Ethan Adams
2eb0cc56fe
satellite/gracefulexit: Check if node already has a piece in the pointer (#3434) 2019-11-05 14:13:45 -05:00
Maximillian von Briesen
78fedf5db3
satellite/gracefulexit: handle piece not found messages from storagenode (#3456)
* If a node claims to fail a transfer due to piece not found, remove that node from the pointer, delete the transfer queue item.
* If the pointer is piece hash verified, penalize the node. Otherwise, do not penalize the node.
2019-11-05 10:04:39 -05:00
Maximillian von Briesen
f9df0ea591
satellite/gracefulexit: check for unknown error in graceful exit disable test
Allow error in graceful exit disable test to be rpcstatus.Unimplemented (grpc) or rpcstatus.Unknown (drpc)
2019-11-01 17:21:30 -04:00
Maximillian von Briesen
590312970d satellite/gracefulexit: add flag for enabling/disabling graceful exit on the satellite (#3437) 2019-11-01 16:21:24 +02:00
Ethan Adams
43103ae13f
lower storage node counts in tests (#3427) 2019-10-31 10:57:54 -04:00
Natalie Villasana
4878135068
satellite/gracefulexit, storagenode/gracefulexit: add timeouts (#3407) 2019-10-30 13:40:57 -04:00
Maximillian von Briesen
cd0940724c satellite/gracefulexit: use sync2 cycle inside satellite graceful exit endpoint (#3394) 2019-10-29 14:40:42 -04:00
Maximillian von Briesen
cd3d3850f9 satellite/gracefulexit: only allow one connection per node to graceful exit endpoint (#3357) 2019-10-29 13:23:17 -04:00
Yingrong Zhao
292e64ee2f
satellite/gracefulexit: check duplicate node id before update pointer (#3380)
* check duplicate node id before update pointer

* add test for transfer failure when pointer already contain the receiving node id

* check exiting and receiving nod are still in the pointer

* check node id only exists once in a pointer

* return error if the existing node doesn't match with the piece info in the pointer

* try to recreate the issue on jenkins

* should not remove exiting node piece in test

* Update satellite/gracefulexit/endpoint.go

Co-Authored-By: Maximillian von Briesen <mobyvb@gmail.com>

* Update satellite/gracefulexit/endpoint.go

Co-Authored-By: Maximillian von Briesen <mobyvb@gmail.com>
2019-10-27 14:20:22 -04:00
Maximillian von Briesen
6df4d7bc73
storagenode/gracefulexit + satellite/gracefulexit: add storagenode-side transfer validation (#3371)
* Make the exiting node check piece hashes, piece IDs, and piece hash signatures before relaying successful transfer data to the satellite.
* Enable immediate graceful exit failure for "successful" transfers that fail satellite-side validation.
* Move transfer piece logic in storagenode worker to separate function (to make the worker easier to understand)
2019-10-25 13:16:20 -04:00
Ethan Adams
f0caa6ce5e
satellite:gracefulexit: Update pointer after success (#3369) 2019-10-25 11:14:22 -04:00
Natalie Villasana
696c567e89
satellite/gracefulexit: add piece hash validation for successful transfer (#3313) 2019-10-24 15:38:40 -04:00
Yingrong Zhao
fa1ac24e19
satellite/gracefulexit: add failure threshold check (#3329)
* add overall failure percentage check and inactive time frame check before sending a response to sno

* update comment

* delete node from transfer queue if it has been inactive for too long

* fix linting error

* add test config value

* fix nil pointer

* add config value into testplanet

* add unit test for overall failure threshold

* move timeframe threshold to chore

* update protolock

* add chore test

* add per peiece failure count logic

* change config name from EndpointMaxFailures to MaxFailuresPerPiece

* address comments

* fix linting error

* add error handling for no row returned from progress table

* fix test for graceful exit chore on storagenode

* fix typo InActive -> Inactive

* improve readability for failure threshold calculation

* update config lock

* change error handling for GetProgress in graceful exit endpoint on the satellite side

* return proper rpc error in endpoint

* add check in chore test for checking finish timestamp and queue
2019-10-24 12:24:42 -04:00
Ethan Adams
3e0d12354a
storagenode/gracefulexit: Implement storage node graceful exit worker - part 1 (#3322) 2019-10-22 16:42:21 -04:00
Ethan Adams
37ab84355f
satellite/gracefulexit: protobuf field name updates (#3284)
rename piece_id to original_piece_id
2019-10-15 15:59:12 -04:00
Ethan Adams
78ccf14837 fix interface and EOF check (#3251) 2019-10-12 07:06:20 -06:00
Ethan Adams
a1275746b4
satellite/gracefulexit: Implement the 'process' endpoint on the satellite (#3223) 2019-10-11 17:18:05 -04:00