Commit Graph

2717 Commits

Author SHA1 Message Date
Isaac Hess
1defd4dbfe
storagenode/piecestore: Respect config.MaxConcurrentRequests for drpc (#3402) 2019-10-28 13:12:49 -06:00
Ethan Adams
7c2daa4dd9
Use original pointer when calling UpdatePieces (#3397) 2019-10-28 14:43:46 -04:00
Ethan Adams
5b0398a718 storagenode/gracefulexit: Exclude finished exits from chore/worker processing. Fix update status bug (#3399) 2019-10-28 13:59:45 -04:00
Nikolay Yurchenko
fef0c51c18 web/satellite: notification plugin (#3352) 2019-10-28 19:33:06 +02:00
Ivan Fraixedes
016be4525a
internal/textcontext: Give proper name compiled bin (#3395)
Give a proper name when a Go binary is compiled but the passed package
is the current working directory which is passed as an empty string.
2019-10-28 17:54:55 +01:00
Michal Niewrzal
56f8b2d626
uplink/storage: remove bucket store (#3376) 2019-10-28 09:40:46 -07:00
Ivan Fraixedes
4f281ef348
uplink: Refactor segments Store Get for metainfo Batch (#3362)
Refactoring of the segments Store interface Get method signature to
force the implementations to not use metainfo Client and be able for the
callers to batch requests.
2019-10-28 17:23:20 +01:00
Nikolay Yurchenko
04b16c8b6c stylelint added, lint errors fixed (#3360) 2019-10-28 17:59:19 +02:00
Ethan Adams
9905f2c61e add piece num to transfer queue PK (#3390) 2019-10-28 11:08:33 -04:00
Egon Elbre
93353df4d6
internal/sync2: make Fence accept context (#3393) 2019-10-28 16:04:31 +02:00
Vitalii Shpital
366da85e0a
web/satellite: ts default exports removed (#3382) 2019-10-28 15:27:12 +02:00
Vitalii Shpital
2105dfbbdd
web/storagenode: stylelint applied (#3363) 2019-10-28 15:19:57 +02:00
Matt Robinson
c8d64bd59e Make windows version more dynamic (#3169)
* Make the windows version dynamic

* Move the patch version to the right place
2019-10-28 12:23:22 +01:00
Bryan White
f17b34d02e
cmd/storagenode-updater: add "...up to date" log line if shoudn't update (#3392) 2019-10-28 11:19:19 +01:00
Kaloyan Raev
b10d444560
Rename --log-path back to --log for storagenode-updater (#3391) 2019-10-28 10:30:27 +02:00
Michal Niewrzal
da2eaa7085
scripts: dev script to start postgres before tests (#3344) 2019-10-27 12:02:10 -07:00
Yingrong Zhao
292e64ee2f
satellite/gracefulexit: check duplicate node id before update pointer (#3380)
* check duplicate node id before update pointer

* add test for transfer failure when pointer already contain the receiving node id

* check exiting and receiving nod are still in the pointer

* check node id only exists once in a pointer

* return error if the existing node doesn't match with the piece info in the pointer

* try to recreate the issue on jenkins

* should not remove exiting node piece in test

* Update satellite/gracefulexit/endpoint.go

Co-Authored-By: Maximillian von Briesen <mobyvb@gmail.com>

* Update satellite/gracefulexit/endpoint.go

Co-Authored-By: Maximillian von Briesen <mobyvb@gmail.com>
2019-10-27 14:20:22 -04:00
Maximillian von Briesen
a4e618fd1f handle context cancelled in satellite graceful exit endpoint (#3388) 2019-10-27 10:14:25 -04:00
Cameron
279cd56253 change uuid conversion (#3384) 2019-10-26 17:51:39 -04:00
paul cannon
1469f7f41f
storagenode/contact: wait for UpdateSelf before start (#3332)
When the contact chore starts running before the monitor service has
provided any useful capacity data, the first outgoing contact has
not-very-helpful data for the satellite. This change causes the contact
chore to wait until capacity data is available. The wait should be quite
short in all reasonable cases: even when a node starts with a lot of
stored pieces and no cached spaceUsedDB data, new data will have been
calculated and cached by the call to
`peer.Storage2.CacheService.Init(ctx)` in `storagenode.cmdRun()` before
`peer.Run(ctx)`.

Change-Id: Ibc26d5c1fc10a23006c00bc3f13ff6cf71f8bf1d
2019-10-26 12:16:25 -05:00
Jeff Wendling
ed48e74e20 gracefulexit: fix build for drpc (#3387)
Change-Id: I335e9f8991a10c9e8a0737bc7c9ea3f04cbe2546
2019-10-26 15:53:35 +02:00
Ethan Adams
e54d290d2e satellite/gracefulexit: Add signatures for success/failed exit finished messages. (#3368)
* add signatures, fix process loop bug, move delete to on success

* added tests for signatures

* PR comment updates

* fixed setting reason by default.

* updates for PR comments

* added signed failure when verificationi fails

* moved to sign_test

* fix panic

* removed testplanet from test
2019-10-25 16:36:26 -04:00
Maximillian von Briesen
6df4d7bc73
storagenode/gracefulexit + satellite/gracefulexit: add storagenode-side transfer validation (#3371)
* Make the exiting node check piece hashes, piece IDs, and piece hash signatures before relaying successful transfer data to the satellite.
* Enable immediate graceful exit failure for "successful" transfers that fail satellite-side validation.
* Move transfer piece logic in storagenode worker to separate function (to make the worker easier to understand)
2019-10-25 13:16:20 -04:00
Kaloyan Raev
65abc95f7e Fix args passed to the storagenode-updater service (#3383) 2019-10-25 18:49:57 +02:00
Kaloyan Raev
aca18109a2
Use new wallet, email and externl-address args in storagenode setup (#3381) 2019-10-25 19:02:30 +03:00
Maximillian von Briesen
472672d52f
docs/storage-node-downtime-tracking: add open issue regarding satellite outages (#3370) 2019-10-25 11:39:11 -04:00
Ethan Adams
f0caa6ce5e
satellite:gracefulexit: Update pointer after success (#3369) 2019-10-25 11:14:22 -04:00
Nikolai Siedov
f4f142bf3e
satellite/console: get, update auth api endpoints added (#3375) 2019-10-25 15:07:17 +03:00
Bryan White
d61b3688f7 internal/version: fix OldMinimum to use old semver type (#3373)
* fix version checker type assertion

* more fixing
2019-10-25 13:24:23 +02:00
Vitalii Shpital
03f399857f
web/satellite: removed ability to check project owner for deletion (#3350) 2019-10-25 13:01:03 +03:00
Vitalii Shpital
3880f7a60e
web/storagenode: vue-svg-loader implemented (#3347) 2019-10-25 12:53:35 +03:00
Ivan Fraixedes
d9d82b0336 uplink: Reduce satellite request using Batch when possible (#3351)
* uplink/metainfo: Return classified Not Found error

Metainfo client Batch method must return the Storj Not Found error class
when the RCP server response with a not found status code as any other
metainfo Client method does.

Also if the error isn't Not Found one, it must wrap the error.

* uplink/storage/streams: Use Batch request in Delete

Change the 2 individual metainfo Client calls that streamStore Delete
method does by a single Batch one.
2019-10-24 14:18:48 -07:00
Michal Niewrzal
7f6893ea86
satellite/metainfo: fix broken object listing (#3348) 2019-10-24 14:05:08 -07:00
Bryan White
0b678c23c7 {internal/version,versioncontrol}: fix old versions (#3359)
* fix old semver

* go jenkins!
2019-10-24 22:24:28 +02:00
Natalie Villasana
696c567e89
satellite/gracefulexit: add piece hash validation for successful transfer (#3313) 2019-10-24 15:38:40 -04:00
Yingrong Zhao
fa1ac24e19
satellite/gracefulexit: add failure threshold check (#3329)
* add overall failure percentage check and inactive time frame check before sending a response to sno

* update comment

* delete node from transfer queue if it has been inactive for too long

* fix linting error

* add test config value

* fix nil pointer

* add config value into testplanet

* add unit test for overall failure threshold

* move timeframe threshold to chore

* update protolock

* add chore test

* add per peiece failure count logic

* change config name from EndpointMaxFailures to MaxFailuresPerPiece

* address comments

* fix linting error

* add error handling for no row returned from progress table

* fix test for graceful exit chore on storagenode

* fix typo InActive -> Inactive

* improve readability for failure threshold calculation

* update config lock

* change error handling for GetProgress in graceful exit endpoint on the satellite side

* return proper rpc error in endpoint

* add check in chore test for checking finish timestamp and queue
2019-10-24 12:24:42 -04:00
Vitalii Shpital
bff5c19de6 web/satellite: temporar removing of payments api calls (#3361) 2019-10-24 17:48:41 +02:00
Isaac Hess
75412e54e5
storagenode/piecestore: Rename liveGRPCRequests back to liveRequests (#3354) 2019-10-23 13:43:43 -06:00
Yehor Butko
e82245e10e
satellite/payments: credit card selection (#3304) 2019-10-23 20:33:24 +03:00
Isaac Hess
14c7648530
storagenode/piecestore: Only limit grpc requests (#3342) 2019-10-23 10:14:02 -06:00
Nikolai Siedov
655e2b3422
sattelite/console: sattellite name API endpoint added (#3349) 2019-10-23 18:36:58 +03:00
Vitalii Shpital
99ccdc1bca
web/satellite: long email bug fixed (#3294) 2019-10-23 16:37:51 +03:00
Vitalii Shpital
810dc80d45
web/satellite: vue-svg-loader implemented (#3307) 2019-10-23 15:26:39 +03:00
Yaroslav Vorobiov
1a304f5ef9
satellite/payments: add payments loop, update pending transactions (#3318) 2019-10-23 15:04:54 +03:00
Michal Niewrzal
521c39bda0
uplink/metainfo: cleanup method names (#3315) 2019-10-22 23:59:56 -07:00
Maximillian von Briesen
abb567f6ae
cmd/satellite: add graceful exit reports command to satellite CLI (#3300)
* update lock file and add comment

* add created at and bytes transferred

* cleanup

* rename db func to GetGracefulExitNodesByTimeFrame

* fix flag

* split into two overlay functions

* := to =

* fix test

* add node not found error class

* fix overlay test

* suggested test changes

* review suggestions

* get exit status from overlay.Get()

* check rows.Err

* fix panic when ExitFinishedAt is nil

* fix comments in cmdGracefulExit
2019-10-22 21:06:01 -04:00
Jeff Wendling
51d5d8656a pkg/rpc: drpc connection pooling
keep a pool of connections open when dialing for drpc. this
makes it so that long lived clients (like lib/uplink's Project)
don't continue to use a bad connection forever. it also allows
for concurrent rpcs.

Change-Id: If649b286050e4f09c413fadc3e1ce88f5fc6e600
2019-10-22 18:15:24 -06:00
JT Olio
2c6fa3c5f8
pkg/rpc: remove read/write deadlines as a mechanism for request timeouts (#3335)
libuplink was incorrectly setting timeouts to 10 seconds still, but
should have been at least 10 minutes. the order sender was setting them
to 1 hour. we don't want timeouts in uplink-side logic as it establishes
a minimum rate on tcp streams.

instead of all of this, just use tcp keep alive. tcp keep alive packets are
sent every 15 seconds and if the peer stops responding the connection
dies. this is enabled by default with go. this will kill tcp connections
when they stop working.

Change-Id: I3d7ad49f71950b3eb43044eedf4b17993116045b
2019-10-22 17:57:24 -06:00
Ethan Adams
3e0d12354a
storagenode/gracefulexit: Implement storage node graceful exit worker - part 1 (#3322) 2019-10-22 16:42:21 -04:00
Nikolai Siedov
3eec4e9070
satellite/console: add REST delete API endpoint (#3337) 2019-10-22 19:17:09 +03:00