Commit Graph

21 Commits

Author SHA1 Message Date
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
littleskunk
7eb6724c92
logging: unify logging around satellite ID, node ID and piece ID (#3491)
* logging: unify logging around satellite ID, node ID and piece ID

* unify segment index
2019-11-05 22:04:07 +01:00
Jeff Wendling
59f81a4a0d groupcancel/ec delete: add a timeout based on completion times
we used to do something similar for puts, but that ended up hurting
more than it helped. since deletes are best effort, we can do it
here to kill long tails or unresponsive nodes.

Change-Id: I89fd2d9dcf519d76c78ddad70bc419d1868d2df1
2019-10-30 16:18:39 -06:00
Natalie Villasana
5453886231 satellite/repair, uplink/ecclient: remove unused expiration arg from ec.Repair and ec.putPiece (#3416) 2019-10-30 11:35:00 -04:00
Maximillian von Briesen
6df4d7bc73
storagenode/gracefulexit + satellite/gracefulexit: add storagenode-side transfer validation (#3371)
* Make the exiting node check piece hashes, piece IDs, and piece hash signatures before relaying successful transfer data to the satellite.
* Enable immediate graceful exit failure for "successful" transfers that fail satellite-side validation.
* Move transfer piece logic in storagenode worker to separate function (to make the worker easier to understand)
2019-10-25 13:16:20 -04:00
Ethan Adams
3e0d12354a
storagenode/gracefulexit: Implement storage node graceful exit worker - part 1 (#3322) 2019-10-22 16:42:21 -04:00
Ivan Fraixedes
21c6737b10
uplink/ecclient: clarify defer logic in putPiece (#3247)
Refactoring the 'defer' function logic to just only have what's important to not forget before returning but simplifying its logic for making easy to understand the overall function logic.
2019-10-16 18:05:22 +02:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
Jess G
7c203b4884
add satelliteSystem to testplanet and update tests (#3066) 2019-09-17 13:14:49 -07:00
Ivan Fraixedes
ccbf73ecc7
uplink/ecclient: Remove unneeded atomic operation (#3036)
Atomic operations are only needed when a variable can be accessed
concurrently, so when it isn't the case there is no need to use them.
2019-09-13 12:47:35 +02:00
Ivan Fraixedes
8a48500ba4
uplink/ecclient: Report success in debug level (#3037)
Packages shouldn't be chatty when the things go as expected unless the
DEBUG log level is set.
2019-09-13 12:04:12 +02:00
Maximillian von Briesen
fb10815229 Repair with hashes (#2925)
* add outline for ECRepairer

* add description of process in TODO comments

* begin download/getting hash for a single piece

* verify piece hash and order limit during download

* fix download piece

* begin filling out ESREpair. Get

* wip move ecclient.Repair to ecrepairer.Repair

* pass satellite signee into repairer

* reconstruct original stripe from pieces

* move rebuildStripe()

* calculate piece size differently, increment successful count

* fix shares slices initialization

* rename stripeData to segment

* do not pad reader in Repair()

* temp debug

* create unsafeRSScheme

* use decode reader

* rename file name to be all lowercase

* make repair downloader async

* declare condition variable inside Get method

* set downloadAndVerifyPiece's in-memory buffer to be share size

* update unusedLimits var

* address comments

* remove unnecessary comments

* move initialization of segmentRepaire to be outside of repairer service

* use ReadAll during download

* remove dots and move hashing to after validating for order limit signature

* wip test

* make sure files exactly at min threshold are repaired

* remove unused code

* use corrput data and write back to storagenode

* only create corrupted node and piece ids once

* add comment

* address nat's comment

* fix linting and checker_test

* update comment

* add comments

* remove "copied from ecclient" comments

* add clarification comments in ec.Repair
2019-09-06 15:20:36 -04:00
Natalie Villasana
9a1b9f8431
uplink/ecclient: change delete logs from err to debug level (#2917) 2019-08-30 17:00:34 -04:00
Egon Elbre
c309bd3fec
lint: add linting for errs package (#2881) 2019-08-27 19:07:12 +03:00
Bryan White
1915b59af3 satellite/repair: monkit improvements (#2773) 2019-08-14 15:40:26 -04:00
Maximillian von Briesen
3a82b63974
uplink/ecclient: performance - close connections faster (#2757) 2019-08-14 10:03:51 -04:00
Cameron
1f837c53eb
uplink/ecclient: read concurrently with dials during download (#2711)
* do dialing in read

* remove unused type clientCloser

* add mutex to lazyPieceReader

* add nodeID to Download.Read trace
2019-08-09 11:01:40 -04:00
Bryan White
e4c10f3311 uplink/ecclient: add more monkit for segment piece info (#2701) 2019-08-05 17:46:32 +03:00
Egon Elbre
4f0d39cc64
don't use global loggers (#2675) 2019-07-31 17:38:44 +03:00
Ivan Fraixedes
abef20930f
storagenode: Report gRPC error when satellite is untrusted (#2658)
* storagenode/piecestore: Unexport endpoint method
  Make an exported endpoint method to be unexported because it's only used
  by the same package and makes easy to change without thinking in
  breaking changes.
* uplink/ecclient: Use structured logger
  Swap sugared logger by the normal structured logger for having the full
  stack traces of the error in the debug message.
* storagenode/piecestore: Send gRPC error codes upload
  Refactoring in the storagenode/piecestore to send gRPC status error codes
  when some of the methods involved by upload return an error.
  
  The uplink related to uploads has also been modified to retrieve the
  gRPC status code when an error is returned by the server.
2019-07-30 18:58:08 +02:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00