storj/pkg/transport/transport.go

176 lines
5.4 KiB
Go
Raw Normal View History

2019-01-24 20:15:10 +00:00
// Copyright (C) 2019 Storj Labs, Inc.
Transport Client (#89) * port changes * Task monitor and setup merge from the staging * Restructure + additional interface * Add NewOverlayClient * integrated DHT client interface * added test for interface * PR comments addressed * lint issue * added generated protobuf * adding new interface * added the interface framework * deleted file * fixes compilation errors and integrates new dhtcclient interface * merged netstat latest changes and dht new interface chagnes * fixed the address's port * adding comments * PR comments addressed * netclient interface dial method added * rename and integrated transportclient with minio gateway * rename and code clean up * made changes based on the Dennis's changes on the kad-client * Code review comment changes based on kaloyan review comments * reverted the changes to be similar to master * removed unused file * renamed to transportclient * added the review changes * store the address of the client * updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters * updated the test case to test the bad address passed condition * updated the code per code review comments * Bolt backed overlay cache (#94) * wip * add separate `Process` tests for bolt and redis-backed overlay * more testing * fix gitignore * fix linter error * goimports goimports GOIMPORTS GoImPortS!!!! * fix port madness * forgot to add * add `mux` as handler and shorten context timeouts * gofreakingimports * fix comments * refactor test & add logger/monkit registry * debugging travis * add comment * Set redisAddress to empty string for bolt-test * travis experiment * refactoring tests * Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache * Automatically build, tag and push docker images on merge to master (#103) * port changes * build overlay on successful merge to master * fixes to Makefile * permissions * dep ensure * gopath * let's try vgo * remove dep * maybe alpine is the issue * tagging go version on build * stupid vgo * vgo * adding tags to push * quotes * local linting fixes & stupid travis * prepend storjlabs to docker tag (#108) * port changes * fixing tag name * Use continue instead of return in table tests (#106) I did a dumb mistake for some of the table tests, which made some of the test cases not being executed. * pkg/kademlia tests and restructuring (#97) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * files created * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * remove bkad dependencie from tests * wip * wip * wip * wip * wip * updated coyle/kademlia * wip * cleanup * ports * overlay upgraded * linter fixes * piecestore kademlia newID * add changes from kad demo * PR comments addresses * go func * force travis build * fixed merge conflicts * fixed merge conflicts * Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests * linter issues * linting issues * fixed merge conflicts * linter is stupid * Coyle/docker fix (#109) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge branch 'master' of https://github.com/storj/storj * fixing tag name * no idea * testing * changes * testing on travis * testing * changes to travis build * new approach * Merge branch 'master' into coyle/docker-fix * hardcode version (#111) * hardcode version * adding coveralls / code coverage (#112) * adding coveralls * adding code coverage badge * fixing badges * verbose * swap tests and coverage * extra line * maybe * maybe * moar * gover maybe * testing * cleanup * protos/netstate: remove stuff we're not using (#100) * protos/netstate: remove stuff we're not using * protos/netstate: add metadata field for segmentstore * fix netstate client test * pkg/process: start replacing pkg/process with cobra helpers (#98) * Implement psclient interface (#107) * Implement psclient interface * Add string method to pieceID type * try to fix linter errors * Whoops missed an error * More linter errors * Typo * Lol double typo * Get everything working, begin adding tests for psclient rpc * goimports * Forgot to change the piecestore cli when changed the piecestore code * Fix CLI * remove ID length, added validator to pieceID * Move grpc ranger to client Change client PUT api to take a reader rather than return a writer * GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser * Forgot to remove offset * Added message upon successful store * Do that thing dennis and kaloyan wanted * goimports * Make closeConn a part of the interface for psclient * Use interface * Removed uneccessary new lines * goimport * Whoops * Actually we don't want to use the interface in Piece Ranger * Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples * Make comments look nicer * modified transport client based on the the design discussion * modified transport client based on the the design discussion * added the as discussed connection cache interface functionality * added the as discussed connection cache interface functionality * transport client changes * transport client per code review changes * per the code review comments * transport client incorporates review comments * fixes lint warnings * lint warning fixes....client interface has to be Client * client.go changes * transport.go changes * added test case * added test cases * comment fix * comment fix
2018-06-29 19:28:06 +01:00
// See LICENSE for copying information.
package transport
import (
"context"
"net"
"time"
Transport Client (#89) * port changes * Task monitor and setup merge from the staging * Restructure + additional interface * Add NewOverlayClient * integrated DHT client interface * added test for interface * PR comments addressed * lint issue * added generated protobuf * adding new interface * added the interface framework * deleted file * fixes compilation errors and integrates new dhtcclient interface * merged netstat latest changes and dht new interface chagnes * fixed the address's port * adding comments * PR comments addressed * netclient interface dial method added * rename and integrated transportclient with minio gateway * rename and code clean up * made changes based on the Dennis's changes on the kad-client * Code review comment changes based on kaloyan review comments * reverted the changes to be similar to master * removed unused file * renamed to transportclient * added the review changes * store the address of the client * updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters * updated the test case to test the bad address passed condition * updated the code per code review comments * Bolt backed overlay cache (#94) * wip * add separate `Process` tests for bolt and redis-backed overlay * more testing * fix gitignore * fix linter error * goimports goimports GOIMPORTS GoImPortS!!!! * fix port madness * forgot to add * add `mux` as handler and shorten context timeouts * gofreakingimports * fix comments * refactor test & add logger/monkit registry * debugging travis * add comment * Set redisAddress to empty string for bolt-test * travis experiment * refactoring tests * Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache * Automatically build, tag and push docker images on merge to master (#103) * port changes * build overlay on successful merge to master * fixes to Makefile * permissions * dep ensure * gopath * let's try vgo * remove dep * maybe alpine is the issue * tagging go version on build * stupid vgo * vgo * adding tags to push * quotes * local linting fixes & stupid travis * prepend storjlabs to docker tag (#108) * port changes * fixing tag name * Use continue instead of return in table tests (#106) I did a dumb mistake for some of the table tests, which made some of the test cases not being executed. * pkg/kademlia tests and restructuring (#97) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * files created * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * remove bkad dependencie from tests * wip * wip * wip * wip * wip * updated coyle/kademlia * wip * cleanup * ports * overlay upgraded * linter fixes * piecestore kademlia newID * add changes from kad demo * PR comments addresses * go func * force travis build * fixed merge conflicts * fixed merge conflicts * Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests * linter issues * linting issues * fixed merge conflicts * linter is stupid * Coyle/docker fix (#109) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge branch 'master' of https://github.com/storj/storj * fixing tag name * no idea * testing * changes * testing on travis * testing * changes to travis build * new approach * Merge branch 'master' into coyle/docker-fix * hardcode version (#111) * hardcode version * adding coveralls / code coverage (#112) * adding coveralls * adding code coverage badge * fixing badges * verbose * swap tests and coverage * extra line * maybe * maybe * moar * gover maybe * testing * cleanup * protos/netstate: remove stuff we're not using (#100) * protos/netstate: remove stuff we're not using * protos/netstate: add metadata field for segmentstore * fix netstate client test * pkg/process: start replacing pkg/process with cobra helpers (#98) * Implement psclient interface (#107) * Implement psclient interface * Add string method to pieceID type * try to fix linter errors * Whoops missed an error * More linter errors * Typo * Lol double typo * Get everything working, begin adding tests for psclient rpc * goimports * Forgot to change the piecestore cli when changed the piecestore code * Fix CLI * remove ID length, added validator to pieceID * Move grpc ranger to client Change client PUT api to take a reader rather than return a writer * GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser * Forgot to remove offset * Added message upon successful store * Do that thing dennis and kaloyan wanted * goimports * Make closeConn a part of the interface for psclient * Use interface * Removed uneccessary new lines * goimport * Whoops * Actually we don't want to use the interface in Piece Ranger * Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples * Make comments look nicer * modified transport client based on the the design discussion * modified transport client based on the the design discussion * added the as discussed connection cache interface functionality * added the as discussed connection cache interface functionality * transport client changes * transport client per code review changes * per the code review comments * transport client incorporates review comments * fixes lint warnings * lint warning fixes....client interface has to be Client * client.go changes * transport.go changes * added test case * added test cases * comment fix * comment fix
2018-06-29 19:28:06 +01:00
"google.golang.org/grpc"
2019-01-30 20:47:21 +00:00
"storj.io/storj/pkg/identity"
"storj.io/storj/pkg/pb"
"storj.io/storj/pkg/peertls/tlsopts"
Transport Client (#89) * port changes * Task monitor and setup merge from the staging * Restructure + additional interface * Add NewOverlayClient * integrated DHT client interface * added test for interface * PR comments addressed * lint issue * added generated protobuf * adding new interface * added the interface framework * deleted file * fixes compilation errors and integrates new dhtcclient interface * merged netstat latest changes and dht new interface chagnes * fixed the address's port * adding comments * PR comments addressed * netclient interface dial method added * rename and integrated transportclient with minio gateway * rename and code clean up * made changes based on the Dennis's changes on the kad-client * Code review comment changes based on kaloyan review comments * reverted the changes to be similar to master * removed unused file * renamed to transportclient * added the review changes * store the address of the client * updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters * updated the test case to test the bad address passed condition * updated the code per code review comments * Bolt backed overlay cache (#94) * wip * add separate `Process` tests for bolt and redis-backed overlay * more testing * fix gitignore * fix linter error * goimports goimports GOIMPORTS GoImPortS!!!! * fix port madness * forgot to add * add `mux` as handler and shorten context timeouts * gofreakingimports * fix comments * refactor test & add logger/monkit registry * debugging travis * add comment * Set redisAddress to empty string for bolt-test * travis experiment * refactoring tests * Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache * Automatically build, tag and push docker images on merge to master (#103) * port changes * build overlay on successful merge to master * fixes to Makefile * permissions * dep ensure * gopath * let's try vgo * remove dep * maybe alpine is the issue * tagging go version on build * stupid vgo * vgo * adding tags to push * quotes * local linting fixes & stupid travis * prepend storjlabs to docker tag (#108) * port changes * fixing tag name * Use continue instead of return in table tests (#106) I did a dumb mistake for some of the table tests, which made some of the test cases not being executed. * pkg/kademlia tests and restructuring (#97) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * files created * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * remove bkad dependencie from tests * wip * wip * wip * wip * wip * updated coyle/kademlia * wip * cleanup * ports * overlay upgraded * linter fixes * piecestore kademlia newID * add changes from kad demo * PR comments addresses * go func * force travis build * fixed merge conflicts * fixed merge conflicts * Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests * linter issues * linting issues * fixed merge conflicts * linter is stupid * Coyle/docker fix (#109) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge branch 'master' of https://github.com/storj/storj * fixing tag name * no idea * testing * changes * testing on travis * testing * changes to travis build * new approach * Merge branch 'master' into coyle/docker-fix * hardcode version (#111) * hardcode version * adding coveralls / code coverage (#112) * adding coveralls * adding code coverage badge * fixing badges * verbose * swap tests and coverage * extra line * maybe * maybe * moar * gover maybe * testing * cleanup * protos/netstate: remove stuff we're not using (#100) * protos/netstate: remove stuff we're not using * protos/netstate: add metadata field for segmentstore * fix netstate client test * pkg/process: start replacing pkg/process with cobra helpers (#98) * Implement psclient interface (#107) * Implement psclient interface * Add string method to pieceID type * try to fix linter errors * Whoops missed an error * More linter errors * Typo * Lol double typo * Get everything working, begin adding tests for psclient rpc * goimports * Forgot to change the piecestore cli when changed the piecestore code * Fix CLI * remove ID length, added validator to pieceID * Move grpc ranger to client Change client PUT api to take a reader rather than return a writer * GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser * Forgot to remove offset * Added message upon successful store * Do that thing dennis and kaloyan wanted * goimports * Make closeConn a part of the interface for psclient * Use interface * Removed uneccessary new lines * goimport * Whoops * Actually we don't want to use the interface in Piece Ranger * Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples * Make comments look nicer * modified transport client based on the the design discussion * modified transport client based on the the design discussion * added the as discussed connection cache interface functionality * added the as discussed connection cache interface functionality * transport client changes * transport client per code review changes * per the code review comments * transport client incorporates review comments * fixes lint warnings * lint warning fixes....client interface has to be Client * client.go changes * transport.go changes * added test case * added test cases * comment fix * comment fix
2018-06-29 19:28:06 +01:00
)
// Observer implements the ConnSuccess and ConnFailure methods
// for Discovery and other services to use
type Observer interface {
ConnSuccess(ctx context.Context, node *pb.Node)
ConnFailure(ctx context.Context, node *pb.Node, err error)
}
// Client defines the interface to an transport client.
type Client interface {
DialNode(ctx context.Context, node *pb.Node, opts ...grpc.DialOption) (*grpc.ClientConn, error)
DialAddress(ctx context.Context, address string, opts ...grpc.DialOption) (*grpc.ClientConn, error)
2019-01-30 20:47:21 +00:00
Identity() *identity.FullIdentity
Add Versioning Server (#1576) * Initial Webserver Draft for Version Controlling * Rename type to avoid confusion * Move Function Calls into Version Package * Fix Linting and Language Typos * Fix Linting and Spelling Mistakes * Include Copyright * Include Copyright * Adjust Version-Control Server to return list of Versions * Linting * Improve Request Handling and Readability * Add Configuration File Option Add Systemd Service file * Add Logging to File * Smaller Changes * Add Semantic Versioning and refuses outdated Software from Startup (#1612) * implements internal Semantic Version library * adds version logging + reporting to process * Advance SemVer struct for easier handling * Add Accepted Version Store * Fix Function * Restructure * Type Conversion * Handle Version String properly * Add Note about array index * Set temporary Default Version * Add Copyright * Adding Version to Dashboard * Adding Version Info Log * Renaming and adding CheckerProcess * Iteration Sync * Iteration V2 * linting * made LogAndReportVersion a go routine * Refactor to Go Routine * Add Context to Go Routine and allow Operation if Lookup to Control Server fails * Handle Unmarshal properly * Linting * Relocate Version Checks * Relocating Version Check and specified default Version for now * Linting Error Prevention * Refuse Startup on outdated Version * Add Startup Check Function * Straighten Logging * Dont force Shutdown if --dev flag is set * Create full Service/Peer Structure for ControlServer * Linting * Straighting Naming * Finish VersionControl Service Layout * Improve Error Handling * Change Listening Address * Move Checker Function * Remove VersionControl Peer * Linting * Linting * Create VersionClient Service * Renaming * Add Version Client to Peer Definitions * Linting and Renaming * Linting * Remove Transport Checks for now * Move to Client Side Flag * Remove check * Linting * Transport Client Version Intro * Adding Version Client to Transport Client * Add missing parameter * Adding Version Check, to set Allowed = true * Set Default to true, testing * Restructuring Code * Uplink Changes * Add more proper Defaults * Renaming of Version struct * Dont pass Service use Pointer * Set Defaults for Versioning Checks * Put HTTP Server in go routine * Add Versioncontrol to Storj-Sim * Testplanet Fixes * Linting * Add Error Handling and new Server Struct * Move Lock slightly * Reduce Race Potentials * Remove unnecessary files * Linting * Add Proper Transport Handling * small fixes * add fence for allowed check * Add Startup Version Check and Service Naming * make errormessage private * Add Comments about VersionedClient * Linting * Remove Checks that refuse outgoing connections * Remove release cmd * Add Release Script * Linting * Update to use correct Values * Move vars private and set minimum default versions for testing builds * Remove VersionedClient * Better Error Handling and naked return removal * Straighten the Regex and string conversion * Change Check to allows testplanet and storj-sim to run without the need to pass an LDFlag * Cosmetic Change to Dashboard * Cleanup Returns and remove commented code * Remove Version Check if no build options are passed in * Pass in Config Values instead of Pointers * Handle missed Error * Update Endpoint URL * Change Type of Release Flag * Add additional Logging * Remove Versions Logging of other Services * minor fixes Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
2019-04-03 20:13:39 +01:00
WithObservers(obs ...Observer) Client
AlertSuccess(ctx context.Context, node *pb.Node)
AlertFail(ctx context.Context, node *pb.Node, err error)
}
// Timeouts contains all of the timeouts configurable for a transport
type Timeouts struct {
Request time.Duration
Dial time.Duration
}
Transport Client (#89) * port changes * Task monitor and setup merge from the staging * Restructure + additional interface * Add NewOverlayClient * integrated DHT client interface * added test for interface * PR comments addressed * lint issue * added generated protobuf * adding new interface * added the interface framework * deleted file * fixes compilation errors and integrates new dhtcclient interface * merged netstat latest changes and dht new interface chagnes * fixed the address's port * adding comments * PR comments addressed * netclient interface dial method added * rename and integrated transportclient with minio gateway * rename and code clean up * made changes based on the Dennis's changes on the kad-client * Code review comment changes based on kaloyan review comments * reverted the changes to be similar to master * removed unused file * renamed to transportclient * added the review changes * store the address of the client * updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters * updated the test case to test the bad address passed condition * updated the code per code review comments * Bolt backed overlay cache (#94) * wip * add separate `Process` tests for bolt and redis-backed overlay * more testing * fix gitignore * fix linter error * goimports goimports GOIMPORTS GoImPortS!!!! * fix port madness * forgot to add * add `mux` as handler and shorten context timeouts * gofreakingimports * fix comments * refactor test & add logger/monkit registry * debugging travis * add comment * Set redisAddress to empty string for bolt-test * travis experiment * refactoring tests * Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache * Automatically build, tag and push docker images on merge to master (#103) * port changes * build overlay on successful merge to master * fixes to Makefile * permissions * dep ensure * gopath * let's try vgo * remove dep * maybe alpine is the issue * tagging go version on build * stupid vgo * vgo * adding tags to push * quotes * local linting fixes & stupid travis * prepend storjlabs to docker tag (#108) * port changes * fixing tag name * Use continue instead of return in table tests (#106) I did a dumb mistake for some of the table tests, which made some of the test cases not being executed. * pkg/kademlia tests and restructuring (#97) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * files created * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * remove bkad dependencie from tests * wip * wip * wip * wip * wip * updated coyle/kademlia * wip * cleanup * ports * overlay upgraded * linter fixes * piecestore kademlia newID * add changes from kad demo * PR comments addresses * go func * force travis build * fixed merge conflicts * fixed merge conflicts * Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests * linter issues * linting issues * fixed merge conflicts * linter is stupid * Coyle/docker fix (#109) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge branch 'master' of https://github.com/storj/storj * fixing tag name * no idea * testing * changes * testing on travis * testing * changes to travis build * new approach * Merge branch 'master' into coyle/docker-fix * hardcode version (#111) * hardcode version * adding coveralls / code coverage (#112) * adding coveralls * adding code coverage badge * fixing badges * verbose * swap tests and coverage * extra line * maybe * maybe * moar * gover maybe * testing * cleanup * protos/netstate: remove stuff we're not using (#100) * protos/netstate: remove stuff we're not using * protos/netstate: add metadata field for segmentstore * fix netstate client test * pkg/process: start replacing pkg/process with cobra helpers (#98) * Implement psclient interface (#107) * Implement psclient interface * Add string method to pieceID type * try to fix linter errors * Whoops missed an error * More linter errors * Typo * Lol double typo * Get everything working, begin adding tests for psclient rpc * goimports * Forgot to change the piecestore cli when changed the piecestore code * Fix CLI * remove ID length, added validator to pieceID * Move grpc ranger to client Change client PUT api to take a reader rather than return a writer * GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser * Forgot to remove offset * Added message upon successful store * Do that thing dennis and kaloyan wanted * goimports * Make closeConn a part of the interface for psclient * Use interface * Removed uneccessary new lines * goimport * Whoops * Actually we don't want to use the interface in Piece Ranger * Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples * Make comments look nicer * modified transport client based on the the design discussion * modified transport client based on the the design discussion * added the as discussed connection cache interface functionality * added the as discussed connection cache interface functionality * transport client changes * transport client per code review changes * per the code review comments * transport client incorporates review comments * fixes lint warnings * lint warning fixes....client interface has to be Client * client.go changes * transport.go changes * added test case * added test cases * comment fix * comment fix
2018-06-29 19:28:06 +01:00
// Transport interface structure
type Transport struct {
tlsOpts *tlsopts.Options
observers []Observer
timeouts Timeouts
Transport Client (#89) * port changes * Task monitor and setup merge from the staging * Restructure + additional interface * Add NewOverlayClient * integrated DHT client interface * added test for interface * PR comments addressed * lint issue * added generated protobuf * adding new interface * added the interface framework * deleted file * fixes compilation errors and integrates new dhtcclient interface * merged netstat latest changes and dht new interface chagnes * fixed the address's port * adding comments * PR comments addressed * netclient interface dial method added * rename and integrated transportclient with minio gateway * rename and code clean up * made changes based on the Dennis's changes on the kad-client * Code review comment changes based on kaloyan review comments * reverted the changes to be similar to master * removed unused file * renamed to transportclient * added the review changes * store the address of the client * updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters * updated the test case to test the bad address passed condition * updated the code per code review comments * Bolt backed overlay cache (#94) * wip * add separate `Process` tests for bolt and redis-backed overlay * more testing * fix gitignore * fix linter error * goimports goimports GOIMPORTS GoImPortS!!!! * fix port madness * forgot to add * add `mux` as handler and shorten context timeouts * gofreakingimports * fix comments * refactor test & add logger/monkit registry * debugging travis * add comment * Set redisAddress to empty string for bolt-test * travis experiment * refactoring tests * Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache * Automatically build, tag and push docker images on merge to master (#103) * port changes * build overlay on successful merge to master * fixes to Makefile * permissions * dep ensure * gopath * let's try vgo * remove dep * maybe alpine is the issue * tagging go version on build * stupid vgo * vgo * adding tags to push * quotes * local linting fixes & stupid travis * prepend storjlabs to docker tag (#108) * port changes * fixing tag name * Use continue instead of return in table tests (#106) I did a dumb mistake for some of the table tests, which made some of the test cases not being executed. * pkg/kademlia tests and restructuring (#97) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * files created * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * remove bkad dependencie from tests * wip * wip * wip * wip * wip * updated coyle/kademlia * wip * cleanup * ports * overlay upgraded * linter fixes * piecestore kademlia newID * add changes from kad demo * PR comments addresses * go func * force travis build * fixed merge conflicts * fixed merge conflicts * Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests * linter issues * linting issues * fixed merge conflicts * linter is stupid * Coyle/docker fix (#109) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge branch 'master' of https://github.com/storj/storj * fixing tag name * no idea * testing * changes * testing on travis * testing * changes to travis build * new approach * Merge branch 'master' into coyle/docker-fix * hardcode version (#111) * hardcode version * adding coveralls / code coverage (#112) * adding coveralls * adding code coverage badge * fixing badges * verbose * swap tests and coverage * extra line * maybe * maybe * moar * gover maybe * testing * cleanup * protos/netstate: remove stuff we're not using (#100) * protos/netstate: remove stuff we're not using * protos/netstate: add metadata field for segmentstore * fix netstate client test * pkg/process: start replacing pkg/process with cobra helpers (#98) * Implement psclient interface (#107) * Implement psclient interface * Add string method to pieceID type * try to fix linter errors * Whoops missed an error * More linter errors * Typo * Lol double typo * Get everything working, begin adding tests for psclient rpc * goimports * Forgot to change the piecestore cli when changed the piecestore code * Fix CLI * remove ID length, added validator to pieceID * Move grpc ranger to client Change client PUT api to take a reader rather than return a writer * GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser * Forgot to remove offset * Added message upon successful store * Do that thing dennis and kaloyan wanted * goimports * Make closeConn a part of the interface for psclient * Use interface * Removed uneccessary new lines * goimport * Whoops * Actually we don't want to use the interface in Piece Ranger * Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples * Make comments look nicer * modified transport client based on the the design discussion * modified transport client based on the the design discussion * added the as discussed connection cache interface functionality * added the as discussed connection cache interface functionality * transport client changes * transport client per code review changes * per the code review comments * transport client incorporates review comments * fixes lint warnings * lint warning fixes....client interface has to be Client * client.go changes * transport.go changes * added test case * added test cases * comment fix * comment fix
2018-06-29 19:28:06 +01:00
}
// NewClient returns a transport client with a default timeout for requests
func NewClient(tlsOpts *tlsopts.Options, obs ...Observer) Client {
return NewClientWithTimeouts(tlsOpts, Timeouts{}, obs...)
}
// NewClientWithTimeouts returns a transport client with a specified timeout for requests
func NewClientWithTimeouts(tlsOpts *tlsopts.Options, timeouts Timeouts, obs ...Observer) Client {
if timeouts.Request == 0 {
timeouts.Request = defaultRequestTimeout
}
if timeouts.Dial == 0 {
timeouts.Dial = defaultDialTimeout
}
return &Transport{
tlsOpts: tlsOpts,
timeouts: timeouts,
observers: obs,
}
}
// DialNode returns a grpc connection with tls to a node.
//
// Use this method for communicating with nodes as it is more secure than
// DialAddress. The connection will be established successfully only if the
// target node has the private key for the requested node ID.
2018-11-19 14:40:01 +00:00
func (transport *Transport) DialNode(ctx context.Context, node *pb.Node, opts ...grpc.DialOption) (conn *grpc.ClientConn, err error) {
defer mon.Task()(&ctx, "node: "+node.Id.String()[0:8])(&err)
Add Versioning Server (#1576) * Initial Webserver Draft for Version Controlling * Rename type to avoid confusion * Move Function Calls into Version Package * Fix Linting and Language Typos * Fix Linting and Spelling Mistakes * Include Copyright * Include Copyright * Adjust Version-Control Server to return list of Versions * Linting * Improve Request Handling and Readability * Add Configuration File Option Add Systemd Service file * Add Logging to File * Smaller Changes * Add Semantic Versioning and refuses outdated Software from Startup (#1612) * implements internal Semantic Version library * adds version logging + reporting to process * Advance SemVer struct for easier handling * Add Accepted Version Store * Fix Function * Restructure * Type Conversion * Handle Version String properly * Add Note about array index * Set temporary Default Version * Add Copyright * Adding Version to Dashboard * Adding Version Info Log * Renaming and adding CheckerProcess * Iteration Sync * Iteration V2 * linting * made LogAndReportVersion a go routine * Refactor to Go Routine * Add Context to Go Routine and allow Operation if Lookup to Control Server fails * Handle Unmarshal properly * Linting * Relocate Version Checks * Relocating Version Check and specified default Version for now * Linting Error Prevention * Refuse Startup on outdated Version * Add Startup Check Function * Straighten Logging * Dont force Shutdown if --dev flag is set * Create full Service/Peer Structure for ControlServer * Linting * Straighting Naming * Finish VersionControl Service Layout * Improve Error Handling * Change Listening Address * Move Checker Function * Remove VersionControl Peer * Linting * Linting * Create VersionClient Service * Renaming * Add Version Client to Peer Definitions * Linting and Renaming * Linting * Remove Transport Checks for now * Move to Client Side Flag * Remove check * Linting * Transport Client Version Intro * Adding Version Client to Transport Client * Add missing parameter * Adding Version Check, to set Allowed = true * Set Default to true, testing * Restructuring Code * Uplink Changes * Add more proper Defaults * Renaming of Version struct * Dont pass Service use Pointer * Set Defaults for Versioning Checks * Put HTTP Server in go routine * Add Versioncontrol to Storj-Sim * Testplanet Fixes * Linting * Add Error Handling and new Server Struct * Move Lock slightly * Reduce Race Potentials * Remove unnecessary files * Linting * Add Proper Transport Handling * small fixes * add fence for allowed check * Add Startup Version Check and Service Naming * make errormessage private * Add Comments about VersionedClient * Linting * Remove Checks that refuse outgoing connections * Remove release cmd * Add Release Script * Linting * Update to use correct Values * Move vars private and set minimum default versions for testing builds * Remove VersionedClient * Better Error Handling and naked return removal * Straighten the Regex and string conversion * Change Check to allows testplanet and storj-sim to run without the need to pass an LDFlag * Cosmetic Change to Dashboard * Cleanup Returns and remove commented code * Remove Version Check if no build options are passed in * Pass in Config Values instead of Pointers * Handle missed Error * Update Endpoint URL * Change Type of Release Flag * Add additional Logging * Remove Versions Logging of other Services * minor fixes Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
2019-04-03 20:13:39 +01:00
if node.Address == nil || node.Address.Address == "" {
Transport Client (#89) * port changes * Task monitor and setup merge from the staging * Restructure + additional interface * Add NewOverlayClient * integrated DHT client interface * added test for interface * PR comments addressed * lint issue * added generated protobuf * adding new interface * added the interface framework * deleted file * fixes compilation errors and integrates new dhtcclient interface * merged netstat latest changes and dht new interface chagnes * fixed the address's port * adding comments * PR comments addressed * netclient interface dial method added * rename and integrated transportclient with minio gateway * rename and code clean up * made changes based on the Dennis's changes on the kad-client * Code review comment changes based on kaloyan review comments * reverted the changes to be similar to master * removed unused file * renamed to transportclient * added the review changes * store the address of the client * updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters * updated the test case to test the bad address passed condition * updated the code per code review comments * Bolt backed overlay cache (#94) * wip * add separate `Process` tests for bolt and redis-backed overlay * more testing * fix gitignore * fix linter error * goimports goimports GOIMPORTS GoImPortS!!!! * fix port madness * forgot to add * add `mux` as handler and shorten context timeouts * gofreakingimports * fix comments * refactor test & add logger/monkit registry * debugging travis * add comment * Set redisAddress to empty string for bolt-test * travis experiment * refactoring tests * Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache * Automatically build, tag and push docker images on merge to master (#103) * port changes * build overlay on successful merge to master * fixes to Makefile * permissions * dep ensure * gopath * let's try vgo * remove dep * maybe alpine is the issue * tagging go version on build * stupid vgo * vgo * adding tags to push * quotes * local linting fixes & stupid travis * prepend storjlabs to docker tag (#108) * port changes * fixing tag name * Use continue instead of return in table tests (#106) I did a dumb mistake for some of the table tests, which made some of the test cases not being executed. * pkg/kademlia tests and restructuring (#97) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * files created * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * remove bkad dependencie from tests * wip * wip * wip * wip * wip * updated coyle/kademlia * wip * cleanup * ports * overlay upgraded * linter fixes * piecestore kademlia newID * add changes from kad demo * PR comments addresses * go func * force travis build * fixed merge conflicts * fixed merge conflicts * Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests * linter issues * linting issues * fixed merge conflicts * linter is stupid * Coyle/docker fix (#109) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge branch 'master' of https://github.com/storj/storj * fixing tag name * no idea * testing * changes * testing on travis * testing * changes to travis build * new approach * Merge branch 'master' into coyle/docker-fix * hardcode version (#111) * hardcode version * adding coveralls / code coverage (#112) * adding coveralls * adding code coverage badge * fixing badges * verbose * swap tests and coverage * extra line * maybe * maybe * moar * gover maybe * testing * cleanup * protos/netstate: remove stuff we're not using (#100) * protos/netstate: remove stuff we're not using * protos/netstate: add metadata field for segmentstore * fix netstate client test * pkg/process: start replacing pkg/process with cobra helpers (#98) * Implement psclient interface (#107) * Implement psclient interface * Add string method to pieceID type * try to fix linter errors * Whoops missed an error * More linter errors * Typo * Lol double typo * Get everything working, begin adding tests for psclient rpc * goimports * Forgot to change the piecestore cli when changed the piecestore code * Fix CLI * remove ID length, added validator to pieceID * Move grpc ranger to client Change client PUT api to take a reader rather than return a writer * GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser * Forgot to remove offset * Added message upon successful store * Do that thing dennis and kaloyan wanted * goimports * Make closeConn a part of the interface for psclient * Use interface * Removed uneccessary new lines * goimport * Whoops * Actually we don't want to use the interface in Piece Ranger * Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples * Make comments look nicer * modified transport client based on the the design discussion * modified transport client based on the the design discussion * added the as discussed connection cache interface functionality * added the as discussed connection cache interface functionality * transport client changes * transport client per code review changes * per the code review comments * transport client incorporates review comments * fixes lint warnings * lint warning fixes....client interface has to be Client * client.go changes * transport.go changes * added test case * added test cases * comment fix * comment fix
2018-06-29 19:28:06 +01:00
return nil, Error.New("no address")
}
dialOption, err := transport.tlsOpts.DialOption(node.Id)
if err != nil {
return nil, err
}
options := append([]grpc.DialOption{
dialOption,
grpc.WithBlock(),
grpc.FailOnNonTempDialError(true),
grpc.WithContextDialer(func(ctx context.Context, addr string) (net.Conn, error) {
conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", addr)
if err != nil {
return nil, err
}
return &timeoutConn{conn: conn, timeout: transport.timeouts.Request}, nil
}),
}, opts...)
timedCtx, cancel := context.WithTimeout(ctx, transport.timeouts.Dial)
defer cancel()
conn, err = grpc.DialContext(timedCtx, node.GetAddress().Address, options...)
if err != nil {
2019-02-06 13:19:14 +00:00
if err == context.Canceled {
return nil, err
}
transport.AlertFail(timedCtx, node, err)
return nil, Error.Wrap(err)
}
transport.AlertSuccess(timedCtx, node)
return conn, nil
}
// DialAddress returns a grpc connection with tls to an IP address.
//
// Do not use this method unless having a good reason. In most cases DialNode
// should be used for communicating with nodes as it is more secure than
// DialAddress.
2018-11-19 14:40:01 +00:00
func (transport *Transport) DialAddress(ctx context.Context, address string, opts ...grpc.DialOption) (conn *grpc.ClientConn, err error) {
defer mon.Task()(&ctx)(&err)
options := append([]grpc.DialOption{
transport.tlsOpts.DialUnverifiedIDOption(),
grpc.WithBlock(),
grpc.FailOnNonTempDialError(true),
grpc.WithContextDialer(func(ctx context.Context, addr string) (net.Conn, error) {
conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", addr)
if err != nil {
return nil, err
}
return &timeoutConn{conn: conn, timeout: transport.timeouts.Request}, nil
}),
}, opts...)
timedCtx, cancel := context.WithTimeout(ctx, transport.timeouts.Dial)
defer cancel()
// TODO: this should also call alertFail or alertSuccess with the node id. We should be able
// to get gRPC to give us the node id after dialing?
conn, err = grpc.DialContext(timedCtx, address, options...)
2019-02-06 13:19:14 +00:00
if err == context.Canceled {
return nil, err
}
return conn, Error.Wrap(err)
Transport Client (#89) * port changes * Task monitor and setup merge from the staging * Restructure + additional interface * Add NewOverlayClient * integrated DHT client interface * added test for interface * PR comments addressed * lint issue * added generated protobuf * adding new interface * added the interface framework * deleted file * fixes compilation errors and integrates new dhtcclient interface * merged netstat latest changes and dht new interface chagnes * fixed the address's port * adding comments * PR comments addressed * netclient interface dial method added * rename and integrated transportclient with minio gateway * rename and code clean up * made changes based on the Dennis's changes on the kad-client * Code review comment changes based on kaloyan review comments * reverted the changes to be similar to master * removed unused file * renamed to transportclient * added the review changes * store the address of the client * updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters * updated the test case to test the bad address passed condition * updated the code per code review comments * Bolt backed overlay cache (#94) * wip * add separate `Process` tests for bolt and redis-backed overlay * more testing * fix gitignore * fix linter error * goimports goimports GOIMPORTS GoImPortS!!!! * fix port madness * forgot to add * add `mux` as handler and shorten context timeouts * gofreakingimports * fix comments * refactor test & add logger/monkit registry * debugging travis * add comment * Set redisAddress to empty string for bolt-test * travis experiment * refactoring tests * Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache * Automatically build, tag and push docker images on merge to master (#103) * port changes * build overlay on successful merge to master * fixes to Makefile * permissions * dep ensure * gopath * let's try vgo * remove dep * maybe alpine is the issue * tagging go version on build * stupid vgo * vgo * adding tags to push * quotes * local linting fixes & stupid travis * prepend storjlabs to docker tag (#108) * port changes * fixing tag name * Use continue instead of return in table tests (#106) I did a dumb mistake for some of the table tests, which made some of the test cases not being executed. * pkg/kademlia tests and restructuring (#97) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * files created * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * remove bkad dependencie from tests * wip * wip * wip * wip * wip * updated coyle/kademlia * wip * cleanup * ports * overlay upgraded * linter fixes * piecestore kademlia newID * add changes from kad demo * PR comments addresses * go func * force travis build * fixed merge conflicts * fixed merge conflicts * Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests * linter issues * linting issues * fixed merge conflicts * linter is stupid * Coyle/docker fix (#109) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge branch 'master' of https://github.com/storj/storj * fixing tag name * no idea * testing * changes * testing on travis * testing * changes to travis build * new approach * Merge branch 'master' into coyle/docker-fix * hardcode version (#111) * hardcode version * adding coveralls / code coverage (#112) * adding coveralls * adding code coverage badge * fixing badges * verbose * swap tests and coverage * extra line * maybe * maybe * moar * gover maybe * testing * cleanup * protos/netstate: remove stuff we're not using (#100) * protos/netstate: remove stuff we're not using * protos/netstate: add metadata field for segmentstore * fix netstate client test * pkg/process: start replacing pkg/process with cobra helpers (#98) * Implement psclient interface (#107) * Implement psclient interface * Add string method to pieceID type * try to fix linter errors * Whoops missed an error * More linter errors * Typo * Lol double typo * Get everything working, begin adding tests for psclient rpc * goimports * Forgot to change the piecestore cli when changed the piecestore code * Fix CLI * remove ID length, added validator to pieceID * Move grpc ranger to client Change client PUT api to take a reader rather than return a writer * GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser * Forgot to remove offset * Added message upon successful store * Do that thing dennis and kaloyan wanted * goimports * Make closeConn a part of the interface for psclient * Use interface * Removed uneccessary new lines * goimport * Whoops * Actually we don't want to use the interface in Piece Ranger * Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples * Make comments look nicer * modified transport client based on the the design discussion * modified transport client based on the the design discussion * added the as discussed connection cache interface functionality * added the as discussed connection cache interface functionality * transport client changes * transport client per code review changes * per the code review comments * transport client incorporates review comments * fixes lint warnings * lint warning fixes....client interface has to be Client * client.go changes * transport.go changes * added test case * added test cases * comment fix * comment fix
2018-06-29 19:28:06 +01:00
}
// Identity is a getter for the transport's identity
2019-01-30 20:47:21 +00:00
func (transport *Transport) Identity() *identity.FullIdentity {
return transport.tlsOpts.Ident
}
// WithObservers returns a new transport including the listed observers.
Add Versioning Server (#1576) * Initial Webserver Draft for Version Controlling * Rename type to avoid confusion * Move Function Calls into Version Package * Fix Linting and Language Typos * Fix Linting and Spelling Mistakes * Include Copyright * Include Copyright * Adjust Version-Control Server to return list of Versions * Linting * Improve Request Handling and Readability * Add Configuration File Option Add Systemd Service file * Add Logging to File * Smaller Changes * Add Semantic Versioning and refuses outdated Software from Startup (#1612) * implements internal Semantic Version library * adds version logging + reporting to process * Advance SemVer struct for easier handling * Add Accepted Version Store * Fix Function * Restructure * Type Conversion * Handle Version String properly * Add Note about array index * Set temporary Default Version * Add Copyright * Adding Version to Dashboard * Adding Version Info Log * Renaming and adding CheckerProcess * Iteration Sync * Iteration V2 * linting * made LogAndReportVersion a go routine * Refactor to Go Routine * Add Context to Go Routine and allow Operation if Lookup to Control Server fails * Handle Unmarshal properly * Linting * Relocate Version Checks * Relocating Version Check and specified default Version for now * Linting Error Prevention * Refuse Startup on outdated Version * Add Startup Check Function * Straighten Logging * Dont force Shutdown if --dev flag is set * Create full Service/Peer Structure for ControlServer * Linting * Straighting Naming * Finish VersionControl Service Layout * Improve Error Handling * Change Listening Address * Move Checker Function * Remove VersionControl Peer * Linting * Linting * Create VersionClient Service * Renaming * Add Version Client to Peer Definitions * Linting and Renaming * Linting * Remove Transport Checks for now * Move to Client Side Flag * Remove check * Linting * Transport Client Version Intro * Adding Version Client to Transport Client * Add missing parameter * Adding Version Check, to set Allowed = true * Set Default to true, testing * Restructuring Code * Uplink Changes * Add more proper Defaults * Renaming of Version struct * Dont pass Service use Pointer * Set Defaults for Versioning Checks * Put HTTP Server in go routine * Add Versioncontrol to Storj-Sim * Testplanet Fixes * Linting * Add Error Handling and new Server Struct * Move Lock slightly * Reduce Race Potentials * Remove unnecessary files * Linting * Add Proper Transport Handling * small fixes * add fence for allowed check * Add Startup Version Check and Service Naming * make errormessage private * Add Comments about VersionedClient * Linting * Remove Checks that refuse outgoing connections * Remove release cmd * Add Release Script * Linting * Update to use correct Values * Move vars private and set minimum default versions for testing builds * Remove VersionedClient * Better Error Handling and naked return removal * Straighten the Regex and string conversion * Change Check to allows testplanet and storj-sim to run without the need to pass an LDFlag * Cosmetic Change to Dashboard * Cleanup Returns and remove commented code * Remove Version Check if no build options are passed in * Pass in Config Values instead of Pointers * Handle missed Error * Update Endpoint URL * Change Type of Release Flag * Add additional Logging * Remove Versions Logging of other Services * minor fixes Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
2019-04-03 20:13:39 +01:00
func (transport *Transport) WithObservers(obs ...Observer) Client {
tr := &Transport{tlsOpts: transport.tlsOpts, timeouts: transport.timeouts}
tr.observers = append(tr.observers, transport.observers...)
tr.observers = append(tr.observers, obs...)
return tr
}
// AlertFail alerts any subscribed observers of the failure 'err' for 'node'
func (transport *Transport) AlertFail(ctx context.Context, node *pb.Node, err error) {
defer mon.Task()(&ctx)(nil)
for _, o := range transport.observers {
o.ConnFailure(ctx, node, err)
}
}
// AlertSuccess alerts any subscribed observers of success for 'node'
func (transport *Transport) AlertSuccess(ctx context.Context, node *pb.Node) {
defer mon.Task()(&ctx)(nil)
for _, o := range transport.observers {
o.ConnSuccess(ctx, node)
}
}