* adds netstate rpc server pagination, mocks pagination in test/util.go
* updates ns client example, combines ns client and server test to netstate_test, adds pagination to bolt client
* better organizes netstate test calls
* wip breaking netstate test into smaller tests
* wip modularizing netstate tests
* adds some test panics
* wip netstate test attempts
* testing bug in netstate TestDeleteAuth
* wip fixes global variable problem, still issues with list
* wip fixes get request params and args
* fixes bug in path when using MakePointers helper fn
* updates mockdb list func, adds test, changes Limit to int
* fixes merge conflicts
* fixes broken tests from merge
* remove unnecessary PointerEntry struct
* removes error when Get returns nil value from boltdb
* breaks boltdb client tests into smaller tests
* renames AssertNoErr test helper to HandleErr
* adds StartingKey and Limit parameters to redis list func, adds beginning of redis tests
* adds helper func for mockdb List function
* if no starting key provided for netstate List, the first value in storage will be used
* adds basic pagination for redis List function, adds tests
* adds list limit to call in overlay/server.go
* streamlines/fixes some nits from review
* removes use of obsolete EncryptedUnencryptedSize
* uses MockKeyValueStore instead of redis instance in redis client test
* changes test to expect nil returned for getting missing key
* remove error from `KeyValueStore#Get`
* fix bolt test
* Merge pull request #1 from bryanchriswhite/nat-pagination
remove error from `KeyValueStore#Get`
* adds Get returning error back to KeyValueStore interface and affected clients
* trying to appease travis: returns errors in Get calls in overlay/cache and cache_test
* handles redis get error when no key found
* port changes
* Task monitor and setup merge from the staging
* Restructure + additional interface
* Add NewOverlayClient
* integrated DHT client interface
* added test for interface
* PR comments addressed
* lint issue
* added generated protobuf
* adding new interface
* added the interface framework
* deleted file
* fixes compilation errors and integrates new dhtcclient interface
* merged netstat latest changes and dht new interface chagnes
* fixed the address's port
* adding comments
* PR comments addressed
* netclient interface dial method added
* rename and integrated transportclient with minio gateway
* rename and code clean up
* made changes based on the Dennis's changes on the kad-client
* Code review comment changes based on kaloyan review comments
* reverted the changes to be similar to master
* removed unused file
* renamed to transportclient
* added the review changes
* store the address of the client
* updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters
* updated the test case to test the bad address passed condition
* updated the code per code review comments
* Bolt backed overlay cache (#94)
* wip
* add separate `Process` tests for bolt and redis-backed overlay
* more testing
* fix gitignore
* fix linter error
* goimports goimports GOIMPORTS GoImPortS!!!!
* fix port madness
* forgot to add
* add `mux` as handler and shorten context timeouts
* gofreakingimports
* fix comments
* refactor test & add logger/monkit registry
* debugging travis
* add comment
* Set redisAddress to empty string for bolt-test
* travis experiment
* refactoring tests
* Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache
* Automatically build, tag and push docker images on merge to master (#103)
* port changes
* build overlay on successful merge to master
* fixes to Makefile
* permissions
* dep ensure
* gopath
* let's try vgo
* remove dep
* maybe alpine is the issue
* tagging go version on build
* stupid vgo
* vgo
* adding tags to push
* quotes
* local linting fixes & stupid travis
* prepend storjlabs to docker tag (#108)
* port changes
* fixing tag name
* Use continue instead of return in table tests (#106)
I did a dumb mistake for some of the table tests, which made some of the
test cases not being executed.
* pkg/kademlia tests and restructuring (#97)
* port changes
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* files created
* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests
* wip
* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests
* wip
* remove bkad dependencie from tests
* wip
* wip
* wip
* wip
* wip
* updated coyle/kademlia
* wip
* cleanup
* ports
* overlay upgraded
* linter fixes
* piecestore kademlia newID
* add changes from kad demo
* PR comments addresses
* go func
* force travis build
* fixed merge conflicts
* fixed merge conflicts
* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests
* linter issues
* linting issues
* fixed merge conflicts
* linter is stupid
* Coyle/docker fix (#109)
* port changes
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Merge branch 'master' of https://github.com/storj/storj
* fixing tag name
* no idea
* testing
* changes
* testing on travis
* testing
* changes to travis build
* new approach
* Merge branch 'master' into coyle/docker-fix
* hardcode version (#111)
* hardcode version
* adding coveralls / code coverage (#112)
* adding coveralls
* adding code coverage badge
* fixing badges
* verbose
* swap tests and coverage
* extra line
* maybe
* maybe
* moar
* gover maybe
* testing
* cleanup
* protos/netstate: remove stuff we're not using (#100)
* protos/netstate: remove stuff we're not using
* protos/netstate: add metadata field for segmentstore
* fix netstate client test
* pkg/process: start replacing pkg/process with cobra helpers (#98)
* Implement psclient interface (#107)
* Implement psclient interface
* Add string method to pieceID type
* try to fix linter errors
* Whoops missed an error
* More linter errors
* Typo
* Lol double typo
* Get everything working, begin adding tests for psclient rpc
* goimports
* Forgot to change the piecestore cli when changed the piecestore code
* Fix CLI
* remove ID length, added validator to pieceID
* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer
* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser
* Forgot to remove offset
* Added message upon successful store
* Do that thing dennis and kaloyan wanted
* goimports
* Make closeConn a part of the interface for psclient
* Use interface
* Removed uneccessary new lines
* goimport
* Whoops
* Actually we don't want to use the interface in Piece Ranger
* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples
* Make comments look nicer
* modified transport client based on the the design discussion
* modified transport client based on the the design discussion
* added the as discussed connection cache interface functionality
* added the as discussed connection cache interface functionality
* transport client changes
* transport client per code review changes
* per the code review comments
* transport client incorporates review comments
* fixes lint warnings
* lint warning fixes....client interface has to be Client
* client.go changes
* transport.go changes
* added test case
* added test cases
* comment fix
* comment fix
* Implement psclient interface
* Add string method to pieceID type
* try to fix linter errors
* Whoops missed an error
* More linter errors
* Typo
* Lol double typo
* Get everything working, begin adding tests for psclient rpc
* goimports
* Forgot to change the piecestore cli when changed the piecestore code
* Fix CLI
* remove ID length, added validator to pieceID
* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer
* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser
* Forgot to remove offset
* Added message upon successful store
* Do that thing dennis and kaloyan wanted
* goimports
* Make closeConn a part of the interface for psclient
* Use interface
* Removed uneccessary new lines
* goimport
* Whoops
* Actually we don't want to use the interface in Piece Ranger
* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples
* Make comments look nicer
* port changes
* build overlay on successful merge to master
* fixes to Makefile
* permissions
* dep ensure
* gopath
* let's try vgo
* remove dep
* maybe alpine is the issue
* tagging go version on build
* stupid vgo
* vgo
* adding tags to push
* quotes
* local linting fixes & stupid travis
* internal/test: switch errors to error classes
if you construct an error directly at package init time, you
won't get useful stack traces or anything. zeebo/errs expects that
you always construct an error (.New) when the error actually
happens. instead, you call Class at init time, then use (Class).Has
to test for error type membership.
* fix linter
* fix test
* piecestore: connect to kad
* piecestore: Linter errors
* piecestore: added pstore config command line utility
* piecestore: removed main.go, implement methods and structs
* piecestore: Import Cam's config code into piecestore-farmer
* piecestore: moving farmer to urfavecli
* piecestore: added create command
* piecestore: Removed old config, added server start code to cli
* piecestore: Get server code working
* piecestore: Changed default dir for storing piece store data; added ttl to config
* piecestore: Generate id; add bootstrap ip for kad
* piecestore: Separate kad port and server port
* piecestore: goimports
* piecestore: Removed print
* piecestore: use pkg/process
* piecestore: Better config file
* piecestore: base58 encode for id
* piecestore: base58 encode and clean up cli
* piecestore: Typo
* piecestore: removed unnecessary variable
* piecestore: Fixed more typos
* piecestore: Place data in a directory based on nodeid
* piecestore: base58 encode instead
* piecestore: Add dependency to go.mod
* piecestore: Fix typo in rpc server start; clear data on failed piece upload
* add reference to dht to overlay client struct
* wip
* wip
* Implement FindNode
* get nodes
* WIP
* Merge in Dennis kademlia code, get it working with our code
* ping and moar
* WIP trying to get cache working with kademlia
* WIP more wiring up
* WIP
* Update service cli commands
* WIP
* added GetNodes
* added nodes to Kbucket
* default transport changed to TCP
* GetBuckets interface changed
* filling in more routing
* timestamp methods
* removed store
* Added initial network overlay explorer page
* Updating and building with dockerfile
* Working on adding bootstrap node code
* WIP merging in dennis' code
* WIP
* connects cache to pkg/kademlia implementation
* WIP redis cache
* testing
* Add bootstrap network function for CLI usage
* cleanup
* call bootstrap on init network
* Add BootstrapNetwork function to interface
* Merge in dennis kad code
* WIP updates to redis/overlay client interface
* WIP trying to get the DHT connected to the cache
* go mod & test
* deps
* Bootstrap node now setting up correctly
- Need to pass it through CLI commands better
* WIP adding refresh and walk functions, added cli flags
- added cli flags for custom bootstrap port and ip
* PR comments addressed
* adding FindStorageNodes to overlay cache
* fix GetBucket
* using SplitHostPort
* Use JoinHostPort
* updates to findstoragenodes response and request
* WIP merge in progress, having issues with a panic
* wip
* adjustments
* update port for dht bootstrap test
* Docker
* wip
* dockerfile
* fixes
* makefile changes
* Update port in NewKademlia call
* Update local kademlia DHT config
* kubernetes yaml
* cleanup
* making tests pass
* k8s yaml
* lint issues
* Edit cli flags to allow for configurable bootstrap IP and Port args
* cleanup
* cache walking the network now
* Rough prototype of Walk function laid out
* Move walk function into bootstrap function
* Update dht.go
* changes to yaml
* goimports
* wip
* wip
* get nodes
* ping and moar
* added GetNodes
* added nodes to Kbucket
* default transport changed to TCP
* GetBuckets interface changed
* filling in more routing
* timestamp methods
* removed store
* testing
* cleanup
* go mod & test
* deps
* PR comments addressed
* fix GetBucket
* using SplitHostPort
* Use JoinHostPort
* added spawn scripts
* wip
* Make server not require length
* Added test for 0 length
* Added test for accepting length of 0
* linter errors
* Make store not take anything more than an id and maybe a ttl if you want
* initial commit for PUT request authorization
* inital auth for put request
* auth. request working for all req
* deleted library
* removed .db files
* work in progress for modifying test suite to accomodate credentials
* modified tests
* gofmt, fixed code based on suggestions; test passed
* gofmt again
* merged nat's update on pointers, passed tests, cleanup from git rebase
* fixed fmt
* fixed fmt on tests
* work in progress
* reduced code
* fixed naming conventions
* added line in code
* fixed server bug, merged new code to server, added env
* fixed linter; getting cright issues on piecestore
* added comments for what passes on the creds
* added spawn scripts
* Determine random id for storing
* Moved determine id to rpc example
* Added tests
* Better test
* goimports
* Updated tests
* Fix typos
* Added piecestore
* gofmt
* Added requested changes
* Added cli
* Removed ranger because I wanted something that can stand alone
* Add example of http server using piece store
* Changed piecestore code to make it more optial for error handelling
* Merged with piecestore
* Added missing package
* Forgot io import
* gofmt
* gofmt
* Forgot io
* Make path by hash exported
* updated to simplify again whoops
* Updated server to work real good
* Forgot ampersand
* Updated to match FilePiece
* Merged in cam's delete code
* Remove unused io
* Added RPC code
* Give the download request a reader
* Removed http server stuff; changed receive stream to say io.reader
* Added expiration date to shardInfo
* gRPC Ranger
* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions
* Adapt to latest changes in piece store rpc api
* added ttl info request
* Initialize grpcRanger type with named fields
* Move scripts to http server pr; added close method for Retrieve api
* added rpc server tests for getting piece meta data and retrieval routes
* Adapt to PieceStreamReader now being a ReadCloser
* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf
* Imported cams test
* Bump gometalinter deadline
* Adapt to package name changes
* Remove Garbage
* Adapt to latest changes in piece store rpc api
* NewCustomRoute constructor to allow mocking the gRPC client
* Name struct values in constructor.
* Added piecestore
* gofmt
* Added requested changes
* Added cli
* Removed ranger because I wanted something that can stand alone
* Add example of http server using piece store
* Changed piecestore code to make it more optial for error handelling
* Merged with piecestore
* Added missing package
* Forgot io import
* gofmt
* gofmt
* Forgot io
* Make path by hash exported
* updated to simplify again whoops
* Updated server to work real good
* Forgot ampersand
* Updated to match FilePiece
* Merged in cam's delete code
* Remove unused io
* Added RPC code
* Give the download request a reader
* Removed http server stuff; changed receive stream to say io.reader
* Added expiration date to shardInfo
* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions
* added ttl info request
* Move scripts to http server pr; added close method for Retrieve api
* added rpc server tests for getting piece meta data and retrieval routes
* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf
* Imported cams test
* Bump gometalinter deadline
* WIP adding tests
* added tests for store and delete routes
* Add changes as requested by Kaloyan, also cleaned up some code
* Get the code actually working whoops
* More cleanup
* Separating database calls from api.go
* need to rename expiration
* Added some changes requested by JT
* Fix read size
* Fixed total amount to read
* added tests
* Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages
* Moved rpc client and server to piece store
* Moved piecestore protobuf to the /protos folder
* Cleaned up messages
* Clean up linter errors
* Added missing sqlite import
* Add ability to do iterative reads and writes to pstore
* Incrementally read data
* Fix linter and import errors
* Solve linter Error
* Change return types
* begin test refactor
* refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go
* Cleaning up tests
* Added retrieve tests
* refactored delete tests
* Deleted old tests
* Updated cmd/piecestore to reflect changes to piecestore
* Refactored server tests and server/client store code
* gofmt
* WIP implementing TTL struct
* Read 4k at a time when Retrieving
* implemented ttl struct
* Accidentally removed fpiece dependency?
* Double resolve merge conflict
* Moved client to the examples since it is an example
* Change hash to id in protobuf. TODO: Change client and server to reflect these changes
* Update client and protobuf
* changed hash to id
* Handle eof properly in retrieve
* Id -> ID
* forgot to change import for client after moving
* Moved client and server main to examples
* Made changes requested by JT
* checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close()
* Print not fatal
* Handle overflow when reading from server
* added const IDLength
* removed types from comments
* Add reader/writer for download data from and uploading data to server
* goimports and comments
* fixed nits, casts, added OK const, DBCleanup now exits program on error
* Add stream reader and writer for server
* Fix errors
* i beforee except after c lol
* customizable data dir
* Forgot to rename variable
* customizable data dir
* Handle closing of server stream properly
* linter
* pass on inserting the same data twice
* linter
* linter
* Do the easy things JT asked for
* Handle overflow from reads properly; handle custom db path
* Handle overflow for server stream read; TODO Combine server and client stream reads
* Allow for TTL of 0 to stay forever
* Make Client cleaner for user
* integrated eestream's serve-pieces functionality
* added ref to http request function
* created dummy bucket list
* Initialized the buckets with files with hardcoded sample data
* supports upload Object(s)
* uploads to corresponding folders
* code cleanup for review
* updated based on code review comments
* updates based on missed code review comments
* updated with review comments
* implemented review comments
* merged latest and tested
* added filepath.Join()
* updates based on the comments
* fixes the eestreamer parameter due to merge
* Moved filepiece into storj
* Fix linter errors
* Seek comment for linter
* gofmt/golinter accidentally removed import
* Fix small typos
* Use the weird iota. P cool dude ✌️
* Do things the cool way
* Changes requested by kaloyan
* didn't need test main
* Path encryption library
* Use base64 instead of hex encoding
* Prepend version number to encrypted path segments
* Remove redundant var alias
* Simplified returns
* wrap errors
* Buffered eestream EncodeReader
* Extracted fillBuffer() helper function
* Slow channels will be closed if there are still at least k fast channels
* Doc comment for maxBufferMemory
* Use timer more efficiently
* Timer initialization should be inside the for-loop
* Parallel copy of encoded data to reader buffer channels
* Transfer input read errors to output encoded readers
* minimum and optimum thresholds
* Use time.AfterFunc
* Simplify error handling in constructor
* adds pointer to netstate proto file
* generated updated netstate proto
* changes boltdb netstate to save pointers as values
* updates netstate Put to save Pointers, updates client example to put a pointer, adds grpc status errors, updates tests, changes boltdb 'File' struct to 'PointerEntry'
* updates netstate client example and client test to save pointers, updates netstate List and Delete
* begins adding netstate-http tests
* removes netstate http service
* re-adds netstate auth
* updates boltdb netstate test
* changes encrypted_unencrypted_size from int64 to bytes in netstate proto
* updates READMEs
* Added piecestore
* gofmt
* Added requested changes
* Added cli
* Removed ranger because I wanted something that can stand alone
* Changed piecestore code to make it more optial for error handelling
* Added missing package
* Forgot io import
* gofmt
* Make path by hash exported
* updated to simplify again whoops
* Forgot ampersand
* Updated to match FilePiece
* Change store to use io.CopyN
* Clear golinter errors
* Updated go.mod
* Updated go.mod
* initial commit
* created httpreq struct
* checked against env key
* wrapped up comparing xapikey to flag input
* added readme
* added example and updated code with viper
* added ConstantTimeCompare function
* fixed spacing
* updated with comments, and updated readme
* updated main.go with setting env variable
* fixed linter errors
* AES GCM implementation and unit test code
* modified and tested per the code review comments
* modified and tested per the code review comments
* updated to return err
* removed the debug printf commented code
* support of aes-gcm
* updated the renaming convention per GOLANG coding standards
* Initial Go & C biniding with libstorj
* Initial GO & C bindings with libstorj
* fixing the callback
* moved a millimeter :-) on c to go to gone....
* added error condition, per review comment
* removed the .idea directory and also movie.avi file
* deleting files not to be in this pull request
* Revert "deleting files not to be in this pull request"
This reverts commit 026b834fe00f6b20a7566e71973fe224c12f533f.
* deleting files not to be in this pull request
* resolving conflicts
* syncing the file to master
* Use aes gcm in eestream rs tests
* Use aes gcm in serve example
* Fixed comment
* adds proto files for netstate crud
* moves netstate grpc client lib into pkg/netstate where grpc netstate service is defined
* starts adding grpc client and server tests
* moves creation of grpc server into cmd/netstate/main.go, removes pkg/netstate/service.go, adds more client testing
* changed all 'Path' and 'Value' fields from strings to bytes, updated tests
* changes Get and Delete in proto file to receive 'requests' instead of 'file paths', adds tests for Get, List, and Delete
* changes netstate-routes to get 'fileValue' bytes not 'fileInfo'
* adds example rpc client in 'examples' and adds more specific debug logs
* adds readmes for netstate rpc services and updates netstate-routes
