* license code with agplv3
now that we have clabot, license everything by default with agplv3.
since we have CLAs, we can relicense certain parts (such as the
s3 gateway and libstorj) later.
* Merge remote-tracking branch 'origin/master' into agpl
* separated commands and actions from main.go, created delete command
* refactored piecestore-farmer to use cobra CLI
* added piecestore-farmer main.go and long command descriptions
* pkg/process: start replacing pkg/process with cobra helpers
* address review comments
* addressed github comments, rename pstore.DetermineID to GenerateID, remove utils.NewID and use pstore.GenerateID instead
* addressed comments
* pulled proc and got CLI working with process package
* goimports
* stdout formatting
* WIP adding tests for pstoreFarmer CLI, fixed bug in check for preexisting config
* copyright header, change exported variable
* changed error formatting to be consistent
* more tests
* stdout, add (\n)s in zap logs, set home inside SetConfigPath() instead of passing in
* resolved merge conflicts
* goimports and fixed merge error
* removed generateID func to use kademlia.NewID instead, refactored to match kademlia.NewKademlia()
* adds netstate rpc server pagination, mocks pagination in test/util.go
* updates ns client example, combines ns client and server test to netstate_test, adds pagination to bolt client
* better organizes netstate test calls
* wip breaking netstate test into smaller tests
* wip modularizing netstate tests
* adds some test panics
* wip netstate test attempts
* testing bug in netstate TestDeleteAuth
* wip fixes global variable problem, still issues with list
* wip fixes get request params and args
* fixes bug in path when using MakePointers helper fn
* updates mockdb list func, adds test, changes Limit to int
* fixes merge conflicts
* fixes broken tests from merge
* remove unnecessary PointerEntry struct
* removes error when Get returns nil value from boltdb
* breaks boltdb client tests into smaller tests
* renames AssertNoErr test helper to HandleErr
* adds StartingKey and Limit parameters to redis list func, adds beginning of redis tests
* adds helper func for mockdb List function
* if no starting key provided for netstate List, the first value in storage will be used
* adds basic pagination for redis List function, adds tests
* adds list limit to call in overlay/server.go
* streamlines/fixes some nits from review
* removes use of obsolete EncryptedUnencryptedSize
* uses MockKeyValueStore instead of redis instance in redis client test
* changes test to expect nil returned for getting missing key
* remove error from `KeyValueStore#Get`
* fix bolt test
* Merge pull request #1 from bryanchriswhite/nat-pagination
remove error from `KeyValueStore#Get`
* adds Get returning error back to KeyValueStore interface and affected clients
* trying to appease travis: returns errors in Get calls in overlay/cache and cache_test
* handles redis get error when no key found
* port changes
* Task monitor and setup merge from the staging
* Restructure + additional interface
* Add NewOverlayClient
* integrated DHT client interface
* added test for interface
* PR comments addressed
* lint issue
* added generated protobuf
* adding new interface
* added the interface framework
* deleted file
* fixes compilation errors and integrates new dhtcclient interface
* merged netstat latest changes and dht new interface chagnes
* fixed the address's port
* adding comments
* PR comments addressed
* netclient interface dial method added
* rename and integrated transportclient with minio gateway
* rename and code clean up
* made changes based on the Dennis's changes on the kad-client
* Code review comment changes based on kaloyan review comments
* reverted the changes to be similar to master
* removed unused file
* renamed to transportclient
* added the review changes
* store the address of the client
* updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters
* updated the test case to test the bad address passed condition
* updated the code per code review comments
* Bolt backed overlay cache (#94)
* wip
* add separate `Process` tests for bolt and redis-backed overlay
* more testing
* fix gitignore
* fix linter error
* goimports goimports GOIMPORTS GoImPortS!!!!
* fix port madness
* forgot to add
* add `mux` as handler and shorten context timeouts
* gofreakingimports
* fix comments
* refactor test & add logger/monkit registry
* debugging travis
* add comment
* Set redisAddress to empty string for bolt-test
* travis experiment
* refactoring tests
* Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache
* Automatically build, tag and push docker images on merge to master (#103)
* port changes
* build overlay on successful merge to master
* fixes to Makefile
* permissions
* dep ensure
* gopath
* let's try vgo
* remove dep
* maybe alpine is the issue
* tagging go version on build
* stupid vgo
* vgo
* adding tags to push
* quotes
* local linting fixes & stupid travis
* prepend storjlabs to docker tag (#108)
* port changes
* fixing tag name
* Use continue instead of return in table tests (#106)
I did a dumb mistake for some of the table tests, which made some of the
test cases not being executed.
* pkg/kademlia tests and restructuring (#97)
* port changes
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* files created
* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests
* wip
* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests
* wip
* remove bkad dependencie from tests
* wip
* wip
* wip
* wip
* wip
* updated coyle/kademlia
* wip
* cleanup
* ports
* overlay upgraded
* linter fixes
* piecestore kademlia newID
* add changes from kad demo
* PR comments addresses
* go func
* force travis build
* fixed merge conflicts
* fixed merge conflicts
* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests
* linter issues
* linting issues
* fixed merge conflicts
* linter is stupid
* Coyle/docker fix (#109)
* port changes
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Merge branch 'master' of https://github.com/storj/storj
* fixing tag name
* no idea
* testing
* changes
* testing on travis
* testing
* changes to travis build
* new approach
* Merge branch 'master' into coyle/docker-fix
* hardcode version (#111)
* hardcode version
* adding coveralls / code coverage (#112)
* adding coveralls
* adding code coverage badge
* fixing badges
* verbose
* swap tests and coverage
* extra line
* maybe
* maybe
* moar
* gover maybe
* testing
* cleanup
* protos/netstate: remove stuff we're not using (#100)
* protos/netstate: remove stuff we're not using
* protos/netstate: add metadata field for segmentstore
* fix netstate client test
* pkg/process: start replacing pkg/process with cobra helpers (#98)
* Implement psclient interface (#107)
* Implement psclient interface
* Add string method to pieceID type
* try to fix linter errors
* Whoops missed an error
* More linter errors
* Typo
* Lol double typo
* Get everything working, begin adding tests for psclient rpc
* goimports
* Forgot to change the piecestore cli when changed the piecestore code
* Fix CLI
* remove ID length, added validator to pieceID
* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer
* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser
* Forgot to remove offset
* Added message upon successful store
* Do that thing dennis and kaloyan wanted
* goimports
* Make closeConn a part of the interface for psclient
* Use interface
* Removed uneccessary new lines
* goimport
* Whoops
* Actually we don't want to use the interface in Piece Ranger
* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples
* Make comments look nicer
* modified transport client based on the the design discussion
* modified transport client based on the the design discussion
* added the as discussed connection cache interface functionality
* added the as discussed connection cache interface functionality
* transport client changes
* transport client per code review changes
* per the code review comments
* transport client incorporates review comments
* fixes lint warnings
* lint warning fixes....client interface has to be Client
* client.go changes
* transport.go changes
* added test case
* added test cases
* comment fix
* comment fix
* Implement psclient interface
* Add string method to pieceID type
* try to fix linter errors
* Whoops missed an error
* More linter errors
* Typo
* Lol double typo
* Get everything working, begin adding tests for psclient rpc
* goimports
* Forgot to change the piecestore cli when changed the piecestore code
* Fix CLI
* remove ID length, added validator to pieceID
* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer
* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser
* Forgot to remove offset
* Added message upon successful store
* Do that thing dennis and kaloyan wanted
* goimports
* Make closeConn a part of the interface for psclient
* Use interface
* Removed uneccessary new lines
* goimport
* Whoops
* Actually we don't want to use the interface in Piece Ranger
* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples
* Make comments look nicer
* port changes
* build overlay on successful merge to master
* fixes to Makefile
* permissions
* dep ensure
* gopath
* let's try vgo
* remove dep
* maybe alpine is the issue
* tagging go version on build
* stupid vgo
* vgo
* adding tags to push
* quotes
* local linting fixes & stupid travis
* internal/test: switch errors to error classes
if you construct an error directly at package init time, you
won't get useful stack traces or anything. zeebo/errs expects that
you always construct an error (.New) when the error actually
happens. instead, you call Class at init time, then use (Class).Has
to test for error type membership.
* fix linter
* fix test
* piecestore: connect to kad
* piecestore: Linter errors
* piecestore: added pstore config command line utility
* piecestore: removed main.go, implement methods and structs
* piecestore: Import Cam's config code into piecestore-farmer
* piecestore: moving farmer to urfavecli
* piecestore: added create command
* piecestore: Removed old config, added server start code to cli
* piecestore: Get server code working
* piecestore: Changed default dir for storing piece store data; added ttl to config
* piecestore: Generate id; add bootstrap ip for kad
* piecestore: Separate kad port and server port
* piecestore: goimports
* piecestore: Removed print
* piecestore: use pkg/process
* piecestore: Better config file
* piecestore: base58 encode for id
* piecestore: base58 encode and clean up cli
* piecestore: Typo
* piecestore: removed unnecessary variable
* piecestore: Fixed more typos
* piecestore: Place data in a directory based on nodeid
* piecestore: base58 encode instead
* piecestore: Add dependency to go.mod
* piecestore: Fix typo in rpc server start; clear data on failed piece upload
* add reference to dht to overlay client struct
* wip
* wip
* Implement FindNode
* get nodes
* WIP
* Merge in Dennis kademlia code, get it working with our code
* ping and moar
* WIP trying to get cache working with kademlia
* WIP more wiring up
* WIP
* Update service cli commands
* WIP
* added GetNodes
* added nodes to Kbucket
* default transport changed to TCP
* GetBuckets interface changed
* filling in more routing
* timestamp methods
* removed store
* Added initial network overlay explorer page
* Updating and building with dockerfile
* Working on adding bootstrap node code
* WIP merging in dennis' code
* WIP
* connects cache to pkg/kademlia implementation
* WIP redis cache
* testing
* Add bootstrap network function for CLI usage
* cleanup
* call bootstrap on init network
* Add BootstrapNetwork function to interface
* Merge in dennis kad code
* WIP updates to redis/overlay client interface
* WIP trying to get the DHT connected to the cache
* go mod & test
* deps
* Bootstrap node now setting up correctly
- Need to pass it through CLI commands better
* WIP adding refresh and walk functions, added cli flags
- added cli flags for custom bootstrap port and ip
* PR comments addressed
* adding FindStorageNodes to overlay cache
* fix GetBucket
* using SplitHostPort
* Use JoinHostPort
* updates to findstoragenodes response and request
* WIP merge in progress, having issues with a panic
* wip
* adjustments
* update port for dht bootstrap test
* Docker
* wip
* dockerfile
* fixes
* makefile changes
* Update port in NewKademlia call
* Update local kademlia DHT config
* kubernetes yaml
* cleanup
* making tests pass
* k8s yaml
* lint issues
* Edit cli flags to allow for configurable bootstrap IP and Port args
* cleanup
* cache walking the network now
* Rough prototype of Walk function laid out
* Move walk function into bootstrap function
* Update dht.go
* changes to yaml
* goimports
* wip
* wip
* get nodes
* ping and moar
* added GetNodes
* added nodes to Kbucket
* default transport changed to TCP
* GetBuckets interface changed
* filling in more routing
* timestamp methods
* removed store
* testing
* cleanup
* go mod & test
* deps
* PR comments addressed
* fix GetBucket
* using SplitHostPort
* Use JoinHostPort
* added spawn scripts
* wip
* Make server not require length
* Added test for 0 length
* Added test for accepting length of 0
* linter errors
* Make store not take anything more than an id and maybe a ttl if you want
* initial commit for PUT request authorization
* inital auth for put request
* auth. request working for all req
* deleted library
* removed .db files
* work in progress for modifying test suite to accomodate credentials
* modified tests
* gofmt, fixed code based on suggestions; test passed
* gofmt again
* merged nat's update on pointers, passed tests, cleanup from git rebase
* fixed fmt
* fixed fmt on tests
* work in progress
* reduced code
* fixed naming conventions
* added line in code
* fixed server bug, merged new code to server, added env
* fixed linter; getting cright issues on piecestore
* added comments for what passes on the creds
* added spawn scripts
* Determine random id for storing
* Moved determine id to rpc example
* Added tests
* Better test
* goimports
* Updated tests
* Fix typos
* Added piecestore
* gofmt
* Added requested changes
* Added cli
* Removed ranger because I wanted something that can stand alone
* Add example of http server using piece store
* Changed piecestore code to make it more optial for error handelling
* Merged with piecestore
* Added missing package
* Forgot io import
* gofmt
* gofmt
* Forgot io
* Make path by hash exported
* updated to simplify again whoops
* Updated server to work real good
* Forgot ampersand
* Updated to match FilePiece
* Merged in cam's delete code
* Remove unused io
* Added RPC code
* Give the download request a reader
* Removed http server stuff; changed receive stream to say io.reader
* Added expiration date to shardInfo
* gRPC Ranger
* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions
* Adapt to latest changes in piece store rpc api
* added ttl info request
* Initialize grpcRanger type with named fields
* Move scripts to http server pr; added close method for Retrieve api
* added rpc server tests for getting piece meta data and retrieval routes
* Adapt to PieceStreamReader now being a ReadCloser
* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf
* Imported cams test
* Bump gometalinter deadline
* Adapt to package name changes
* Remove Garbage
* Adapt to latest changes in piece store rpc api
* NewCustomRoute constructor to allow mocking the gRPC client
* Name struct values in constructor.
* Added piecestore
* gofmt
* Added requested changes
* Added cli
* Removed ranger because I wanted something that can stand alone
* Add example of http server using piece store
* Changed piecestore code to make it more optial for error handelling
* Merged with piecestore
* Added missing package
* Forgot io import
* gofmt
* gofmt
* Forgot io
* Make path by hash exported
* updated to simplify again whoops
* Updated server to work real good
* Forgot ampersand
* Updated to match FilePiece
* Merged in cam's delete code
* Remove unused io
* Added RPC code
* Give the download request a reader
* Removed http server stuff; changed receive stream to say io.reader
* Added expiration date to shardInfo
* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions
* added ttl info request
* Move scripts to http server pr; added close method for Retrieve api
* added rpc server tests for getting piece meta data and retrieval routes
* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf
* Imported cams test
* Bump gometalinter deadline
* WIP adding tests
* added tests for store and delete routes
* Add changes as requested by Kaloyan, also cleaned up some code
* Get the code actually working whoops
* More cleanup
* Separating database calls from api.go
* need to rename expiration
* Added some changes requested by JT
* Fix read size
* Fixed total amount to read
* added tests
* Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages
* Moved rpc client and server to piece store
* Moved piecestore protobuf to the /protos folder
* Cleaned up messages
* Clean up linter errors
* Added missing sqlite import
* Add ability to do iterative reads and writes to pstore
* Incrementally read data
* Fix linter and import errors
* Solve linter Error
* Change return types
* begin test refactor
* refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go
* Cleaning up tests
* Added retrieve tests
* refactored delete tests
* Deleted old tests
* Updated cmd/piecestore to reflect changes to piecestore
* Refactored server tests and server/client store code
* gofmt
* WIP implementing TTL struct
* Read 4k at a time when Retrieving
* implemented ttl struct
* Accidentally removed fpiece dependency?
* Double resolve merge conflict
* Moved client to the examples since it is an example
* Change hash to id in protobuf. TODO: Change client and server to reflect these changes
* Update client and protobuf
* changed hash to id
* Handle eof properly in retrieve
* Id -> ID
* forgot to change import for client after moving
* Moved client and server main to examples
* Made changes requested by JT
* checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close()
* Print not fatal
* Handle overflow when reading from server
* added const IDLength
* removed types from comments
* Add reader/writer for download data from and uploading data to server
* goimports and comments
* fixed nits, casts, added OK const, DBCleanup now exits program on error
* Add stream reader and writer for server
* Fix errors
* i beforee except after c lol
* customizable data dir
* Forgot to rename variable
* customizable data dir
* Handle closing of server stream properly
* linter
* pass on inserting the same data twice
* linter
* linter
* Do the easy things JT asked for
* Handle overflow from reads properly; handle custom db path
* Handle overflow for server stream read; TODO Combine server and client stream reads
* Allow for TTL of 0 to stay forever
* Make Client cleaner for user
* integrated eestream's serve-pieces functionality
* added ref to http request function
* created dummy bucket list
* Initialized the buckets with files with hardcoded sample data
* supports upload Object(s)
* uploads to corresponding folders
* code cleanup for review
* updated based on code review comments
* updates based on missed code review comments
* updated with review comments
* implemented review comments
* merged latest and tested
* added filepath.Join()
* updates based on the comments
* fixes the eestreamer parameter due to merge
* Moved filepiece into storj
* Fix linter errors
* Seek comment for linter
* gofmt/golinter accidentally removed import
* Fix small typos
* Use the weird iota. P cool dude ✌️
* Do things the cool way
* Changes requested by kaloyan
* didn't need test main
* Path encryption library
* Use base64 instead of hex encoding
* Prepend version number to encrypted path segments
* Remove redundant var alias
* Simplified returns
* wrap errors
* Buffered eestream EncodeReader
* Extracted fillBuffer() helper function
* Slow channels will be closed if there are still at least k fast channels
* Doc comment for maxBufferMemory
* Use timer more efficiently
* Timer initialization should be inside the for-loop
* Parallel copy of encoded data to reader buffer channels
* Transfer input read errors to output encoded readers
* minimum and optimum thresholds
* Use time.AfterFunc
* Simplify error handling in constructor
