* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
* working on put request for nsclient
* working on put request for nsclient
* netstate put
* netstate put
* wip testing client
* wip - testing client
and working through some errors
* wip - testing client
and working through some errors
* put request works
* put request works for client
* get request working
* get request working
* get request working-minor edit
* get request working-minor edit
* list request works
* list request works
* working through delete error
* working through delete error
* fixed exp client, still working through delete error
* fixed exp client, still working through delete error
* delete works; fixed formatting issues
* delete works; fixed formatting issues
* deleted comment
* deleted comment
* resolving merge conflicts
* resolving merge conflict
* fixing merge conflict
* implemented and modified kayloyans paths file
* working on testing
* added test for path_test.go
* fixed string, read through netstate test
* deleted env variables
* initial commit for mocking out grpc client- got it working
* mocked grpc client
* mock put passed test
* 2 tests pass for PUT with mock
* put requests test pass, wip- want mini review
* get tests pass mock
* list test working
* initial commit for list test
* all list req. working, starting on delete tests
* delete tests passed
* cleaned up tests
* resolved merge conflicts
* resolved merge conflicts
* fixed linter errors
* fixed error found in travis
* initial commit for fixes from PR comments
* fixed pr comments and linting
* added error handling for api creds, and rebased
* fixes from dennis comments
* fixed pr with dennis suggestioon
* added copyrights to files
* fixed casing per dennis great comment
* fixed travis complaint on sprintf
* Updates to config handling
- Add functions to load in configs
- Standardize location and naming of config files
- Configuration over convention style of config file handling for each
service
* update config handling and correclty handle cli flags being set
* generate configs from default if no config is found
- renamed pointerdbDB to pointerdb for clarity in config file
- set sane default for pkg/overlay boltDB file
- set srvPort to default to 8082 to avoid port collision on default
setting
* linter updates
* move boltdb path vars into function
* update tests to handle config environment changes
* --fix exec test mocks
* update tests to use viper instead of flag library
* fix typo
* add redis-server to services in travis for tests
* update examples with new config env function signature
* fix tests
* WIP creating admin node service
- WIP changing the process pkg to accept multiple services
- WIP looping over services passed to process
- add netstate/service.go file and abstract it for service processing
* implement goroutine to launch each process
* goroutines working with multiple services
* code review fixes
* more code updates for review
* Add pkg lock and mod files back in
* code review updates
* update process.Main with better concurrent error handling
* Update error handling and pass ctx to StartService
* Update error handling with channel implementation
* Merge in upstream changes
- Simplify error handling channels
* updates
* Updates per reviewable
* fix test
* Setup test exec
* Scaffold test setup
* process main test working
* update admin process test
* Test multiple processes done
* Add error classes for testing, test main logger error
* Updates to tests
* Update how process.Main() handles configs
* Complete merge
* Update Gopkg and add Copyright
* Fix cyclical import issue
- Added .coverprofile to gitignore
- Update admin main.go function call
* remove unnecessary line
* Updates
* DRY up cmd/netstate package
* update service function calls
* updates
* Trying no-ops in examples
* rename netstate to pointerdb
* trying to fix merge
* dep ensure and run tests
* remove flag.Parse
* Update deps
* Skip offending test in pkg/process, to be fixed later
* adds netstate rpc server pagination, mocks pagination in test/util.go
* updates ns client example, combines ns client and server test to netstate_test, adds pagination to bolt client
* better organizes netstate test calls
* wip breaking netstate test into smaller tests
* wip modularizing netstate tests
* adds some test panics
* wip netstate test attempts
* testing bug in netstate TestDeleteAuth
* wip fixes global variable problem, still issues with list
* wip fixes get request params and args
* fixes bug in path when using MakePointers helper fn
* updates mockdb list func, adds test, changes Limit to int
* fixes merge conflicts
* fixes broken tests from merge
* remove unnecessary PointerEntry struct
* removes error when Get returns nil value from boltdb
* breaks boltdb client tests into smaller tests
* renames AssertNoErr test helper to HandleErr
* adds StartingKey and Limit parameters to redis list func, adds beginning of redis tests
* adds helper func for mockdb List function
* if no starting key provided for netstate List, the first value in storage will be used
* adds basic pagination for redis List function, adds tests
* adds list limit to call in overlay/server.go
* streamlines/fixes some nits from review
* removes use of obsolete EncryptedUnencryptedSize
* uses MockKeyValueStore instead of redis instance in redis client test
* changes test to expect nil returned for getting missing key
* remove error from `KeyValueStore#Get`
* fix bolt test
* Merge pull request #1 from bryanchriswhite/nat-pagination
remove error from `KeyValueStore#Get`
* adds Get returning error back to KeyValueStore interface and affected clients
* trying to appease travis: returns errors in Get calls in overlay/cache and cache_test
* handles redis get error when no key found
* Implement psclient interface
* Add string method to pieceID type
* try to fix linter errors
* Whoops missed an error
* More linter errors
* Typo
* Lol double typo
* Get everything working, begin adding tests for psclient rpc
* goimports
* Forgot to change the piecestore cli when changed the piecestore code
* Fix CLI
* remove ID length, added validator to pieceID
* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer
* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser
* Forgot to remove offset
* Added message upon successful store
* Do that thing dennis and kaloyan wanted
* goimports
* Make closeConn a part of the interface for psclient
* Use interface
* Removed uneccessary new lines
* goimport
* Whoops
* Actually we don't want to use the interface in Piece Ranger
* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples
* Make comments look nicer
* added spawn scripts
* wip
* Make server not require length
* Added test for 0 length
* Added test for accepting length of 0
* linter errors
* Make store not take anything more than an id and maybe a ttl if you want
* initial commit for PUT request authorization
* inital auth for put request
* auth. request working for all req
* deleted library
* removed .db files
* work in progress for modifying test suite to accomodate credentials
* modified tests
* gofmt, fixed code based on suggestions; test passed
* gofmt again
* merged nat's update on pointers, passed tests, cleanup from git rebase
* fixed fmt
* fixed fmt on tests
* work in progress
* reduced code
* fixed naming conventions
* added line in code
* fixed server bug, merged new code to server, added env
* fixed linter; getting cright issues on piecestore
* added comments for what passes on the creds
* added spawn scripts
* Determine random id for storing
* Moved determine id to rpc example
* Added tests
* Better test
* goimports
* Updated tests
* Fix typos
* Added piecestore
* gofmt
* Added requested changes
* Added cli
* Removed ranger because I wanted something that can stand alone
* Add example of http server using piece store
* Changed piecestore code to make it more optial for error handelling
* Merged with piecestore
* Added missing package
* Forgot io import
* gofmt
* gofmt
* Forgot io
* Make path by hash exported
* updated to simplify again whoops
* Updated server to work real good
* Forgot ampersand
* Updated to match FilePiece
* Merged in cam's delete code
* Remove unused io
* Added RPC code
* Give the download request a reader
* Removed http server stuff; changed receive stream to say io.reader
* Added expiration date to shardInfo
* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions
* added ttl info request
* Move scripts to http server pr; added close method for Retrieve api
* added rpc server tests for getting piece meta data and retrieval routes
* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf
* Imported cams test
* Bump gometalinter deadline
* WIP adding tests
* added tests for store and delete routes
* Add changes as requested by Kaloyan, also cleaned up some code
* Get the code actually working whoops
* More cleanup
* Separating database calls from api.go
* need to rename expiration
* Added some changes requested by JT
* Fix read size
* Fixed total amount to read
* added tests
* Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages
* Moved rpc client and server to piece store
* Moved piecestore protobuf to the /protos folder
* Cleaned up messages
* Clean up linter errors
* Added missing sqlite import
* Add ability to do iterative reads and writes to pstore
* Incrementally read data
* Fix linter and import errors
* Solve linter Error
* Change return types
* begin test refactor
* refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go
* Cleaning up tests
* Added retrieve tests
* refactored delete tests
* Deleted old tests
* Updated cmd/piecestore to reflect changes to piecestore
* Refactored server tests and server/client store code
* gofmt
* WIP implementing TTL struct
* Read 4k at a time when Retrieving
* implemented ttl struct
* Accidentally removed fpiece dependency?
* Double resolve merge conflict
* Moved client to the examples since it is an example
* Change hash to id in protobuf. TODO: Change client and server to reflect these changes
* Update client and protobuf
* changed hash to id
* Handle eof properly in retrieve
* Id -> ID
* forgot to change import for client after moving
* Moved client and server main to examples
* Made changes requested by JT
* checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close()
* Print not fatal
* Handle overflow when reading from server
* added const IDLength
* removed types from comments
* Add reader/writer for download data from and uploading data to server
* goimports and comments
* fixed nits, casts, added OK const, DBCleanup now exits program on error
* Add stream reader and writer for server
* Fix errors
* i beforee except after c lol
* customizable data dir
* Forgot to rename variable
* customizable data dir
* Handle closing of server stream properly
* linter
* pass on inserting the same data twice
* linter
* linter
* Do the easy things JT asked for
* Handle overflow from reads properly; handle custom db path
* Handle overflow for server stream read; TODO Combine server and client stream reads
* Allow for TTL of 0 to stay forever
* Make Client cleaner for user
* Path encryption library
* Use base64 instead of hex encoding
* Prepend version number to encrypted path segments
* Remove redundant var alias
* Simplified returns
* wrap errors
* Buffered eestream EncodeReader
* Extracted fillBuffer() helper function
* Slow channels will be closed if there are still at least k fast channels
* Doc comment for maxBufferMemory
* Use timer more efficiently
* Timer initialization should be inside the for-loop
* Parallel copy of encoded data to reader buffer channels
* Transfer input read errors to output encoded readers
* minimum and optimum thresholds
* Use time.AfterFunc
* Simplify error handling in constructor
* adds pointer to netstate proto file
* generated updated netstate proto
* changes boltdb netstate to save pointers as values
* updates netstate Put to save Pointers, updates client example to put a pointer, adds grpc status errors, updates tests, changes boltdb 'File' struct to 'PointerEntry'
* updates netstate client example and client test to save pointers, updates netstate List and Delete
* begins adding netstate-http tests
* removes netstate http service
* re-adds netstate auth
* updates boltdb netstate test
* changes encrypted_unencrypted_size from int64 to bytes in netstate proto
* updates READMEs
* initial commit
* created httpreq struct
* checked against env key
* wrapped up comparing xapikey to flag input
* added readme
* added example and updated code with viper
* added ConstantTimeCompare function
* fixed spacing
* updated with comments, and updated readme
* updated main.go with setting env variable
* fixed linter errors
* AES GCM implementation and unit test code
* modified and tested per the code review comments
* modified and tested per the code review comments
* updated to return err
* removed the debug printf commented code
* support of aes-gcm
* updated the renaming convention per GOLANG coding standards
* Initial Go & C biniding with libstorj
* Initial GO & C bindings with libstorj
* fixing the callback
* moved a millimeter :-) on c to go to gone....
* added error condition, per review comment
* removed the .idea directory and also movie.avi file
* deleting files not to be in this pull request
* Revert "deleting files not to be in this pull request"
This reverts commit 026b834fe00f6b20a7566e71973fe224c12f533f.
* deleting files not to be in this pull request
* resolving conflicts
* syncing the file to master
* Use aes gcm in eestream rs tests
* Use aes gcm in serve example
* Fixed comment
* adds proto files for netstate crud
* moves netstate grpc client lib into pkg/netstate where grpc netstate service is defined
* starts adding grpc client and server tests
* moves creation of grpc server into cmd/netstate/main.go, removes pkg/netstate/service.go, adds more client testing
* changed all 'Path' and 'Value' fields from strings to bytes, updated tests
* changes Get and Delete in proto file to receive 'requests' instead of 'file paths', adds tests for Get, List, and Delete
* changes netstate-routes to get 'fileValue' bytes not 'fileInfo'
* adds example rpc client in 'examples' and adds more specific debug logs
* adds readmes for netstate rpc services and updates netstate-routes
* WIP eestream avoid waiting for slow pieces
* Use non-blocking select to harvest all available inbufs
* Better way to check for elapsed time in tests
* Improve comment
* Determine EOF based on expected size
* Remove unused readerError type
* Configurable readers channel size
* Close properly decodedReader in tests
* Use context for properly closing the decodedReader
* Handle infectious errors using the new descendent classes
* Refactor decodedReader.Read() into helper functions
* Reenable TestRSErrors
* Test with Rangers
* Remove obsolete comment
* HTTPRanger
* Migrate eestream library to use ReadCloser instead of Reader
* Use MultiReadCloser instead of MultiReader
* Adapt example cli cmds to latest serverbox changes
* Close LazyReadCloser only if only generated
* Close ReadClosers sequentially
* Close response body on unexpected status code
* Avoid double close of the pipe reader
* Better formatting for imports
* Exit instead of panicking
