This is a naive implementation of the overlay worker.
Future improvements / to dos:
- Walk through the cache and remove nodes that don't respond
- Better look ups for new nodes
- Better random ID generation
- Kademlia hooks for automatically adding new nodes to the cache
* adding refresh cache functionality, added schedule function
* update put in db
* Tests passing
* wip overlay tests for cache refresh
* update scheduler code
* update refresh function
* WIP adding random lookups to refresh worker
* remove quit channel
* updates fire on schedule and the refresh function finds near nodes
* updates to refresh function, getting more buckets and nodes
* updates to refresh function and cache operations
* add cancellation to context, fix k number of nodes in lookups
* Unit test covarege increased for kademlia pkg
go style formatting added
Removed DHT param from newTestKademlia method, added comments for Bucket methods that informs that these tests will need to be updated
unnecessary comment deleted from newTestKademlia
Adjust Segment Store to the updated interface (#160)
* Adjust Segment Store to the updated interface
* Move /pkg/storage/segment to /pkg/storage/segments
* Fix overlay client tests
* Revert changes in NewOverlayClient return value
* Rename `rem` to `seg`
* Implement Meta()
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
Add files for testing builds in docker (#161)
* Add files for testing builds in docker
* Make tests check for redis running before trying to start redis-server, which may not exist.
* Clean redis server before any tests use it.
* Add more debugging for travis
* Explicitly requiring redis for travis
pkg/provider: with pkg/provider merged, make a single heavy client binary, gateway binary, and deprecate old services (#165)
* pkg/provider: with pkg/provider merged, make a single heavy client binary and deprecate old services
* add setup to gw binary too
* captplanet: output what addresses everything is listening on
* revert peertls/io_util changes
* define config flag across all commands
* use trimsuffix
fix docker makefile (#170)
* fix makefile
protos: update protobufs with go generate (#169)
the import for timestamp and duration should use
the path provided by a standard protocol buffer library
installation
Refactor List in PointerDB (#163)
* Refactor List in Pointer DB
* Fix pointerdb-client example
* Fix issue in Path type related to empty paths
* Test for the PointerDB service with some fixes
* Fixed debug message in example: trancated --> more
* GoDoc comments for unexported methods
* TODO comment to check if Put is overwriting
* Log warning if protobuf timestamp cannot be converted
* TODO comment to make ListPageLimit configurable
* Rename 'segment' package to 'segments' to reflect folder name
Minio integration with Object store (#156)
* initial WIP integration with Object store
* List WIP
* minio listobject function changes complete
* Code review changes and work in progress for the mock objectstore unit testing cases
* Warning fix redeclaration of err
* Warning fix redeclaration of err
* code review comments & unit testing inprogress
* fix compilation bug
* Fixed code review comments & added GetObject Mock test case
* rearraged the mock test file and gateway storj test file in to the proper directory
* added the missing file
* code clean up
* fix lint error on the mock generated code
* modified per code review comments
* added the PutObject mock test case
* added the GetObjectInfo mock test case
* added listobject mock test case
* fixed package from storj to miniogw
* resolved the gateway-storj.go initialization merge conflict
update readme (#174)
added assertion for unused errors (#152)
merging this PR to avoid future issues
updating github user to personal account (#171)
Test coverage ranger (#168)
* Fixed go panic for corner case
* Initial test coverage for ranger pkg
streamstore: add passthrough implementation (#176)
this doesn't implement streamstore, this just allows us to try and
get the june demo working again in the meantime
StatDB (#144)
* add statdb proto and example client
* server logic
* update readme
* remove boltdb from service.go
* sqlite3
* add statdb server executable file
* create statdb node table if it does not exist already
* get UpdateBatch working
* update based on jt review
* remove some commented lines
* fix linting issues
* reformat
* apiKey -> APIKey
* update statdb client apiKey->APIKey
Update README.md
Update README.md
overlay: correct dockerfile db (#179)
cmd/hc, cmd/gw, cmd/captplanet: simplify setup/run commands (#178)
also allows much more customization of services within captain planet,
such as reconfiguring the overlay service to use redis
pkg/process: don't require json formatting (#177)
Cleanup metadata across layers (#180)
* Cleanup metadata across layers
* Fix pointer db tests
Kademlia Routing Table (#164)
* adds comment
* runs deps
* creates boltdb kademlia routing table
* protobuf updates
* adds reverselist to mockkeyvaluestore interface
* xor wip
* xor wip
* fixes xor sort
* runs go fmt
* fixes
* goimports again
* trying to fix travis tests
* fixes mock tests
Ranger refactoring (#158)
* Fixed go panic for corner case
* Cosmetic changes, and small error fixes
miniogw: log all errors (#182)
* miniogw: log all errors
* tests added
* doc comment to satisfy linter
* fix test failure
Jennifer added to CLA list
* Temporary fix for storage/redis list method test
* begins adding inline segment support for segmentstore
* adds PeekThresholdReader struct plus Read and isInline methods
* moves PeekThresholdReader to peek.go, adds more simplified Read function
* adds PeekThresholdReader tests
* reverts Read function to earlier version, updates tests to use ReadFull instead
* Get function now handles inline type pointers
* adds correct type Size and ExpirationDate to inline segment
* fixes return value in Put func error condition
* moves thresholdBuf and Read tests into a table test
* adds border case test, fixes redundant parts
* passes sizedReader size to makeRemotePointer
* unit tests for service.go in overlay package added
* comments removed
* unit tests for overlay package updated
* updated code for starting redis server
* add statdb proto and example client
* server logic
* update readme
* remove boltdb from service.go
* sqlite3
* add statdb server executable file
* create statdb node table if it does not exist already
* get UpdateBatch working
* update based on jt review
* remove some commented lines
* fix linting issues
* reformat
* apiKey -> APIKey
* update statdb client apiKey->APIKey
* initial WIP integration with Object store
* List WIP
* minio listobject function changes complete
* Code review changes and work in progress for the mock objectstore unit testing cases
* Warning fix redeclaration of err
* Warning fix redeclaration of err
* code review comments & unit testing inprogress
* fix compilation bug
* Fixed code review comments & added GetObject Mock test case
* rearraged the mock test file and gateway storj test file in to the proper directory
* added the missing file
* code clean up
* fix lint error on the mock generated code
* modified per code review comments
* added the PutObject mock test case
* added the GetObjectInfo mock test case
* added listobject mock test case
* fixed package from storj to miniogw
* resolved the gateway-storj.go initialization merge conflict
* Refactor List in Pointer DB
* Fix pointerdb-client example
* Fix issue in Path type related to empty paths
* Test for the PointerDB service with some fixes
* Fixed debug message in example: trancated --> more
* GoDoc comments for unexported methods
* TODO comment to check if Put is overwriting
* Log warning if protobuf timestamp cannot be converted
* TODO comment to make ListPageLimit configurable
* Rename 'segment' package to 'segments' to reflect folder name
* pkg/provider: with pkg/provider merged, make a single heavy client binary and deprecate old services
* add setup to gw binary too
* captplanet: output what addresses everything is listening on
* revert peertls/io_util changes
* define config flag across all commands
* use trimsuffix
* Add files for testing builds in docker
* Make tests check for redis running before trying to start redis-server, which may not exist.
* Clean redis server before any tests use it.
* Add more debugging for travis
* Explicitly requiring redis for travis
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
* Adjust Segment Store to the updated interface
* Move /pkg/storage/segment to /pkg/storage/segments
* Fix overlay client tests
* Revert changes in NewOverlayClient return value
* Rename `rem` to `seg`
* Implement Meta()
* working on put request for nsclient
* working on put request for nsclient
* netstate put
* netstate put
* wip testing client
* wip - testing client
and working through some errors
* wip - testing client
and working through some errors
* put request works
* put request works for client
* get request working
* get request working
* get request working-minor edit
* get request working-minor edit
* list request works
* list request works
* working through delete error
* working through delete error
* fixed exp client, still working through delete error
* fixed exp client, still working through delete error
* delete works; fixed formatting issues
* delete works; fixed formatting issues
* deleted comment
* deleted comment
* resolving merge conflicts
* resolving merge conflict
* fixing merge conflict
* implemented and modified kayloyans paths file
* working on testing
* added test for path_test.go
* fixed string, read through netstate test
* deleted env variables
* initial commit for mocking out grpc client- got it working
* mocked grpc client
* mock put passed test
* 2 tests pass for PUT with mock
* put requests test pass, wip- want mini review
* get tests pass mock
* list test working
* initial commit for list test
* all list req. working, starting on delete tests
* delete tests passed
* cleaned up tests
* resolved merge conflicts
* resolved merge conflicts
* fixed linter errors
* fixed error found in travis
* initial commit for fixes from PR comments
* fixed pr comments and linting
* added error handling for api creds, and rebased
* fixes from dennis comments
* fixed pr with dennis suggestioon
* added copyrights to files
* fixed casing per dennis great comment
* fixed travis complaint on sprintf
* Add dockerfile and yaml for setting up piecestore servers
* Fix dockerfile for @aleitner (#115)
* Fix dockerfile for @aleitner
* Move files for @coyle
* Update yaml
* My linter had some errors so I resolved them
* Make jenkins do the needful
* Make piecestore-farmer look like overlay's build process
* Fix service spec to work in staging
* Make Jenkins push images, but not deploy them, yet.
* Modify entrypoint to fit new verbs
* Update piecestore-farmer entrypoint script to handle new app output
* Updates to config handling
- Add functions to load in configs
- Standardize location and naming of config files
- Configuration over convention style of config file handling for each
service
* update config handling and correclty handle cli flags being set
* generate configs from default if no config is found
- renamed pointerdbDB to pointerdb for clarity in config file
- set sane default for pkg/overlay boltDB file
- set srvPort to default to 8082 to avoid port collision on default
setting
* linter updates
* move boltdb path vars into function
* update tests to handle config environment changes
* --fix exec test mocks
* update tests to use viper instead of flag library
* fix typo
* add redis-server to services in travis for tests
* update examples with new config env function signature
* fix tests
