0cd58ec770
58 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
aligeti
|
ff52a1e7e1
|
Minio integration with Object store (#156)
* initial WIP integration with Object store * List WIP * minio listobject function changes complete * Code review changes and work in progress for the mock objectstore unit testing cases * Warning fix redeclaration of err * Warning fix redeclaration of err * code review comments & unit testing inprogress * fix compilation bug * Fixed code review comments & added GetObject Mock test case * rearraged the mock test file and gateway storj test file in to the proper directory * added the missing file * code clean up * fix lint error on the mock generated code * modified per code review comments * added the PutObject mock test case * added the GetObjectInfo mock test case * added listobject mock test case * fixed package from storj to miniogw * resolved the gateway-storj.go initialization merge conflict |
||
JT Olio
|
45a3c2e974
|
pkg/provider: with pkg/provider merged, make a single heavy client binary, gateway binary, and deprecate old services (#165)
* pkg/provider: with pkg/provider merged, make a single heavy client binary and deprecate old services * add setup to gw binary too * captplanet: output what addresses everything is listening on * revert peertls/io_util changes * define config flag across all commands * use trimsuffix |
||
|
JT Olio
|
5f6607935b
|
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
|
||
Cameron
|
6463b87ebe
|
Errcheck (#133)
* add errcheck * fixed linter errors * fixes * errcheck fixes in pkg/paths * Fix errchecks in PieceID.Derive * Fix ecclient tests * Move closeConn a little bit above in the exectution flow * fix new lint errors * Fatalf -> Printf * address eclipsed errors * rename err to closeErr * rename err to closeErr for differentiation |
||
|
aligeti
|
8d8350fea7 |
Implement miniogw in terms of ObjectStore (#117)
* port changes * Task monitor and setup merge from the staging * Restructure + additional interface * Add NewOverlayClient * integrated DHT client interface * added test for interface * PR comments addressed * lint issue * added generated protobuf * adding new interface * added the interface framework * deleted file * fixes compilation errors and integrates new dhtcclient interface * merged netstat latest changes and dht new interface chagnes * fixed the address's port * adding comments * PR comments addressed * netclient interface dial method added * rename and integrated transportclient with minio gateway * rename and code clean up * made changes based on the Dennis's changes on the kad-client * Code review comment changes based on kaloyan review comments * reverted the changes to be similar to master * removed unused file * renamed to transportclient * added the review changes * store the address of the client * updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters * updated the test case to test the bad address passed condition * updated the code per code review comments * Bolt backed overlay cache (#94) * wip * add separate `Process` tests for bolt and redis-backed overlay * more testing * fix gitignore * fix linter error * goimports goimports GOIMPORTS GoImPortS!!!! * fix port madness * forgot to add * add `mux` as handler and shorten context timeouts * gofreakingimports * fix comments * refactor test & add logger/monkit registry * debugging travis * add comment * Set redisAddress to empty string for bolt-test * travis experiment * refactoring tests * Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache * Automatically build, tag and push docker images on merge to master (#103) * port changes * build overlay on successful merge to master * fixes to Makefile * permissions * dep ensure * gopath * let's try vgo * remove dep * maybe alpine is the issue * tagging go version on build * stupid vgo * vgo * adding tags to push * quotes * local linting fixes & stupid travis * prepend storjlabs to docker tag (#108) * port changes * fixing tag name * Use continue instead of return in table tests (#106) I did a dumb mistake for some of the table tests, which made some of the test cases not being executed. * pkg/kademlia tests and restructuring (#97) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * files created * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * remove bkad dependencie from tests * wip * wip * wip * wip * wip * updated coyle/kademlia * wip * cleanup * ports * overlay upgraded * linter fixes * piecestore kademlia newID * add changes from kad demo * PR comments addresses * go func * force travis build * fixed merge conflicts * fixed merge conflicts * Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests * linter issues * linting issues * fixed merge conflicts * linter is stupid * Coyle/docker fix (#109) * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge branch 'master' of https://github.com/storj/storj * fixing tag name * no idea * testing * changes * testing on travis * testing * changes to travis build * new approach * Merge branch 'master' into coyle/docker-fix * hardcode version (#111) * hardcode version * adding coveralls / code coverage (#112) * adding coveralls * adding code coverage badge * fixing badges * verbose * swap tests and coverage * extra line * maybe * maybe * moar * gover maybe * testing * cleanup * protos/netstate: remove stuff we're not using (#100) * protos/netstate: remove stuff we're not using * protos/netstate: add metadata field for segmentstore * fix netstate client test * pkg/process: start replacing pkg/process with cobra helpers (#98) * Implement psclient interface (#107) * Implement psclient interface * Add string method to pieceID type * try to fix linter errors * Whoops missed an error * More linter errors * Typo * Lol double typo * Get everything working, begin adding tests for psclient rpc * goimports * Forgot to change the piecestore cli when changed the piecestore code * Fix CLI * remove ID length, added validator to pieceID * Move grpc ranger to client Change client PUT api to take a reader rather than return a writer * GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser * Forgot to remove offset * Added message upon successful store * Do that thing dennis and kaloyan wanted * goimports * Make closeConn a part of the interface for psclient * Use interface * Removed uneccessary new lines * goimport * Whoops * Actually we don't want to use the interface in Piece Ranger * Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples * Make comments look nicer * modified transport client based on the the design discussion * modified transport client based on the the design discussion * added the as discussed connection cache interface functionality * added the as discussed connection cache interface functionality * transport client changes * transport client per code review changes * per the code review comments * transport client incorporates review comments * fixes lint warnings * lint warning fixes....client interface has to be Client * initial draft of Objectstore * transport client review changes * client.go changes * transport.go changes * added test case * added test cases * streams iface * comment fix * object store changes * comment fix * initialized the objectstore in gw * Added PutObject with test support for encryption file * added object store test cases * tested & integrated the objectstore with miniogw * handled the ranger and paths * indentation change * Kalyon's code review comments resolution implemented after the 30min code review meeting * Compilation error fix * fixes the tavis build warnings * corrects the ListObject return type to be slice of slice * corrects the ListObject return type to be slice of slice * added the serialization using protobuf * added the unmarshalling of data in getobject() * Jt's Review comments * Kaloyan's review comments, moved the unmarshalling logic and other minor code indentation fixes * more code reivew * more code reivew * Changes the expiration time to zeroTime and added error check in putObject function * Changes the expiration time to zeroTime and added error check in putObject function * minor warning fix- had to add a comment and fix the wording * added a TODO comment per kaloyan * code clean up removed unused variables |
||
Kaloyan Raev
|
d8f1ec1db6
|
ECClient (#110)
* WIP ECClient * Get returns RangeCloser * Introduce RedundancyStrategy * Constructor takes max buffer memory * Remove unnecessary NopCloser wrapper * Added telemetry * Tests * Adapt to PSClient from master * Decode should report error if empty rrs map is passed * collectErrors helper * Move to /pkg/storage * Move to /pkg/storage/ec * Rename ecclient.go to client.go * Better logging * Rename ec.ECClient to ec.Client * Fix some test execution * Adopt Transport Client from master |
||
|
aligeti
|
fa65f449ed
|
light client upload file support -V3 gateway (#57)
* integrated eestream's serve-pieces functionality * added ref to http request function * created dummy bucket list * Initialized the buckets with files with hardcoded sample data * supports upload Object(s) * uploads to corresponding folders * code cleanup for review * updated based on code review comments * updates based on missed code review comments * updated with review comments * implemented review comments * merged latest and tested * added filepath.Join() * updates based on the comments * fixes the eestreamer parameter due to merge |
||
|
JT Olds
|
fc7cfde9f7 | pkg/miniogw: a minio-based s3 gateway for storj. just stubs right now (#34) |