11e12bff5f
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JT Olio
|
5f6607935b
|
captplanet (#159)
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
|
||
Kaloyan Raev
|
445044b856 |
Adjust Segment Store to the updated interface (#160)
* Adjust Segment Store to the updated interface * Move /pkg/storage/segment to /pkg/storage/segments * Fix overlay client tests * Revert changes in NewOverlayClient return value * Rename `rem` to `seg` * Implement Meta() |
||
Bryan White
|
5b913c45b9
|
Transport security (#63)
* wip initial transport security * wip: transport security (add tests / refactor) * wip tests * refactoring - still wip * refactor, improve tests * wip tls testing * fix typo * wip testing * wip testing * wip * tls_test passing * code-style improvemente / refactor; service and tls tests passing! * code-style auto-format * add TestNewServer_LoadTLS * refactor; test improvements * refactor * add client cert * port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * files created * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * wip * add separate `Process` tests for bolt and redis-backed overlay * more testing * fix gitignore * fix linter error * goimports goimports GOIMPORTS GoImPortS!!!! * wip * fix port madness * forgot to add * add `mux` as handler and shorten context timeouts * gofreakingimports * fix comments * refactor test & add logger/monkit registry * debugging travis * add comment * Set redisAddress to empty string for bolt-test * Merge remote-tracking branch 'upstream/master' into coyle/kad-tests * Merge branch 'tls' into tls-upstream * tls: add client cert refactor refactor; test improvements add TestNewServer_LoadTLS code-style auto-format code-style improvemente / refactor; service and tls tests passing! tls_test passing wip wip testing wip testing fix typo wip tls testing refactor, improve tests refactoring - still wip wip tests wip: transport security (add tests / refactor) wip initial transport security * fixing linter things * wip * remove bkad dependencie from tests * wip * wip * wip * wip * wip * updated coyle/kademlia * wip * cleanup * ports * overlay upgraded * linter fixes * piecestore kademlia newID * Merge branch 'master' into tls-upstream * master: Add error to the return values of Ranger.Range method (#90) udp-forwarding: demo week work! (#84) * Merge branch 'kad-tests' into tls-upstream * kad-tests: piecestore kademlia newID linter fixes overlay upgraded ports cleanup wip updated coyle/kademlia wip wip wip wip wip remove bkad dependencie from tests wip wip files created port changes * wip * finish merging service tests * add test for different client/server certs * wip * Merge branch 'master' into tls-upstream * master: Add context to Ranger.Range method (#99) Coyle/kad client (#91) * wip * wip; refactoring/cleanup * wip * Merge branch 'master' into tls * master: Bolt backed overlay cache (#94) internal/test: switch errors to error classes (#96) * wip - test passing * cleanup * remove port.go * cleanup * Merge branch 'master' into tls * master: hardcode version (#111) Coyle/docker fix (#109) pkg/kademlia tests and restructuring (#97) Use continue instead of return in table tests (#106) prepend storjlabs to docker tag (#108) Automatically build, tag and push docker images on merge to master (#103) * more belated merging * more belated merging * more belated merging * add copyrights * cleanup * goimports * refactoring * wip * wip * implement `TLSFileOptions#loadTLS`, refactoring: `peertls.TestNewClient_LoadTLS` is the failing holdout; Still trying to figure out why I'm getting ECDSA verification is failing. * not sure if actually working: Tests are now passing (no more "ECDSA verification failed"); however, `len(*tls.Certificates.Certificate) == 1` which I don't think should be the case if the root and leaf are being created correctly. * Experimenting/bugfixing?: I think leaf certs should be properly signed by the parent now but not entirely certain. It's also unclear to me why in `VerifyPeerCertificate`, `len(rawCerts) == 1` when the certs should contain both the root and leaf afaik. * Properly write/read certificate chain (root/leaf): I think I'm now properly reading and writing the root and leaf certificate chain such that they're both being received by `VerifyPeerCertificate`. The next step is to parse the certificates with `x509.ParseCertificate` (or similar) and verify that the public keys and signatures match. * Add tls certificate chain signature veification (spike): + `VerifyPeerCertificate` verifies signatures of certificates using the key of it's parent if there is one; otherwise, it verifies the certificate is self-signed + TODO: refactor + TODO: test * refactoring `VerifyPeerCertificate` * cleanup * refactor * Merge branch 'master' into tls * master: Remove some structural folders we don't seem to be using. (#125) license code with agplv3 (#126) Update .clabot (#124) added team memebers (#123) clabot file added (#121) ECClient (#110) docker image issue fixed (#118) Piecestore Farmer CLI (#92) Define Path type (#101) adds netstate pagination (#95) Transport Client (#89) Implement psclient interface (#107) pkg/process: start replacing pkg/process with cobra helpers (#98) protos/netstate: remove stuff we're not using (#100) adding coveralls / code coverage (#112) * responding to review feedback / cleanup / add copywrite headers * suggestions * realitive * Merge pull request #1 from coyle/coyle/tls suggestions * remove unnecessary `_`s * Merge branch 'tls' of github.com:bryanchriswhite/storj into tls * 'tls' of github.com:bryanchriswhite/storj: realitive suggestions * Responding to review feedback: + refactor `VerifyPeerCertificate` * remove tls expiration * remove "hosts" and "clien option" from tls options * goimports * linter fixes |
||
Dennis Coyle
|
9e40141cbc
|
Coyle/kad client (#91)
* port changes * Merge remote-tracking branch 'upstream/master' * Merge remote-tracking branch 'upstream/master' * Restructure + additional interface * Add NewOverlayClient * added test for interface * PR comments addressed * lint issue * added generated protobuf * fixed merge conflicts * adding comments * PR comments addressed |