Commit Graph

5 Commits

Author SHA1 Message Date
Maximillian von Briesen
d934733c4d Move api key from pdb client method calls to pdb client constructor (#194)
* move api key from pdb client method calls to pdb client constructor

* update pdb example client
2018-08-02 13:09:50 -06:00
Kaloyan Raev
7e136db9cf
Cleanup metadata across layers (#180)
* Cleanup metadata across layers

* Fix pointer db tests
2018-07-30 21:57:50 +03:00
Kaloyan Raev
daf391e473
Refactor List in PointerDB (#163)
* Refactor List in Pointer DB

* Fix pointerdb-client example

* Fix issue in Path type related to empty paths

* Test for the PointerDB service with some fixes

* Fixed debug message in example: trancated --> more

* GoDoc comments for unexported methods

* TODO comment to check if Put is overwriting

* Log warning if protobuf timestamp cannot be converted

* TODO comment to make ListPageLimit configurable

* Rename 'segment' package to 'segments' to reflect folder name
2018-07-27 09:02:59 +03:00
JT Olio
5f6607935b
captplanet (#159)
* captplanet

I kind of went overboard this weekend.

The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!

To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.

Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.

A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
 * Kademlia
 * OverlayCache
 * PointerDB
 * StatDB
 * PieceStore
 * etc.

A Provider is a collection of Responsibilities that
share an Identity, such as:
 * The heavy client
 * The farmer
 * The gateway

An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.

Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.

To make a Responsibility, one should create a "config"
struct, such as:

```
type Config struct {
  RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
  SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```

To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.

cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.

Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
 * cmd/captplanet/farmer/main.go (a new farmer binary)
 * cmd/captplanet/hc/main.go (a new heavy client binary)
 * cmd/captplanet/gw/main.go (a new minio gateway binary)

Usage:

```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```

Configuration is placed by default in `~/.storj/capt/`

Other changes:

 * introduces new config structs for currently existing
   Responsibilities that conform to the new Responsibility
   interface. Please see the `pkg/*/config.go` files for
   examples.

 * integrates the PointerDB API key with other global
   configuration via flags, instead of through environment
   variables through viper like it's been doing. (ultimately
   this should also change to use the PointerDB config
   struct but this is an okay shortterm solution).

 * changes the Overlay cache to use a URL for database
   configuration instead of separate redis and bolt config
   settings.

 * stubs out some peer identity skeleton code (but not the
   meat).

 * Fixes the SegmentStore to use the overlay client and
   pointerdb clients instead of gRPC client code directly

 * Leaves a very clear spot where we need to tie the object to
   stream to segment store together. There's sort of a "golden
   spike" opportunity to connect all the train tracks together
   at the bottom of pkg/miniogw/config.go, labeled with a
   bunch of TODOs.

Future stuff:

 * I now prefer this design over the original
   pkg/process.Service thing I had been pushing before (sorry!)

 * The experience of trying to have multiple farmers
   configurable concurrently led me to prefer config structs
   over global flags (I finally came around) or using viper
   directly. I think global flags are okay sometimes but in
   general going forward we should try and get all relevant
   config into config structs.

 * If you all like this direction, I think we can go delete my
   old Service interfaces and a bunch of flags and clean up a
   bunch of stuff.

 * If you don't like this direction, it's no sweat at all, and
   despite how much code there is here I'm not very tied to any
   of this! Considering a lot of this was written between midnight
   and 6 am, it might not be any good!

* bind tests
2018-07-24 10:08:28 -06:00
nfarah86
e228d090e8
Mutex/nsclient- WIP (#104)
* working on put request for nsclient

* working on put request for nsclient

* netstate put

* netstate put

* wip testing client

* wip - testing client
 and working through some errors

* wip - testing client
 and working through some errors

* put request works

* put request works for client

* get request working

* get request working

* get request working-minor edit

* get request working-minor edit

* list request works

* list request works

* working through delete error

* working through delete error

* fixed exp client, still working through delete error

* fixed exp client, still working through delete error

* delete works; fixed formatting issues

* delete works; fixed formatting issues

* deleted comment

* deleted comment

* resolving merge conflicts

* resolving merge conflict

* fixing merge conflict

* implemented and modified kayloyans paths file

* working on testing

* added test for path_test.go

* fixed string, read through netstate test

* deleted env variables

* initial commit for mocking out grpc client- got it working

* mocked grpc client

* mock put passed test

* 2 tests pass for PUT with mock

* put requests test pass, wip- want mini review

* get tests pass mock

* list test working

* initial commit for list test

* all list req. working, starting on delete tests

* delete tests passed

* cleaned up tests

* resolved merge conflicts

* resolved merge conflicts

* fixed linter errors

* fixed error found in travis

* initial commit for fixes from PR comments

* fixed pr comments and linting

* added error handling for api creds, and rebased

* fixes from dennis comments

* fixed pr with dennis suggestioon

* added copyrights to files

* fixed casing per dennis great comment

* fixed travis complaint on sprintf
2018-07-19 15:57:22 -07:00