storj/pkg/pointerdb/pdbclient/client.go

151 lines
3.9 KiB
Go
Raw Normal View History

Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
// Copyright (C) 2018 Storj Labs, Inc.
// See LICENSE for copying information.
package pdbclient
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
import (
"context"
"github.com/golang/protobuf/proto"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
monkit "gopkg.in/spacemonkeygo/monkit.v2"
p "storj.io/storj/pkg/paths"
"storj.io/storj/pkg/pb"
"storj.io/storj/pkg/provider"
"storj.io/storj/storage"
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
)
var (
mon = monkit.Package()
)
// PointerDB creates a grpcClient
type PointerDB struct {
grpcClient pb.PointerDBClient
Node Identity (#193) * peertls: don't log errors for double close understood that this part of the code is undergoing heavy change right now, but just want to make sure this fix gets incorporated somewhere * git cleanup: node-id stuff * cleanup * rename identity_util.go * wip `CertificateAuthority` refactor * refactoring * gitignore update * wip * Merge remote-tracking branch 'storj/doubleclose' into node-id3 * storj/doubleclose: peertls: don't log errors for double close * add peertls tests & gomports * wip: + refactor + style changes + cleanup + [wip] add version to CA and identity configs + [wip] heavy client setup * refactor * wip: + refactor + style changes + add `CAConfig.Load` + add `CAConfig.Save` * wip: + add `LoadOrCreate` and `Create` to CA and Identity configs + add overwrite to CA and identity configs + heavy client setup + refactor + style changes + cleanup * wip * fixing things * fixing things * wip hc setup * hc setup: + refactor + bugfixing * improvements based on reveiw feedback * goimports * improvements: + responding to review feedback + refactor * feedback-based improvements * feedback-based improvements * feedback-based improvements * feedback-based improvements * feedback-based improvements * feedback-based improvements * cleanup * refactoring CA and Identity structs * Merge branch 'master' into node-id3 * move version field to setup config structs for CA and identity * fix typo * responding to revieiw feedback * responding to revieiw feedback * responding to revieiw feedback * responding to revieiw feedback * responding to revieiw feedback * responding to revieiw feedback * Merge branch 'master' into node-id3 * fix gateway setup finally * go imports * fix `FullCertificateAuthority.GenerateIdentity` * cleanup overlay tests * bugfixing * update ca/identity setup * go imports * fix peertls test copy/paste fail * responding to review feedback * setup tweaking * update farmer setup
2018-08-13 09:39:45 +01:00
APIKey []byte
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
}
// a compiler trick to make sure *Overlay implements Client
var _ Client = (*PointerDB)(nil)
// ListItem is a single item in a listing
type ListItem struct {
Path p.Path
Pointer *pb.Pointer
IsPrefix bool
}
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
// Client services offerred for the interface
type Client interface {
Put(ctx context.Context, path p.Path, pointer *pb.Pointer) error
Get(ctx context.Context, path p.Path) (*pb.Pointer, error)
List(ctx context.Context, prefix, startAfter, endBefore p.Path,
recursive bool, limit int, metaFlags uint32) (
items []ListItem, more bool, err error)
Delete(ctx context.Context, path p.Path) error
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
}
// NewClient initializes a new pointerdb client
func NewClient(identity *provider.FullIdentity, address string, APIKey []byte) (*PointerDB, error) {
dialOpt, err := identity.DialOption()
if err != nil {
return nil, err
}
c, err := clientConnection(address, dialOpt)
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
if err != nil {
return nil, err
}
return &PointerDB{
grpcClient: c,
Node Identity (#193) * peertls: don't log errors for double close understood that this part of the code is undergoing heavy change right now, but just want to make sure this fix gets incorporated somewhere * git cleanup: node-id stuff * cleanup * rename identity_util.go * wip `CertificateAuthority` refactor * refactoring * gitignore update * wip * Merge remote-tracking branch 'storj/doubleclose' into node-id3 * storj/doubleclose: peertls: don't log errors for double close * add peertls tests & gomports * wip: + refactor + style changes + cleanup + [wip] add version to CA and identity configs + [wip] heavy client setup * refactor * wip: + refactor + style changes + add `CAConfig.Load` + add `CAConfig.Save` * wip: + add `LoadOrCreate` and `Create` to CA and Identity configs + add overwrite to CA and identity configs + heavy client setup + refactor + style changes + cleanup * wip * fixing things * fixing things * wip hc setup * hc setup: + refactor + bugfixing * improvements based on reveiw feedback * goimports * improvements: + responding to review feedback + refactor * feedback-based improvements * feedback-based improvements * feedback-based improvements * feedback-based improvements * feedback-based improvements * feedback-based improvements * cleanup * refactoring CA and Identity structs * Merge branch 'master' into node-id3 * move version field to setup config structs for CA and identity * fix typo * responding to revieiw feedback * responding to revieiw feedback * responding to revieiw feedback * responding to revieiw feedback * responding to revieiw feedback * responding to revieiw feedback * Merge branch 'master' into node-id3 * fix gateway setup finally * go imports * fix `FullCertificateAuthority.GenerateIdentity` * cleanup overlay tests * bugfixing * update ca/identity setup * go imports * fix peertls test copy/paste fail * responding to review feedback * setup tweaking * update farmer setup
2018-08-13 09:39:45 +01:00
APIKey: APIKey,
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
}, nil
}
captplanet (#159) * captplanet I kind of went overboard this weekend. The major goal of this changeset is to provide an environment for local development where all of the various services can be easily run together. Developing on Storj v3 should be as easy as running a setup command and a run command! To do this, this changeset introduces a new tool called captplanet, which combines the powers of the Overlay Cache, the PointerDB, the PieceStore, Kademlia, the Minio Gateway, etc. Running 40 farmers and a heavy client inside the same process forced a rethinking of the "services" that we had. To avoid confusion by reusing prior terms, this changeset introduces two new types: Providers and Responsibilities. I wanted to avoid as many merge conflicts as possible, so I left the existing Services and code for now, but if people like this route we can clean up the duplication. A Responsibility is a collection of gRPC methods and corresponding state. The following systems are examples of Responsibilities: * Kademlia * OverlayCache * PointerDB * StatDB * PieceStore * etc. A Provider is a collection of Responsibilities that share an Identity, such as: * The heavy client * The farmer * The gateway An Identity is a public/private key pair, a node id, etc. Farmers all need different Identities, so captplanet needs to support running multiple concurrent Providers with different Identities. Each Responsibility and Provider should allow for configuration of multiple copies on its own so creating Responsibilities and Providers use a new workflow. To make a Responsibility, one should create a "config" struct, such as: ``` type Config struct { RepairThreshold int `help:"If redundancy falls below this number of pieces, repair is triggered" default:"30"` SuccessThreshold int `help:"If redundancy is above this number then no additional uploads are needed" default:"40"` } ``` To use "config" structs, this changeset introduces another new library called 'cfgstruct', which allows for the configuration of arbitrary structs through flagsets, and thus through cobra and viper. cfgstruct relies on Go's "struct tags" feature to document help information and default values. Config structs can be configured via cfgstruct.Bind for binding the struct to a flagset. Because this configuration system makes setup and configuration easier *in general*, additional commands are provided that allow for easy standup of separate Providers. Please make sure to check out: * cmd/captplanet/farmer/main.go (a new farmer binary) * cmd/captplanet/hc/main.go (a new heavy client binary) * cmd/captplanet/gw/main.go (a new minio gateway binary) Usage: ``` $ go install -v storj.io/storj/cmd/captplanet $ captplanet setup $ captplanet run ``` Configuration is placed by default in `~/.storj/capt/` Other changes: * introduces new config structs for currently existing Responsibilities that conform to the new Responsibility interface. Please see the `pkg/*/config.go` files for examples. * integrates the PointerDB API key with other global configuration via flags, instead of through environment variables through viper like it's been doing. (ultimately this should also change to use the PointerDB config struct but this is an okay shortterm solution). * changes the Overlay cache to use a URL for database configuration instead of separate redis and bolt config settings. * stubs out some peer identity skeleton code (but not the meat). * Fixes the SegmentStore to use the overlay client and pointerdb clients instead of gRPC client code directly * Leaves a very clear spot where we need to tie the object to stream to segment store together. There's sort of a "golden spike" opportunity to connect all the train tracks together at the bottom of pkg/miniogw/config.go, labeled with a bunch of TODOs. Future stuff: * I now prefer this design over the original pkg/process.Service thing I had been pushing before (sorry!) * The experience of trying to have multiple farmers configurable concurrently led me to prefer config structs over global flags (I finally came around) or using viper directly. I think global flags are okay sometimes but in general going forward we should try and get all relevant config into config structs. * If you all like this direction, I think we can go delete my old Service interfaces and a bunch of flags and clean up a bunch of stuff. * If you don't like this direction, it's no sweat at all, and despite how much code there is here I'm not very tied to any of this! Considering a lot of this was written between midnight and 6 am, it might not be any good! * bind tests
2018-07-24 17:08:28 +01:00
// a compiler trick to make sure *PointerDB implements Client
var _ Client = (*PointerDB)(nil)
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
// ClientConnection makes a server connection
func clientConnection(serverAddr string, opts ...grpc.DialOption) (pb.PointerDBClient, error) {
conn, err := grpc.Dial(serverAddr, opts...)
if err != nil {
return nil, err
}
return pb.NewPointerDBClient(conn), nil
}
// Put is the interface to make a PUT request, needs Pointer and APIKey
func (pdb *PointerDB) Put(ctx context.Context, path p.Path, pointer *pb.Pointer) (err error) {
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
defer mon.Task()(&ctx)(&err)
_, err = pdb.grpcClient.Put(ctx, &pb.PutRequest{Path: path.String(), Pointer: pointer, APIKey: pdb.APIKey})
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
return err
}
// Get is the interface to make a GET request, needs PATH and APIKey
func (pdb *PointerDB) Get(ctx context.Context, path p.Path) (pointer *pb.Pointer, err error) {
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
defer mon.Task()(&ctx)(&err)
res, err := pdb.grpcClient.Get(ctx, &pb.GetRequest{Path: path.String(), APIKey: pdb.APIKey})
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
if err != nil {
if status.Code(err) == codes.NotFound {
return nil, storage.ErrKeyNotFound.Wrap(err)
}
return nil, Error.Wrap(err)
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
}
pointer = &pb.Pointer{}
err = proto.Unmarshal(res.GetPointer(), pointer)
if err != nil {
return nil, err
}
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
return pointer, nil
}
// List is the interface to make a LIST request, needs StartingPathKey, Limit, and APIKey
func (pdb *PointerDB) List(ctx context.Context, prefix, startAfter, endBefore p.Path,
recursive bool, limit int, metaFlags uint32) (
items []ListItem, more bool, err error) {
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
defer mon.Task()(&ctx)(&err)
res, err := pdb.grpcClient.List(ctx, &pb.ListRequest{
Prefix: prefix.String(),
StartAfter: startAfter.String(),
EndBefore: endBefore.String(),
Recursive: recursive,
Limit: int32(limit),
MetaFlags: metaFlags,
APIKey: pdb.APIKey,
})
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
if err != nil {
return nil, false, err
}
list := res.GetItems()
items = make([]ListItem, len(list))
for i, itm := range list {
items[i] = ListItem{
Path: p.New(itm.GetPath()),
Pointer: itm.GetPointer(),
IsPrefix: itm.IsPrefix,
}
}
return items, res.GetMore(), nil
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
}
// Delete is the interface to make a Delete request, needs Path and APIKey
func (pdb *PointerDB) Delete(ctx context.Context, path p.Path) (err error) {
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
defer mon.Task()(&ctx)(&err)
_, err = pdb.grpcClient.Delete(ctx, &pb.DeleteRequest{Path: path.String(), APIKey: pdb.APIKey})
Mutex/nsclient- WIP (#104) * working on put request for nsclient * working on put request for nsclient * netstate put * netstate put * wip testing client * wip - testing client and working through some errors * wip - testing client and working through some errors * put request works * put request works for client * get request working * get request working * get request working-minor edit * get request working-minor edit * list request works * list request works * working through delete error * working through delete error * fixed exp client, still working through delete error * fixed exp client, still working through delete error * delete works; fixed formatting issues * delete works; fixed formatting issues * deleted comment * deleted comment * resolving merge conflicts * resolving merge conflict * fixing merge conflict * implemented and modified kayloyans paths file * working on testing * added test for path_test.go * fixed string, read through netstate test * deleted env variables * initial commit for mocking out grpc client- got it working * mocked grpc client * mock put passed test * 2 tests pass for PUT with mock * put requests test pass, wip- want mini review * get tests pass mock * list test working * initial commit for list test * all list req. working, starting on delete tests * delete tests passed * cleaned up tests * resolved merge conflicts * resolved merge conflicts * fixed linter errors * fixed error found in travis * initial commit for fixes from PR comments * fixed pr comments and linting * added error handling for api creds, and rebased * fixes from dennis comments * fixed pr with dennis suggestioon * added copyrights to files * fixed casing per dennis great comment * fixed travis complaint on sprintf
2018-07-19 23:57:22 +01:00
return err
}