Commit Graph

16 Commits

Author SHA1 Message Date
Natalie Villasana
80727ae90b adds netstate pagination (#95)
* adds netstate rpc server pagination, mocks pagination in test/util.go

* updates ns client example, combines ns client and server test to netstate_test, adds pagination to bolt client

* better organizes netstate test calls

* wip breaking netstate test into smaller tests

* wip modularizing netstate tests

* adds some test panics

* wip netstate test attempts

* testing bug in netstate TestDeleteAuth

* wip fixes global variable problem, still issues with list

* wip fixes get request params and args

* fixes bug in path when using MakePointers helper fn

* updates mockdb list func, adds test, changes Limit to int

* fixes merge conflicts

* fixes broken tests from merge

* remove unnecessary PointerEntry struct

* removes error when Get returns nil value from boltdb

* breaks boltdb client tests into smaller tests

* renames AssertNoErr test helper to HandleErr

* adds StartingKey and Limit parameters to redis list func, adds beginning of redis tests

* adds helper func for mockdb List function

* if no starting key provided for netstate List, the first value in storage will be used

* adds basic pagination for redis List function, adds tests

* adds list limit to call in overlay/server.go

* streamlines/fixes some nits from review

* removes use of obsolete EncryptedUnencryptedSize

* uses MockKeyValueStore instead of redis instance in redis client test

* changes test to expect nil returned for getting missing key

* remove error from `KeyValueStore#Get`

* fix bolt test

* Merge pull request #1 from bryanchriswhite/nat-pagination

remove error from `KeyValueStore#Get`

* adds Get returning error back to KeyValueStore interface and affected clients

* trying to appease travis: returns errors in Get calls in overlay/cache and cache_test

* handles redis get error when no key found
2018-06-29 16:06:25 -04:00
JT Olio
613d1c25e2 pkg/process: start replacing pkg/process with cobra helpers (#98) 2018-06-27 11:02:49 +03:00
Dennis Coyle
a5ff5016c9
pkg/kademlia tests and restructuring (#97)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* add changes from kad demo

* PR comments addresses

* go func

* force travis build

* fixed merge conflicts

* fixed merge conflicts

* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests

* linter issues

* linting issues

* fixed merge conflicts

* linter is stupid
2018-06-22 09:33:57 -04:00
Bryan White
c6f8144221 Bolt backed overlay cache (#94)
* wip

* add separate `Process` tests for bolt and redis-backed overlay

* more testing

* fix gitignore

* fix linter error

* goimports goimports GOIMPORTS GoImPortS!!!!

* fix port madness

* forgot to add

* add `mux` as handler and shorten context timeouts

* gofreakingimports

* fix comments

* refactor test & add logger/monkit registry

* debugging travis

* add comment

* Set redisAddress to empty string for bolt-test

* travis experiment

* refactoring tests

* Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache
2018-06-20 10:28:46 -04:00
JT Olio
5ee6b76d41 internal/test: switch errors to error classes (#96)
* internal/test: switch errors to error classes

if you construct an error directly at package init time, you
won't get useful stack traces or anything. zeebo/errs expects that
you always construct an error (.New) when the error actually
happens. instead, you call Class at init time, then use (Class).Has
to test for error type membership.

* fix linter

* fix test
2018-06-19 20:03:46 +02:00
Dennis Coyle
9e40141cbc
Coyle/kad client (#91)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Restructure + additional interface

* Add NewOverlayClient

* added test for interface

* PR comments addressed

* lint issue

* added generated protobuf

* fixed merge conflicts

* adding comments

* PR comments addressed
2018-06-19 10:00:15 -04:00
Bryan White
1ebd66d880 Cleanup & bolt-backed cache (#87)
* wip post-demos cleanup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* Reorganize:

storage
├── common
│   └── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common
│   ├── common.go `package storage`
│   └── testing.go `package storage` <--
└── ...

internal
├── app
│   └── cli
├── pkg
│   └── readcloser
└── test
    └── util.go `package test` <--

* remove comment

* add and use goimports

* add test types & fix some lint errors

* better typing

* fixing linter issues/comments

* goimports

* goimports

* more linter fixes; replace panic with NoError assertions in tests

* fix typo/more linter errors

* moar better linter fixes

* even moar better linter fixes

* linter

* add localPort back

* fixing exports, imports, and add comments
2018-06-13 14:22:32 -04:00
Dennis Coyle
e65e115222 port changes (#80) 2018-06-07 10:15:57 -04:00
Dylan Lott
325a70d514 Cache (#67)
* add reference to dht to overlay client struct

* wip

* wip

* Implement FindNode

* get nodes

* WIP

* Merge in Dennis kademlia code, get it working with our code

* ping and moar

* WIP trying to get cache working with kademlia

* WIP more wiring up

* WIP

* Update service cli commands

* WIP

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* Added initial network overlay explorer page

* Updating and building with dockerfile

* Working on adding bootstrap node code

* WIP merging in dennis' code

* WIP

* connects cache to pkg/kademlia implementation

* WIP redis cache

* testing

* Add bootstrap network function for CLI usage

* cleanup

* call bootstrap on init network

* Add BootstrapNetwork function to interface

* Merge in dennis kad code

* WIP updates to redis/overlay client interface

* WIP trying to get the DHT connected to the cache

* go mod & test

* deps

* Bootstrap node now setting up correctly

- Need to pass it through CLI commands better

* WIP adding refresh and walk functions, added cli flags

- added cli flags for custom bootstrap port and ip

* PR comments addressed

* adding FindStorageNodes to overlay cache

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort

* updates to findstoragenodes response and request

* WIP merge in progress, having issues with a panic

* wip

* adjustments

* update port for dht bootstrap test

* Docker

* wip

* dockerfile

* fixes

* makefile changes

* Update port in NewKademlia call

* Update local kademlia DHT config

* kubernetes yaml

* cleanup

* making tests pass

* k8s yaml

* lint issues

* Edit cli flags to allow for configurable bootstrap IP and Port args

* cleanup

* cache walking the network now

* Rough prototype of Walk function laid out

* Move walk function into bootstrap function

* Update dht.go

* changes to yaml

* goimports
2018-06-05 17:06:37 -04:00
Dennis Coyle
e22810da15
Kademlia Package (#58)
* wip

* wip

* get nodes

* ping and moar

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* testing

* cleanup

* go mod & test

* deps

* PR comments addressed

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort
2018-06-05 07:48:19 -04:00
JT Olio
22b1fe4e21 pkg/process: add pkg/telemetry plumbing (#47)
* pkg/process: add pkg/telemetry plumbing

* pkg/process: add debug endpoints

* fix linting
2018-05-30 10:03:44 -04:00
Dennis Coyle
d6ddb65bf8 basic structure (#39)
* wip - basic structure

* cleanup

* zap logger

* handling errors | using urfav/cli

* remove deadcode

* move NewLogger to utils

* single process method

* copyright

* more generic Process

* simplified

* added NewServer Back
2018-05-16 12:47:59 -06:00
JT Olds
4f40280c71 some small import cleanups, go generate overlay protos (#25)
checking in go generated stuff is recommended by
https://blog.golang.org/generate
2018-05-01 13:24:32 -04:00
Dennis Coyle
ecbd5f08c3 Adding gRPC server/clients for overlay (#14)
* overlay proto

* grpc server and client

* fix import to storj

* tests

* change imports

* imports cleanup/comments

* PR comments addressed from @jtolds
2018-04-23 09:54:22 -06:00
Dennis Coyle
2c08e56b20 fix import to storj 2018-04-16 10:09:01 -06:00
Dennis Coyle
1975b98eba grpc server and client 2018-04-16 10:09:01 -06:00