Commit Graph

170 Commits

Author SHA1 Message Date
Jennifer Li Johnson
3230762041
Kademlia Routing Table (#164)
* adds comment

* runs deps

* creates boltdb kademlia routing table

* protobuf updates

* adds reverselist to mockkeyvaluestore interface

* xor wip

* xor wip

* fixes xor sort

* runs go fmt

* fixes

* goimports again

* trying to fix travis tests

* fixes mock tests
2018-07-30 15:25:18 -04:00
Natalie Villasana
80727ae90b adds netstate pagination (#95)
* adds netstate rpc server pagination, mocks pagination in test/util.go

* updates ns client example, combines ns client and server test to netstate_test, adds pagination to bolt client

* better organizes netstate test calls

* wip breaking netstate test into smaller tests

* wip modularizing netstate tests

* adds some test panics

* wip netstate test attempts

* testing bug in netstate TestDeleteAuth

* wip fixes global variable problem, still issues with list

* wip fixes get request params and args

* fixes bug in path when using MakePointers helper fn

* updates mockdb list func, adds test, changes Limit to int

* fixes merge conflicts

* fixes broken tests from merge

* remove unnecessary PointerEntry struct

* removes error when Get returns nil value from boltdb

* breaks boltdb client tests into smaller tests

* renames AssertNoErr test helper to HandleErr

* adds StartingKey and Limit parameters to redis list func, adds beginning of redis tests

* adds helper func for mockdb List function

* if no starting key provided for netstate List, the first value in storage will be used

* adds basic pagination for redis List function, adds tests

* adds list limit to call in overlay/server.go

* streamlines/fixes some nits from review

* removes use of obsolete EncryptedUnencryptedSize

* uses MockKeyValueStore instead of redis instance in redis client test

* changes test to expect nil returned for getting missing key

* remove error from `KeyValueStore#Get`

* fix bolt test

* Merge pull request #1 from bryanchriswhite/nat-pagination

remove error from `KeyValueStore#Get`

* adds Get returning error back to KeyValueStore interface and affected clients

* trying to appease travis: returns errors in Get calls in overlay/cache and cache_test

* handles redis get error when no key found
2018-06-29 16:06:25 -04:00
JT Olio
5ee6b76d41 internal/test: switch errors to error classes (#96)
* internal/test: switch errors to error classes

if you construct an error directly at package init time, you
won't get useful stack traces or anything. zeebo/errs expects that
you always construct an error (.New) when the error actually
happens. instead, you call Class at init time, then use (Class).Has
to test for error type membership.

* fix linter

* fix test
2018-06-19 20:03:46 +02:00
Bryan White
1ebd66d880 Cleanup & bolt-backed cache (#87)
* wip post-demos cleanup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* Reorganize:

storage
├── common
│   └── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common
│   ├── common.go `package storage`
│   └── testing.go `package storage` <--
└── ...

internal
├── app
│   └── cli
├── pkg
│   └── readcloser
└── test
    └── util.go `package test` <--

* remove comment

* add and use goimports

* add test types & fix some lint errors

* better typing

* fixing linter issues/comments

* goimports

* goimports

* more linter fixes; replace panic with NoError assertions in tests

* fix typo/more linter errors

* moar better linter fixes

* even moar better linter fixes

* linter

* add localPort back

* fixing exports, imports, and add comments
2018-06-13 14:22:32 -04:00
Dylan Lott
325a70d514 Cache (#67)
* add reference to dht to overlay client struct

* wip

* wip

* Implement FindNode

* get nodes

* WIP

* Merge in Dennis kademlia code, get it working with our code

* ping and moar

* WIP trying to get cache working with kademlia

* WIP more wiring up

* WIP

* Update service cli commands

* WIP

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* Added initial network overlay explorer page

* Updating and building with dockerfile

* Working on adding bootstrap node code

* WIP merging in dennis' code

* WIP

* connects cache to pkg/kademlia implementation

* WIP redis cache

* testing

* Add bootstrap network function for CLI usage

* cleanup

* call bootstrap on init network

* Add BootstrapNetwork function to interface

* Merge in dennis kad code

* WIP updates to redis/overlay client interface

* WIP trying to get the DHT connected to the cache

* go mod & test

* deps

* Bootstrap node now setting up correctly

- Need to pass it through CLI commands better

* WIP adding refresh and walk functions, added cli flags

- added cli flags for custom bootstrap port and ip

* PR comments addressed

* adding FindStorageNodes to overlay cache

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort

* updates to findstoragenodes response and request

* WIP merge in progress, having issues with a panic

* wip

* adjustments

* update port for dht bootstrap test

* Docker

* wip

* dockerfile

* fixes

* makefile changes

* Update port in NewKademlia call

* Update local kademlia DHT config

* kubernetes yaml

* cleanup

* making tests pass

* k8s yaml

* lint issues

* Edit cli flags to allow for configurable bootstrap IP and Port args

* cleanup

* cache walking the network now

* Rough prototype of Walk function laid out

* Move walk function into bootstrap function

* Update dht.go

* changes to yaml

* goimports
2018-06-05 17:06:37 -04:00
Natalie Villasana
6723064bfb
Enables netstate service to save pointers (#49)
* adds pointer to netstate proto file

* generated updated netstate proto

* changes boltdb netstate to save pointers as values

* updates netstate Put to save Pointers, updates client example to put a pointer, adds grpc status errors, updates tests, changes boltdb 'File' struct to 'PointerEntry'

* updates netstate client example and client test to save pointers, updates netstate List and Delete

* begins adding netstate-http tests

* removes netstate http service

* re-adds netstate auth

* updates boltdb netstate test

* changes encrypted_unencrypted_size from int64 to bytes in netstate proto

* updates READMEs
2018-05-29 22:47:40 -04:00
Dennis Coyle
d6ddb65bf8 basic structure (#39)
* wip - basic structure

* cleanup

* zap logger

* handling errors | using urfav/cli

* remove deadcode

* move NewLogger to utils

* single process method

* copyright

* more generic Process

* simplified

* added NewServer Back
2018-05-16 12:47:59 -06:00
Natalie Villasana
1a06c2b0c7 adding grpc interface to network state (#33)
* adds proto files for netstate crud

* moves netstate grpc client lib into pkg/netstate where grpc netstate service is defined

* starts adding grpc client and server tests

* moves creation of grpc server into cmd/netstate/main.go, removes pkg/netstate/service.go, adds more client testing

* changed all 'Path' and 'Value' fields from strings to bytes, updated tests

* changes Get and Delete in proto file to receive 'requests' instead of 'file paths', adds tests for Get, List, and Delete

* changes netstate-routes to get 'fileValue' bytes not 'fileInfo'

* adds example rpc client in 'examples' and adds more specific debug logs

* adds readmes for netstate rpc services and updates netstate-routes
2018-05-14 18:31:26 -06:00
JT Olds
2ef8e581b2 fix travis, and get a clean lint (#32)
* fix travis

* lint fixes

* travis: enable tests

* fix some tests, disable a broken eestream test for now
2018-05-08 18:02:01 -04:00
Dennis Coyle
5e9b3f7d6a merge conflicts 2018-05-07 14:04:53 -04:00
Natalie Ventura Villasana
7907733629 adds panics to netstate test 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
4e8336f2ab adds zap logger, some flags, some nit fixes 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
d93dacb53e fixes some error handling, updates storj import links 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
8d8fd76748 adds suggested organization, zeebo, err handling 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
5dec8fafaf adds bolt crud interface w put, get, list, delete 2018-05-07 11:03:59 -06:00
Dennis Coyle
bb8d4f8d22 cleanup 2018-04-25 10:55:26 -04:00
Dennis Coyle
af30d8ffa1 fixing import 2018-04-23 09:54:00 -06:00
Dennis Coyle
86e2657bdd PR comments from @jtolds 2018-04-23 09:54:00 -06:00
Dennis Coyle
e6e1dae7fc redis client 2018-04-23 09:54:00 -06:00
Dennis Coyle
250b36f27b wip on structure 2018-04-06 12:32:34 -04:00