Commit Graph

52 Commits

Author SHA1 Message Date
Egon Elbre
ec3d5c0bdd
don't use global loggers (#2671)
* pkg/server: don't use global logger
* satellite/overlay: use correct logger
* pkg/kademlia: use correct logger
* linksharing: use conventional way to pass in logger
* use zaptest in tests
2019-07-31 15:09:45 +03:00
Jennifer Li Johnson
338c1b2265
adds comments (#2666) 2019-07-30 14:08:29 -04:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Egon Elbre
5fe95fe8fa kademlia: restrict maximum find near query (#2598) 2019-07-19 18:46:08 +02:00
Jennifer Li Johnson
a7aa854df8
Routing table antechamber should expire old nodes that failed connection (#2387) 2019-07-01 23:20:12 -04:00
Jennifer Li Johnson
699ccea19f
Creates Routing Table Antechamber (#2318) 2019-07-01 17:20:19 -04:00
JT Olio
1ae5654eba kademlia/routing: add contexts to more places so monkit works (#2188) 2019-06-13 16:51:50 +02:00
JT Olio
43d4f3daf5 discovery: remove graveyard (#2145) 2019-06-07 08:40:51 +03:00
JT Olio
f1641af802 storage: add monkit task to missing places (#2122)
* storage: add monkit task to missing places

Change-Id: I9e17a6b14f7c25bbf698eeecf32785e9add3f26e

* fix tests

Change-Id: Id078276fa3de61a28eb3d01d4e751732ecbb173f

* import order

Change-Id: I814e33755b9f10b5219af37cd828cd75eb3da1a4

* remove part of other commit

Change-Id: Idaa4c95cd65e97567fb466de49718db8203cfbe1
2019-06-05 16:23:10 +02:00
JT Olio
9c5708da32 pkg/*: add monkit task to missing places (#2109) 2019-06-04 13:36:27 +02:00
Kaloyan Raev
8fc5fe1d6f
Refactor pb.Node protobuf (#1785) 2019-04-22 12:07:50 +03:00
Egon Elbre
de15a4fdcf
remove utils.CombineErrors and utils.ErrorGroup (#1603) 2019-03-29 14:30:23 +02:00
Dylan Lott
bacd683879
Adds DumpNodes method to Kademlia Inspector (#1328)
* Wiring up DumpNodes response for Inspector

* Finalize everything and test that it works

* remove changes to overlay inspector

* Replace list with iterate method in DumpNodes

* generate proto

* Merge in nodeinfo changes

* Update iterate function in DumpNodes

* Fix import order
2019-02-28 12:55:27 -07:00
Jennifer Li Johnson
cefdff535a
Replace list calls with iterates in kademlia routing table (#1318)
* notes

* adds bool for option to skip self

* wip getKBucketID is still breaking TestAddNode

* fix (#1322)

* remove panic

* remove iteratebuckets

* implements getKBucketRange

* getNodeIDsWithinKBucket

* removes determineFurthestIDWithinK

* wip

* nodeIsWithinNearestK

* removes bucketIDToKey

* removes extra xor in test

* wip

* rebased

* uses create rt w opts

* lint

* fix lint
2019-02-26 11:07:53 -05:00
Jennifer Li Johnson
b5447c6608
Routing table tests (#1270)
This PR includes a new package called testrouting, which implements a very algorithmically slow but hopefully easy-to-keep-operationally-correct in-memory routing table. The routing table also supports writing out its current structure as a DOT graph for visualization. testrouting is primarily meant to help in coming up with generic routing table integration tests.

This PR also includes a new routing table integration test suite that runs against all current routing table implementations. Our existing routing table passes a lot of the tests, but not all of them, still debugging why. I have confirmed the tests should pass with the visualization graphs though.
2019-02-22 13:39:29 -05:00
Bill Thorp
c598ed034d kad.FindNear fix (#1320)
We realized that the Kademlia FindNear() function was

1. not using XOR distance (AKA _totally broken_)
2. largely a duplicate of the RoutingTable FindNear() function

Changes in this PR:

1. upgraded RoutingTable FindNear() to use iterator and restrictions
2. removed unneeded RoutingTable interface
3. made Kademlia wrap methods that were previously accessed via RoutingTable
4. fixed the tests
2019-02-15 22:23:35 -05:00
Bill Thorp
dfd6589fbe
fix kad refresh bad bucket range (#1315)
fix kad refresh bad bucket range
2019-02-14 10:47:03 -05:00
Egon Elbre
e1a8bbdcb6
Kademlia flags cleanup (#1137) 2019-01-29 08:51:07 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Egon Elbre
eb69ecadec
Storage Node Peer (#1005) 2019-01-10 15:13:27 +02:00
Egon Elbre
9e55a7209d
Fix data-race in UpdateSelf (#991) 2019-01-08 18:01:22 +02:00
Alexander Leitner
c9d900fe4d Process for updating storage node stats in kademlia (#904)
* Process for updating node stats in kademlia

* Mutex lock

* Rename and set up the refresher

* Wrap errors

* Lock should be around the if

* Address comments; Build updateSelf funciton in Kademlia Routing Table

* Added test

* Address comments
2019-01-07 22:05:11 -05:00
Egon Elbre
26c2564bd8
pkg/kademlia: simplify code (#958) 2019-01-02 20:57:11 +02:00
Jennifer Li Johnson
a2fa5c4c5a Proper NodeType Handling (#873)
* adds enums to nodetype

* updating nodetype todos

* ran pb updates

* reorder nodetypes

* adding checks

* wip

* wip

* wip

* bug in test-captplanet

* wip

* add values to storagenode, satellite, captplanet binaries

* Cleanup

* more cleanup

* wip

* lint

* lint

* wip

* fixes bug

* regenerate protos

Change-Id: Id270212e8c7479e52641058042cf23b5317ab773

* limit node type changes to kademlia

Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4

* dpanic

Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e

* review comments

Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
2019-01-02 11:47:34 -07:00
Dylan Lott
7569b7d71b
Node lifecycle hooks (#920)
* wires up first draft of lifecycle methods

* creates interface on transport

* node lifecycle hooks works

* linter fixes

* adds error log at connection success

* chnages Observer interface to use context

* Makes Discovery take its own logger

* WIP

* linter fixes

* Test fixes

* adds in ConnFailure code for cache
2018-12-21 21:51:42 -07:00
Dylan Lott
2f6835a3be
Adds a logger to RoutingTable (#922)
* adds delete functions and tests for cache

* adds logger to routing table struct and passes it through to logger

* add loggers
2018-12-21 17:48:53 -07:00
Bill Thorp
3fa094f3da
Add the ability for Kademlia to do a refresh for each bucket (#908)
* added random id in range, added refresh
2018-12-20 16:45:06 -05:00
aligeti
3c7f62be73
KBucket clean up (#898) 2018-12-18 13:04:46 -05:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Dylan Lott
0d05cb26bf
Kademlia Inspector CLI (#657)
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier
2018-11-21 10:31:27 -07:00
Dennis Coyle
e7e0d1daaa
Coyle/disable mock (#627)
* disabled mock overlay from captplanet
2018-11-20 11:54:52 -05:00
Egon Elbre
fe36d5cd7d
Optimize sortByXOR (#571) 2018-11-02 18:46:59 +02:00
Egon Elbre
830241d845
Remove debug logging for routing table (#572) 2018-11-02 16:55:14 +02:00
Egon Elbre
68796d7964
Ensure we can create kademlia with custom storage (#533) 2018-10-26 19:54:00 +03:00
Egon Elbre
03bd93bba7
Make kademlia use less file-descriptors (#498) 2018-10-18 19:20:23 +03:00
Jennifer Li Johnson
dd525eb978
kademlia get nodes (#444)
* get nodes

* restrictions with getnodes

* increases limit

* iterates with restrictions

* testgetnodes

* Tests meetsRequirements

* update id creation
2018-10-16 11:22:31 -04:00
Dennis Coyle
dee2c137c8
Remove BKAD dependency from pkg/kademlia (#294)
* slowly but surely

* hardcode ID for tests so we can get predictable results

* skipping bad test

* removing tests of bkad

* wip

* new algorithm for worker

* clean up

* remove skipped test

* changes

* uncomment

* fixed conflicts

* maybe done ?

* cleanup

* boot bkad

* wip

* cleanup

* undo change

* fixes

* wip

* wip

* moving nodeID around

* wip

* wip

* fixes

* fixes after merge

* added TODO

* fixed tests post identity

* linter fixes

* wip

* PR review comments

* wip

* fixing tests

* fix tests

* force db directory

* bad test

* fixes race condition

* small cleanups

* adding db folder

* testing

* wip

* cleanup

* cleanup

* linters

* export Restrict

* add timeout

* testing

* linters

* forgot one

* moar fixes from master merge

* PR comments

* moar PR comments

* removed stun flag

* remove duplicate declaration

* remove old tests

* remove timeout

* fix tests

* missed one

* changed StringToID >> IDFromString

* PR comments

* stupid linter

* moevd overlay mock

* fixed merge conflicts

* fixes

* linter
2018-10-08 11:09:37 -04:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Egon Elbre
4486e265bb
Cleanup after tests (#341) 2018-09-11 16:57:12 +03:00
Egon Elbre
00ac266bda Remove zap dependency from boltdb (#334)
* Remove zap dependency from boltdb

* Avoid pulling in testing in main binary
2018-09-10 05:52:53 -04:00
Egon Elbre
83df0ee1b0
Implement ListV2 with storage rework (#303)
1. Added KeyValueStore.Iterate for implementing the different List, ListV2 etc. implementations. This allows for more efficient use of memory depending on the situation.
2. Implemented an inmemory teststore for running tests. This should allow to replace MockKeyValueStore in most places.
3. Rewrote tests
4. Pulled out logger from bolt implementation so it can be used for all other storage implementations.
5. Fixed multiple things in bolt and redis implementations.
2018-09-05 19:10:35 +03:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
Jennifer Li Johnson
154a1eab3a
implements node pkg query method (#251) 2018-08-23 11:20:11 -04:00
Jennifer Li Johnson
15657071d7
implements connection success and fail on kad routing table (#249)
* implements connection success and fail on kad routing table

* modifications from code review

* todo

* test fixes

* passes in node rather than id

* removes rpath

* test fix
2018-08-21 14:44:42 -04:00
Jennifer Li Johnson
91bf6e19e8
Routing Table Replacement Cache (#229)
* creating replacement cache

* wip

* wip

* rewrites replacement cache

* replacement cache tests

* update and remove node tests

* check if dropped node got added to replacement cache from add node method

* wip

* wip

* making changes based on pr
2018-08-17 15:11:46 -04:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
Jennifer Li Johnson
5f8142cdcc
Integrating on disk routing table with kademlia (#166)
* adds comment

* runs deps

* adds print statements for debugging add node bkad

* more print statements

* removes bkad from routing and integrates on disk routing table

tests failing :(

wip

* removes testbootstrap

* kademlia_test not working

* adds kad tests back in

* Adds skips for tests broken due to wip kademlia
2018-08-09 15:20:39 -04:00
Yaroslav Vorobiov
f6fb41c5cc Testcoverage kademlia (#154)
* Unit test covarege increased for kademlia pkg

go style formatting added

Removed DHT param from newTestKademlia method, added comments for Bucket methods that informs that these tests will need to be updated

unnecessary comment deleted from newTestKademlia

Adjust Segment Store to the updated interface (#160)

* Adjust Segment Store to the updated interface

* Move /pkg/storage/segment to /pkg/storage/segments

* Fix overlay client tests

* Revert changes in NewOverlayClient return value

* Rename `rem` to `seg`

* Implement Meta()

captplanet (#159)

* captplanet

I kind of went overboard this weekend.

The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!

To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.

Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.

A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
 * Kademlia
 * OverlayCache
 * PointerDB
 * StatDB
 * PieceStore
 * etc.

A Provider is a collection of Responsibilities that
share an Identity, such as:
 * The heavy client
 * The farmer
 * The gateway

An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.

Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.

To make a Responsibility, one should create a "config"
struct, such as:

```
type Config struct {
  RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
  SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```

To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.

cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.

Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
 * cmd/captplanet/farmer/main.go (a new farmer binary)
 * cmd/captplanet/hc/main.go (a new heavy client binary)
 * cmd/captplanet/gw/main.go (a new minio gateway binary)

Usage:

```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```

Configuration is placed by default in `~/.storj/capt/`

Other changes:

 * introduces new config structs for currently existing
   Responsibilities that conform to the new Responsibility
   interface. Please see the `pkg/*/config.go` files for
   examples.

 * integrates the PointerDB API key with other global
   configuration via flags, instead of through environment
   variables through viper like it's been doing. (ultimately
   this should also change to use the PointerDB config
   struct but this is an okay shortterm solution).

 * changes the Overlay cache to use a URL for database
   configuration instead of separate redis and bolt config
   settings.

 * stubs out some peer identity skeleton code (but not the
   meat).

 * Fixes the SegmentStore to use the overlay client and
   pointerdb clients instead of gRPC client code directly

 * Leaves a very clear spot where we need to tie the object to
   stream to segment store together. There's sort of a "golden
   spike" opportunity to connect all the train tracks together
   at the bottom of pkg/miniogw/config.go, labeled with a
   bunch of TODOs.

Future stuff:

 * I now prefer this design over the original
   pkg/process.Service thing I had been pushing before (sorry!)

 * The experience of trying to have multiple farmers
   configurable concurrently led me to prefer config structs
   over global flags (I finally came around) or using viper
   directly. I think global flags are okay sometimes but in
   general going forward we should try and get all relevant
   config into config structs.

 * If you all like this direction, I think we can go delete my
   old Service interfaces and a bunch of flags and clean up a
   bunch of stuff.

 * If you don't like this direction, it's no sweat at all, and
   despite how much code there is here I'm not very tied to any
   of this! Considering a lot of this was written between midnight
   and 6 am, it might not be any good!

* bind tests

Add files for testing builds in docker (#161)

* Add files for testing builds in docker

* Make tests check for redis running before trying to start redis-server, which may not exist.

* Clean redis server before any tests use it.

* Add more debugging for travis

* Explicitly requiring redis for travis

pkg/provider: with pkg/provider merged, make a single heavy client binary, gateway binary, and deprecate old services (#165)

* pkg/provider: with pkg/provider merged, make a single heavy client binary and deprecate old services

* add setup to gw binary too

* captplanet: output what addresses everything is listening on

* revert peertls/io_util changes

* define config flag across all commands

* use trimsuffix

fix docker makefile (#170)

* fix makefile

protos: update protobufs with go generate (#169)

the import for timestamp and duration should use
the path provided by a standard protocol buffer library
installation

Refactor List in PointerDB (#163)

* Refactor List in Pointer DB

* Fix pointerdb-client example

* Fix issue in Path type related to empty paths

* Test for the PointerDB service with some fixes

* Fixed debug message in example: trancated --> more

* GoDoc comments for unexported methods

* TODO comment to check if Put is overwriting

* Log warning if protobuf timestamp cannot be converted

* TODO comment to make ListPageLimit configurable

* Rename 'segment' package to 'segments' to reflect folder name

Minio integration with Object store (#156)

* initial WIP integration with Object store

* List WIP

* minio listobject function changes complete

* Code review changes and work in progress for the mock objectstore unit testing cases

* Warning fix redeclaration of err

* Warning fix redeclaration of err

*  code review comments & unit testing inprogress

* fix compilation bug

* Fixed code review comments & added GetObject Mock test case

* rearraged the mock test file and gateway storj test file in to the proper directory

* added the missing file

* code clean up

* fix lint error on the mock generated code

* modified per code review comments

* added the PutObject mock test case

* added the GetObjectInfo  mock test case

* added listobject mock test case

* fixed package from storj to miniogw

* resolved the gateway-storj.go initialization merge conflict

update readme (#174)

added assertion for unused errors (#152)

merging this PR to avoid future issues

updating github user to personal account (#171)

Test coverage ranger (#168)

* Fixed go panic for corner case

* Initial test coverage for ranger pkg

streamstore: add passthrough implementation (#176)

this doesn't implement streamstore, this just allows us to try and
get the june demo working again in the meantime

StatDB (#144)

* add statdb proto and example client

* server logic

* update readme

* remove boltdb from service.go

* sqlite3

* add statdb server executable file

* create statdb node table if it does not exist already

* get UpdateBatch working

* update based on jt review

* remove some commented lines

* fix linting issues

* reformat

* apiKey -> APIKey

* update statdb client apiKey->APIKey

Update README.md

Update README.md

overlay: correct dockerfile db (#179)

cmd/hc, cmd/gw, cmd/captplanet: simplify setup/run commands (#178)

also allows much more customization of services within captain planet,
such as reconfiguring the overlay service to use redis

pkg/process: don't require json formatting (#177)

Cleanup metadata across layers (#180)

* Cleanup metadata across layers

* Fix pointer db tests

Kademlia Routing Table (#164)

* adds comment

* runs deps

* creates boltdb kademlia routing table

* protobuf updates

* adds reverselist to mockkeyvaluestore interface

* xor wip

* xor wip

* fixes xor sort

* runs go fmt

* fixes

* goimports again

* trying to fix travis tests

* fixes mock tests

Ranger refactoring (#158)

* Fixed go panic for corner case

* Cosmetic changes, and small error fixes

miniogw: log all errors (#182)

* miniogw: log all errors

* tests added

* doc comment to satisfy linter

* fix test failure

Jennifer added to CLA list

* Temporary fix for storage/redis list method test
2018-08-02 14:36:57 -04:00
Dennis Coyle
a5ff5016c9
pkg/kademlia tests and restructuring (#97)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* add changes from kad demo

* PR comments addresses

* go func

* force travis build

* fixed merge conflicts

* fixed merge conflicts

* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests

* linter issues

* linting issues

* fixed merge conflicts

* linter is stupid
2018-06-22 09:33:57 -04:00
Dylan Lott
325a70d514 Cache (#67)
* add reference to dht to overlay client struct

* wip

* wip

* Implement FindNode

* get nodes

* WIP

* Merge in Dennis kademlia code, get it working with our code

* ping and moar

* WIP trying to get cache working with kademlia

* WIP more wiring up

* WIP

* Update service cli commands

* WIP

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* Added initial network overlay explorer page

* Updating and building with dockerfile

* Working on adding bootstrap node code

* WIP merging in dennis' code

* WIP

* connects cache to pkg/kademlia implementation

* WIP redis cache

* testing

* Add bootstrap network function for CLI usage

* cleanup

* call bootstrap on init network

* Add BootstrapNetwork function to interface

* Merge in dennis kad code

* WIP updates to redis/overlay client interface

* WIP trying to get the DHT connected to the cache

* go mod & test

* deps

* Bootstrap node now setting up correctly

- Need to pass it through CLI commands better

* WIP adding refresh and walk functions, added cli flags

- added cli flags for custom bootstrap port and ip

* PR comments addressed

* adding FindStorageNodes to overlay cache

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort

* updates to findstoragenodes response and request

* WIP merge in progress, having issues with a panic

* wip

* adjustments

* update port for dht bootstrap test

* Docker

* wip

* dockerfile

* fixes

* makefile changes

* Update port in NewKademlia call

* Update local kademlia DHT config

* kubernetes yaml

* cleanup

* making tests pass

* k8s yaml

* lint issues

* Edit cli flags to allow for configurable bootstrap IP and Port args

* cleanup

* cache walking the network now

* Rough prototype of Walk function laid out

* Move walk function into bootstrap function

* Update dht.go

* changes to yaml

* goimports
2018-06-05 17:06:37 -04:00