storj/pkg/overlay/overlay.go

52 lines
1.6 KiB
Go
Raw Normal View History

// Copyright (C) 2018 Storj Labs, Inc.
// See LICENSE for copying information.
2018-04-12 14:50:22 +01:00
package overlay
import (
"context"
Cache (#67) * add reference to dht to overlay client struct * wip * wip * Implement FindNode * get nodes * WIP * Merge in Dennis kademlia code, get it working with our code * ping and moar * WIP trying to get cache working with kademlia * WIP more wiring up * WIP * Update service cli commands * WIP * added GetNodes * added nodes to Kbucket * default transport changed to TCP * GetBuckets interface changed * filling in more routing * timestamp methods * removed store * Added initial network overlay explorer page * Updating and building with dockerfile * Working on adding bootstrap node code * WIP merging in dennis' code * WIP * connects cache to pkg/kademlia implementation * WIP redis cache * testing * Add bootstrap network function for CLI usage * cleanup * call bootstrap on init network * Add BootstrapNetwork function to interface * Merge in dennis kad code * WIP updates to redis/overlay client interface * WIP trying to get the DHT connected to the cache * go mod & test * deps * Bootstrap node now setting up correctly - Need to pass it through CLI commands better * WIP adding refresh and walk functions, added cli flags - added cli flags for custom bootstrap port and ip * PR comments addressed * adding FindStorageNodes to overlay cache * fix GetBucket * using SplitHostPort * Use JoinHostPort * updates to findstoragenodes response and request * WIP merge in progress, having issues with a panic * wip * adjustments * update port for dht bootstrap test * Docker * wip * dockerfile * fixes * makefile changes * Update port in NewKademlia call * Update local kademlia DHT config * kubernetes yaml * cleanup * making tests pass * k8s yaml * lint issues * Edit cli flags to allow for configurable bootstrap IP and Port args * cleanup * cache walking the network now * Rough prototype of Walk function laid out * Move walk function into bootstrap function * Update dht.go * changes to yaml * goimports
2018-06-05 22:06:37 +01:00
"go.uber.org/zap"
"gopkg.in/spacemonkeygo/monkit.v2"
Cache (#67) * add reference to dht to overlay client struct * wip * wip * Implement FindNode * get nodes * WIP * Merge in Dennis kademlia code, get it working with our code * ping and moar * WIP trying to get cache working with kademlia * WIP more wiring up * WIP * Update service cli commands * WIP * added GetNodes * added nodes to Kbucket * default transport changed to TCP * GetBuckets interface changed * filling in more routing * timestamp methods * removed store * Added initial network overlay explorer page * Updating and building with dockerfile * Working on adding bootstrap node code * WIP merging in dennis' code * WIP * connects cache to pkg/kademlia implementation * WIP redis cache * testing * Add bootstrap network function for CLI usage * cleanup * call bootstrap on init network * Add BootstrapNetwork function to interface * Merge in dennis kad code * WIP updates to redis/overlay client interface * WIP trying to get the DHT connected to the cache * go mod & test * deps * Bootstrap node now setting up correctly - Need to pass it through CLI commands better * WIP adding refresh and walk functions, added cli flags - added cli flags for custom bootstrap port and ip * PR comments addressed * adding FindStorageNodes to overlay cache * fix GetBucket * using SplitHostPort * Use JoinHostPort * updates to findstoragenodes response and request * WIP merge in progress, having issues with a panic * wip * adjustments * update port for dht bootstrap test * Docker * wip * dockerfile * fixes * makefile changes * Update port in NewKademlia call * Update local kademlia DHT config * kubernetes yaml * cleanup * making tests pass * k8s yaml * lint issues * Edit cli flags to allow for configurable bootstrap IP and Port args * cleanup * cache walking the network now * Rough prototype of Walk function laid out * Move walk function into bootstrap function * Update dht.go * changes to yaml * goimports
2018-06-05 22:06:37 +01:00
"storj.io/storj/pkg/kademlia"
proto "storj.io/storj/protos/overlay" // naming proto to avoid confusion with this package
2018-04-12 14:50:22 +01:00
)
// Overlay implements our overlay RPC service
Cache (#67) * add reference to dht to overlay client struct * wip * wip * Implement FindNode * get nodes * WIP * Merge in Dennis kademlia code, get it working with our code * ping and moar * WIP trying to get cache working with kademlia * WIP more wiring up * WIP * Update service cli commands * WIP * added GetNodes * added nodes to Kbucket * default transport changed to TCP * GetBuckets interface changed * filling in more routing * timestamp methods * removed store * Added initial network overlay explorer page * Updating and building with dockerfile * Working on adding bootstrap node code * WIP merging in dennis' code * WIP * connects cache to pkg/kademlia implementation * WIP redis cache * testing * Add bootstrap network function for CLI usage * cleanup * call bootstrap on init network * Add BootstrapNetwork function to interface * Merge in dennis kad code * WIP updates to redis/overlay client interface * WIP trying to get the DHT connected to the cache * go mod & test * deps * Bootstrap node now setting up correctly - Need to pass it through CLI commands better * WIP adding refresh and walk functions, added cli flags - added cli flags for custom bootstrap port and ip * PR comments addressed * adding FindStorageNodes to overlay cache * fix GetBucket * using SplitHostPort * Use JoinHostPort * updates to findstoragenodes response and request * WIP merge in progress, having issues with a panic * wip * adjustments * update port for dht bootstrap test * Docker * wip * dockerfile * fixes * makefile changes * Update port in NewKademlia call * Update local kademlia DHT config * kubernetes yaml * cleanup * making tests pass * k8s yaml * lint issues * Edit cli flags to allow for configurable bootstrap IP and Port args * cleanup * cache walking the network now * Rough prototype of Walk function laid out * Move walk function into bootstrap function * Update dht.go * changes to yaml * goimports
2018-06-05 22:06:37 +01:00
type Overlay struct {
kad *kademlia.Kademlia
cache *Cache
Cache (#67) * add reference to dht to overlay client struct * wip * wip * Implement FindNode * get nodes * WIP * Merge in Dennis kademlia code, get it working with our code * ping and moar * WIP trying to get cache working with kademlia * WIP more wiring up * WIP * Update service cli commands * WIP * added GetNodes * added nodes to Kbucket * default transport changed to TCP * GetBuckets interface changed * filling in more routing * timestamp methods * removed store * Added initial network overlay explorer page * Updating and building with dockerfile * Working on adding bootstrap node code * WIP merging in dennis' code * WIP * connects cache to pkg/kademlia implementation * WIP redis cache * testing * Add bootstrap network function for CLI usage * cleanup * call bootstrap on init network * Add BootstrapNetwork function to interface * Merge in dennis kad code * WIP updates to redis/overlay client interface * WIP trying to get the DHT connected to the cache * go mod & test * deps * Bootstrap node now setting up correctly - Need to pass it through CLI commands better * WIP adding refresh and walk functions, added cli flags - added cli flags for custom bootstrap port and ip * PR comments addressed * adding FindStorageNodes to overlay cache * fix GetBucket * using SplitHostPort * Use JoinHostPort * updates to findstoragenodes response and request * WIP merge in progress, having issues with a panic * wip * adjustments * update port for dht bootstrap test * Docker * wip * dockerfile * fixes * makefile changes * Update port in NewKademlia call * Update local kademlia DHT config * kubernetes yaml * cleanup * making tests pass * k8s yaml * lint issues * Edit cli flags to allow for configurable bootstrap IP and Port args * cleanup * cache walking the network now * Rough prototype of Walk function laid out * Move walk function into bootstrap function * Update dht.go * changes to yaml * goimports
2018-06-05 22:06:37 +01:00
logger *zap.Logger
metrics *monkit.Registry
}
2018-04-12 14:50:22 +01:00
// Lookup finds the address of a node in our overlay network
func (o *Overlay) Lookup(ctx context.Context, req *proto.LookupRequest) (*proto.LookupResponse, error) {
na, err := o.cache.Get(ctx, req.NodeID)
Cache (#67) * add reference to dht to overlay client struct * wip * wip * Implement FindNode * get nodes * WIP * Merge in Dennis kademlia code, get it working with our code * ping and moar * WIP trying to get cache working with kademlia * WIP more wiring up * WIP * Update service cli commands * WIP * added GetNodes * added nodes to Kbucket * default transport changed to TCP * GetBuckets interface changed * filling in more routing * timestamp methods * removed store * Added initial network overlay explorer page * Updating and building with dockerfile * Working on adding bootstrap node code * WIP merging in dennis' code * WIP * connects cache to pkg/kademlia implementation * WIP redis cache * testing * Add bootstrap network function for CLI usage * cleanup * call bootstrap on init network * Add BootstrapNetwork function to interface * Merge in dennis kad code * WIP updates to redis/overlay client interface * WIP trying to get the DHT connected to the cache * go mod & test * deps * Bootstrap node now setting up correctly - Need to pass it through CLI commands better * WIP adding refresh and walk functions, added cli flags - added cli flags for custom bootstrap port and ip * PR comments addressed * adding FindStorageNodes to overlay cache * fix GetBucket * using SplitHostPort * Use JoinHostPort * updates to findstoragenodes response and request * WIP merge in progress, having issues with a panic * wip * adjustments * update port for dht bootstrap test * Docker * wip * dockerfile * fixes * makefile changes * Update port in NewKademlia call * Update local kademlia DHT config * kubernetes yaml * cleanup * making tests pass * k8s yaml * lint issues * Edit cli flags to allow for configurable bootstrap IP and Port args * cleanup * cache walking the network now * Rough prototype of Walk function laid out * Move walk function into bootstrap function * Update dht.go * changes to yaml * goimports
2018-06-05 22:06:37 +01:00
if err != nil {
o.logger.Error("Error looking up node", zap.Error(err), zap.String("nodeID", req.NodeID))
return nil, err
}
return &proto.LookupResponse{
NodeAddress: na,
}, nil
2018-04-12 14:50:22 +01:00
}
// FindStorageNodes searches the overlay network for nodes that meet the provided requirements
func (o *Overlay) FindStorageNodes(ctx context.Context, req *proto.FindStorageNodesRequest) (*proto.FindStorageNodesResponse, error) {
Cache (#67) * add reference to dht to overlay client struct * wip * wip * Implement FindNode * get nodes * WIP * Merge in Dennis kademlia code, get it working with our code * ping and moar * WIP trying to get cache working with kademlia * WIP more wiring up * WIP * Update service cli commands * WIP * added GetNodes * added nodes to Kbucket * default transport changed to TCP * GetBuckets interface changed * filling in more routing * timestamp methods * removed store * Added initial network overlay explorer page * Updating and building with dockerfile * Working on adding bootstrap node code * WIP merging in dennis' code * WIP * connects cache to pkg/kademlia implementation * WIP redis cache * testing * Add bootstrap network function for CLI usage * cleanup * call bootstrap on init network * Add BootstrapNetwork function to interface * Merge in dennis kad code * WIP updates to redis/overlay client interface * WIP trying to get the DHT connected to the cache * go mod & test * deps * Bootstrap node now setting up correctly - Need to pass it through CLI commands better * WIP adding refresh and walk functions, added cli flags - added cli flags for custom bootstrap port and ip * PR comments addressed * adding FindStorageNodes to overlay cache * fix GetBucket * using SplitHostPort * Use JoinHostPort * updates to findstoragenodes response and request * WIP merge in progress, having issues with a panic * wip * adjustments * update port for dht bootstrap test * Docker * wip * dockerfile * fixes * makefile changes * Update port in NewKademlia call * Update local kademlia DHT config * kubernetes yaml * cleanup * making tests pass * k8s yaml * lint issues * Edit cli flags to allow for configurable bootstrap IP and Port args * cleanup * cache walking the network now * Rough prototype of Walk function laid out * Move walk function into bootstrap function * Update dht.go * changes to yaml * goimports
2018-06-05 22:06:37 +01:00
// NB: call FilterNodeReputation from node_reputation package to find nodes for storage
// TODO(coyle): need to determine if we will pull the startID and Limit from the request or just use hardcoded data
// for now just using 40 for demos and empty string which will default the Id to the kademlia node doing the lookup
nodes, err := o.kad.GetNodes(ctx, "", 40)
if err != nil {
o.logger.Error("Error getting nodes", zap.Error(err))
return nil, err
}
return &proto.FindStorageNodesResponse{
Node: nodes,
}, nil
2018-04-12 14:50:22 +01:00
}