Commit Graph

30 Commits

Author SHA1 Message Date
Kaloyan Raev
d1639c4157 Merge statdb pkg into overlay pkg (#1570) 2019-03-25 18:25:09 -04:00
Egon Elbre
94e79eda6d
remove overlay endpoint (#1521) 2019-03-23 10:06:11 +02:00
Bill Thorp
66718cc5e6
Development defaults for configuration (#1430)
added --dev command line option, cfgstruct.DevFlag(), and cfgstruct.SetupFlag()
2019-03-12 08:51:06 -04:00
Egon Elbre
12261c0624
Satellite migrations (#1301) 2019-02-14 23:55:21 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
a2fa5c4c5a Proper NodeType Handling (#873)
* adds enums to nodetype

* updating nodetype todos

* ran pb updates

* reorder nodetypes

* adding checks

* wip

* wip

* wip

* bug in test-captplanet

* wip

* add values to storagenode, satellite, captplanet binaries

* Cleanup

* more cleanup

* wip

* lint

* lint

* wip

* fixes bug

* regenerate protos

Change-Id: Id270212e8c7479e52641058042cf23b5317ab773

* limit node type changes to kademlia

Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4

* dpanic

Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e

* review comments

Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
2019-01-02 11:47:34 -07:00
Egon Elbre
d9b9ae6ffa
Cleanup overlay methods and names. (#914) 2018-12-20 15:57:54 +02:00
Michal Niewrzal
bbe1dbb92c
Fix overlay commands after master db merge (#895)
* Fix overlay commands after master db merge

* remove unused var

* remove unused comment
2018-12-18 21:41:31 +01:00
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830)
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
2018-12-14 15:17:30 -05:00
Egon Elbre
c56307f5b9
datarepair: remove unneeded debug message (#839)
* datarepair: remove unneeded debug message
* Make overlay-cache and routing table logs more specific
2018-12-12 17:39:16 +02:00
Bill Thorp
ba5f71810e
added tally to capt run.go (#751)
* added tally to capt run.go, ParseURL -> SplitDBURL
2018-12-12 08:15:34 -05:00
Maximillian von Briesen
6c655d117b
Filter nodes by reputation and IP address (node selection) (#720)
*   Pulls statdb stats into overlay cache whenever cache.Put() is called
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to filter based on node stats
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to exclude duplicate IP addresses
2018-12-04 15:18:26 -05:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Egon Elbre
0ae05cf834
Ensure that we only choose storage nodes (#732) 2018-11-29 16:57:00 +02:00
Maximillian von Briesen
7cf16503b8
Add statdb to overlay cache (node selection) (#711)
* make overlay node rep consistent with statdb node rep

* add statdb to testplanet

* add statdb to overlay cache

* tests
2018-11-27 12:46:12 -05:00
Egon Elbre
6fe16e48ba
Simplify and fix tests (#660) 2018-11-16 18:31:14 +02:00
Egon Elbre
db42772b4d
Enable goimports and fix formatting. (#459) 2018-10-11 23:25:54 +03:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
Bryan White
f932c3ced8 Overlay cache commands (#236)
* wip overlay cache cmd

* overlay cache add and list commands

* use zap

* fixes

* simplify

* add restrictions

* cleanup

* remove defaults for overlay management commands
2018-08-20 14:21:41 -04:00
JT Olio
45a3c2e974
pkg/provider: with pkg/provider merged, make a single heavy client binary, gateway binary, and deprecate old services (#165)
* pkg/provider: with pkg/provider merged, make a single heavy client binary and deprecate old services

* add setup to gw binary too

* captplanet: output what addresses everything is listening on

* revert peertls/io_util changes

* define config flag across all commands

* use trimsuffix
2018-07-26 08:21:35 -06:00
Dylan Lott
15960d4269
Admin node (#131)
* WIP creating admin node service

- WIP changing the process pkg to accept multiple services
- WIP looping over services passed to process
- add netstate/service.go file and abstract it for service processing

* implement goroutine to launch each process

* goroutines working with multiple services

* code review fixes

* more code updates for review

* Add pkg lock and mod files back in

* code review updates

* update process.Main with better concurrent error handling

* Update error handling and pass ctx to StartService

* Update error handling with channel implementation

* Merge in upstream changes

- Simplify error handling channels

* updates

* Updates per reviewable

* fix test

* Setup test exec

* Scaffold test setup

* process main test working

* update admin process test

* Test multiple processes done

* Add error classes for testing, test main logger error

* Updates to tests

* Update how process.Main() handles configs

* Complete merge

* Update Gopkg and add Copyright

* Fix cyclical import issue

- Added .coverprofile to gitignore
- Update admin main.go function call

* remove unnecessary line

* Updates

* DRY up cmd/netstate package

* update service function calls

* updates

* Trying no-ops in examples

* rename netstate to pointerdb

* trying to fix merge

* dep ensure and run tests

* remove flag.Parse

* Update deps

* Skip offending test in pkg/process, to be fixed later
2018-07-09 16:43:32 -06:00
Dennis Coyle
0e8931a299
docker image issue fixed (#118)
* issue fixed

* git

* fixed finally

* ports and suggestions

* fix
2018-07-02 15:18:28 -04:00
Dennis Coyle
281d037468
hardcode version (#111)
* hardcode version
2018-06-22 16:30:08 -04:00
Dennis Coyle
2b4a1ef17d
Coyle/docker fix (#109)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge branch 'master' of https://github.com/storj/storj

* fixing tag name

* no idea

* testing

* changes

* testing on travis

* testing

* changes to travis build

* new approach

* Merge branch 'master' into coyle/docker-fix
2018-06-22 16:17:37 -04:00
Dennis Coyle
a5ff5016c9
pkg/kademlia tests and restructuring (#97)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* add changes from kad demo

* PR comments addresses

* go func

* force travis build

* fixed merge conflicts

* fixed merge conflicts

* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests

* linter issues

* linting issues

* fixed merge conflicts

* linter is stupid
2018-06-22 09:33:57 -04:00
Dennis Coyle
e65e115222 port changes (#80) 2018-06-07 10:15:57 -04:00
Dylan Lott
325a70d514 Cache (#67)
* add reference to dht to overlay client struct

* wip

* wip

* Implement FindNode

* get nodes

* WIP

* Merge in Dennis kademlia code, get it working with our code

* ping and moar

* WIP trying to get cache working with kademlia

* WIP more wiring up

* WIP

* Update service cli commands

* WIP

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* Added initial network overlay explorer page

* Updating and building with dockerfile

* Working on adding bootstrap node code

* WIP merging in dennis' code

* WIP

* connects cache to pkg/kademlia implementation

* WIP redis cache

* testing

* Add bootstrap network function for CLI usage

* cleanup

* call bootstrap on init network

* Add BootstrapNetwork function to interface

* Merge in dennis kad code

* WIP updates to redis/overlay client interface

* WIP trying to get the DHT connected to the cache

* go mod & test

* deps

* Bootstrap node now setting up correctly

- Need to pass it through CLI commands better

* WIP adding refresh and walk functions, added cli flags

- added cli flags for custom bootstrap port and ip

* PR comments addressed

* adding FindStorageNodes to overlay cache

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort

* updates to findstoragenodes response and request

* WIP merge in progress, having issues with a panic

* wip

* adjustments

* update port for dht bootstrap test

* Docker

* wip

* dockerfile

* fixes

* makefile changes

* Update port in NewKademlia call

* Update local kademlia DHT config

* kubernetes yaml

* cleanup

* making tests pass

* k8s yaml

* lint issues

* Edit cli flags to allow for configurable bootstrap IP and Port args

* cleanup

* cache walking the network now

* Rough prototype of Walk function laid out

* Move walk function into bootstrap function

* Update dht.go

* changes to yaml

* goimports
2018-06-05 17:06:37 -04:00
Dennis Coyle
d6ddb65bf8 basic structure (#39)
* wip - basic structure

* cleanup

* zap logger

* handling errors | using urfav/cli

* remove deadcode

* move NewLogger to utils

* single process method

* copyright

* more generic Process

* simplified

* added NewServer Back
2018-05-16 12:47:59 -06:00