Commit Graph

32 Commits

Author SHA1 Message Date
Egon Elbre
0c366c1ed2 Clearer code for node selection (#1173) 2019-01-31 13:49:00 -05:00
Natalie Villasana
8d7944bcf8
new node selection in overlay (#1136) 2019-01-29 14:42:43 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Egon Elbre
5de7f8af7f
Satellite Peer (#1119) 2019-01-23 21:58:44 +02:00
Egon Elbre
6c48d69c38
Disable mounting for bsd-like systems (#1109)
* Disable mounting for bsd-like
* Remove debug code
2019-01-22 15:28:57 +02:00
Egon Elbre
78dc02b758 Satellite Peer (#1034)
* add satellite peer

* Add overlay

* reorganize kademlia

* add RunRefresh

* add refresh to storagenode.Peer

* add discovery

* add agreements and metainfo

* rename

* add datarepair checker

* add repair

* add todo notes for audit

* add testing interface

* add into testplanet

* fixes

* fix compilation errors

* fix compilation errors

* make testplanet run

* remove audit refrences

* ensure that audit tests run

* dev

* checker tests compilable

* fix discovery

* fix compilation

* fix

* fix

* dev

* fix

* disable auth

* fixes

* revert go.mod/sum

* fix linter errors

* fix

* fix copyright

* Add address param for SN dashboard (#1076)

* Rename storj-sdk to storj-sim (#1078)

* Storagenode logs and config improvements  (#1075)

* Add more info to SN logs

* remove config-dir from user config

* add output where config was stored

* add message for successful connection

* fix linter

* remove storage.path from user config

* resolve config path

* move success  message to info

* log improvements

* Remove captplanet (#1070)

* pkg/server: include production cert (#1082)

Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792

* Generate Payments Report (#1079)

* memory.Size: autoformat sizes based on value entropy (#1081)

* Jj/bytes (#1085)

* run tally and rollup

* sets dev default tally and rollup intervals

* nonessential storj-sim edits (#1086)

* Closing context doesn't stop storage node (#1084)

* Print when cancelled

* Close properly

* Don't log nil

* Don't print error when closing dashboard

* Fix panic in inspector if ping fails (#1088)

* Consolidate identity management to identity cli commands (#1083)

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
2019-01-18 08:54:08 -05:00
Egon Elbre
f8906ce000
Change overlay cache database interface (#1047) 2019-01-15 11:08:45 -05:00
Egon Elbre
d9b9ae6ffa
Cleanup overlay methods and names. (#914) 2018-12-20 15:57:54 +02:00
Natalie Villasana
17c60c1f06
moves node selection config setup from uplink to satellite (#891) 2018-12-17 16:05:05 -05:00
Maximillian von Briesen
6c655d117b
Filter nodes by reputation and IP address (node selection) (#720)
*   Pulls statdb stats into overlay cache whenever cache.Put() is called
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to filter based on node stats
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to exclude duplicate IP addresses
2018-12-04 15:18:26 -05:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Egon Elbre
0ae05cf834
Ensure that we only choose storage nodes (#732) 2018-11-29 16:57:00 +02:00
Maximillian von Briesen
b884ee4626
Update protos to fit style guide (easy review) (#702)
* fix overlay.proto and piecestore.proto to meet style guide

* update code to be consistent with regenerated protos
2018-11-23 21:46:53 -05:00
Dennis Coyle
e7e0d1daaa
Coyle/disable mock (#627)
* disabled mock overlay from captplanet
2018-11-20 11:54:52 -05:00
Jennifer Li Johnson
93c5f385a8
Enable checker in captplanet and staging (#643)
* enable checker

* add option to use mock overlay in checker

* adds logs to checker

* appease linter
2018-11-20 10:54:22 -05:00
Egon Elbre
76af9f5171
Add overlay server to testplanet (#677) 2018-11-19 22:39:25 +02:00
Jennifer Li Johnson
7de84a0a40
adds MaxNodes and Start to FindStorageNodes (#569)
* adds MaxNodes and Start params to FindStorageNodesRequest

* use Gets on protos
2018-11-02 14:50:28 -04:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Cameron
f5c115596e filter out excluded nodes when repairing (#426)
* add filter field into OverlayOptions message

* chooseFiltered method, add excluded parameter in populate method

* change excluded type to []dht.NodeID in ChooseFiltered, change comment

* change name filter to excluded_nodes in proto

* implement helper function contains

* delete ChooseFiltered and add its functionality into Choose method to keep original author's history, add excluded argument into Choose calls

* regenerate mock_client.go

* regenerate protobuf

* update test case to use new IDFromString function

* remove old kademlia test code
2018-10-15 13:42:36 -04:00
Jennifer Li Johnson
6fb13896fb
Method to identify injured segments to repair (#398)
* creates checker

* tests offline nodes

* test id injured segs:

* Adds healthy pieces to injured segment struct

* changes inequality

* creates common files

* adds checker benchmarking

* creates more common files

* Replaces pointedb direct db with api call to a new iterate method on pointerdb

* move monkit

* removes identifyrequest proto

* remove healthypieces

* adds benchmarking

creates common file for datarepair

* recreates proto file

* api key on ctx
2018-10-09 12:09:33 -04:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Jennifer Li Johnson
d0f87f0de1
Overlay Bulk Lookup (#279)
* implements bulk node lookup from overlay cache
2018-09-11 00:52:14 -04:00
Egon Elbre
0a2839b2d7
Add IsPrefix support to piecestore (#313) 2018-09-07 17:20:15 +03:00
Jennifer Li Johnson
154a1eab3a
implements node pkg query method (#251) 2018-08-23 11:20:11 -04:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
Jennifer Li Johnson
5f8142cdcc
Integrating on disk routing table with kademlia (#166)
* adds comment

* runs deps

* adds print statements for debugging add node bkad

* more print statements

* removes bkad from routing and integrates on disk routing table

tests failing :(

wip

* removes testbootstrap

* kademlia_test not working

* adds kad tests back in

* Adds skips for tests broken due to wip kademlia
2018-08-09 15:20:39 -04:00
Dennis Coyle
7348bf1e09
Adjustments to how find storage nodes handles limit/amout requested (#192)
* adjusted based on feedback from @jtolds and @kaloyan-raev

* generated protobuf

* fix crash

* WIP

* cleanup

* v >>found

* pointer

* test

* copyrights + cleanup

* fix tests
2018-08-03 09:15:52 -04:00
Dennis Coyle
5de5428d3a
First pass at node restrictions (#146)
* WIP First pass at node restrictions

* adjustments to storage clients

* fix cover

* undo previous import change

* adding copyright

* fix the tests

* check for keys

* moar fixes to tests

* linter

* change how we handle restrictions

* add generated pb.go file

* PR comments addressed

* MockKeyValueStore

* pr comments addressed

* missing )

* past my bedtime

* moar merge issues

* cleanup
2018-08-01 10:15:38 -04:00
Natalie Villasana
80727ae90b adds netstate pagination (#95)
* adds netstate rpc server pagination, mocks pagination in test/util.go

* updates ns client example, combines ns client and server test to netstate_test, adds pagination to bolt client

* better organizes netstate test calls

* wip breaking netstate test into smaller tests

* wip modularizing netstate tests

* adds some test panics

* wip netstate test attempts

* testing bug in netstate TestDeleteAuth

* wip fixes global variable problem, still issues with list

* wip fixes get request params and args

* fixes bug in path when using MakePointers helper fn

* updates mockdb list func, adds test, changes Limit to int

* fixes merge conflicts

* fixes broken tests from merge

* remove unnecessary PointerEntry struct

* removes error when Get returns nil value from boltdb

* breaks boltdb client tests into smaller tests

* renames AssertNoErr test helper to HandleErr

* adds StartingKey and Limit parameters to redis list func, adds beginning of redis tests

* adds helper func for mockdb List function

* if no starting key provided for netstate List, the first value in storage will be used

* adds basic pagination for redis List function, adds tests

* adds list limit to call in overlay/server.go

* streamlines/fixes some nits from review

* removes use of obsolete EncryptedUnencryptedSize

* uses MockKeyValueStore instead of redis instance in redis client test

* changes test to expect nil returned for getting missing key

* remove error from `KeyValueStore#Get`

* fix bolt test

* Merge pull request #1 from bryanchriswhite/nat-pagination

remove error from `KeyValueStore#Get`

* adds Get returning error back to KeyValueStore interface and affected clients

* trying to appease travis: returns errors in Get calls in overlay/cache and cache_test

* handles redis get error when no key found
2018-06-29 16:06:25 -04:00
Dennis Coyle
a5ff5016c9
pkg/kademlia tests and restructuring (#97)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* add changes from kad demo

* PR comments addresses

* go func

* force travis build

* fixed merge conflicts

* fixed merge conflicts

* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests

* linter issues

* linting issues

* fixed merge conflicts

* linter is stupid
2018-06-22 09:33:57 -04:00
Dennis Coyle
9e40141cbc
Coyle/kad client (#91)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Restructure + additional interface

* Add NewOverlayClient

* added test for interface

* PR comments addressed

* lint issue

* added generated protobuf

* fixed merge conflicts

* adding comments

* PR comments addressed
2018-06-19 10:00:15 -04:00