Commit Graph

95 Commits

Author SHA1 Message Date
Egon Elbre
2d00f882a3
Kademlia Dialer (#1092) 2019-01-18 10:00:56 -05:00
Egon Elbre
78dc02b758 Satellite Peer (#1034)
* add satellite peer

* Add overlay

* reorganize kademlia

* add RunRefresh

* add refresh to storagenode.Peer

* add discovery

* add agreements and metainfo

* rename

* add datarepair checker

* add repair

* add todo notes for audit

* add testing interface

* add into testplanet

* fixes

* fix compilation errors

* fix compilation errors

* make testplanet run

* remove audit refrences

* ensure that audit tests run

* dev

* checker tests compilable

* fix discovery

* fix compilation

* fix

* fix

* dev

* fix

* disable auth

* fixes

* revert go.mod/sum

* fix linter errors

* fix

* fix copyright

* Add address param for SN dashboard (#1076)

* Rename storj-sdk to storj-sim (#1078)

* Storagenode logs and config improvements  (#1075)

* Add more info to SN logs

* remove config-dir from user config

* add output where config was stored

* add message for successful connection

* fix linter

* remove storage.path from user config

* resolve config path

* move success  message to info

* log improvements

* Remove captplanet (#1070)

* pkg/server: include production cert (#1082)

Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792

* Generate Payments Report (#1079)

* memory.Size: autoformat sizes based on value entropy (#1081)

* Jj/bytes (#1085)

* run tally and rollup

* sets dev default tally and rollup intervals

* nonessential storj-sim edits (#1086)

* Closing context doesn't stop storage node (#1084)

* Print when cancelled

* Close properly

* Don't log nil

* Don't print error when closing dashboard

* Fix panic in inspector if ping fails (#1088)

* Consolidate identity management to identity cli commands (#1083)

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
2019-01-18 08:54:08 -05:00
Egon Elbre
bbf81f2479 Consolidate identity management to identity cli commands (#1083)
* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
2019-01-18 11:36:58 +01:00
JT Olio
a255e385df memory.Size: autoformat sizes based on value entropy (#1081) 2019-01-17 20:47:57 +02:00
Natalie Villasana
a7d3a8ded7
adds base 10 memory sizes (#1061) 2019-01-15 17:17:37 -05:00
Egon Elbre
f8906ce000
Change overlay cache database interface (#1047) 2019-01-15 11:08:45 -05:00
Alexander Leitner
bfde515391
Clean up Storage node setup (#1013)
* Edit config on Setup

* Default to 1TiB storage space and 500GiB bandwidth

* Use human readable formats

* Use memory

* units of 1024 are measured with KiB/MiB etc

* pkg/cfgstruct: allow values to be configured with human readable sizes

Change-Id: Ic4e9ae461516d1d26fb81f6e44c5ac5cfccf777f

* Modify tests

* Removed comments

* More merge conflict stuff resolved

* Fix lint

* test fixin

Change-Id: I3a008206bf03a4446da19f642a2f9c1f9acaae36

* Remove commented code but secretly leave it in the histroy forever

* Move flag definition to struct
2019-01-14 16:19:15 -05:00
Kaloyan Raev
bde0f09c15
Add Tee and cancellable Copy to sync2 pkg (#1022) 2019-01-11 17:35:26 +02:00
Egon Elbre
8893884044
convert piecestorage into a struct (#1024) 2019-01-11 13:26:39 +02:00
Egon Elbre
34125fe614 Reduce test CA difficulty (#1010)
* reduce test CA difficulty

* reduce docker ca difficulty
2019-01-10 15:23:33 +01:00
Egon Elbre
eb69ecadec
Storage Node Peer (#1005) 2019-01-10 15:13:27 +02:00
Egon Elbre
db5a990719
lint imports grouping (#993) 2019-01-08 16:05:14 +02:00
Egon Elbre
953d77e9af
sync2: add Fence and Sleep (#983) 2019-01-07 21:00:40 +02:00
Jennifer Li Johnson
a2fa5c4c5a Proper NodeType Handling (#873)
* adds enums to nodetype

* updating nodetype todos

* ran pb updates

* reorder nodetypes

* adding checks

* wip

* wip

* wip

* bug in test-captplanet

* wip

* add values to storagenode, satellite, captplanet binaries

* Cleanup

* more cleanup

* wip

* lint

* lint

* wip

* fixes bug

* regenerate protos

Change-Id: Id270212e8c7479e52641058042cf23b5317ab773

* limit node type changes to kademlia

Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4

* dpanic

Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e

* review comments

Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
2019-01-02 11:47:34 -07:00
Egon Elbre
89db0fe9f5
storj-sdk base code (#799) 2019-01-02 20:07:49 +02:00
Bryan White
249244536a
CSR Service (part 2): cert signing rpc (#950)
* CSR Service:

+ implement certificate sign rpc method
+ implement certificate signer client/server
+ refactor `AuthorizationDB#Create`
+ refactor `NewTestIdentity`
+ add `AuthorizationDB#Claim`
+ add `Token#Equal`
+ fix `Authorizations#Marshal` when marshaling identities and certificates
+ tweak `Authorization#String` format
+ cert debugging improvements (jsondiff)
+ receive context arg in `NewTestIdentity`
+ misc. fixes
2019-01-02 12:39:17 -05:00
Bill Thorp
e4b3dab446
Add config editing via the CLI for storage nodes (#919)
* added editor and updated storagenode cmd
* switched to custom yaml gen
2019-01-02 10:12:07 -05:00
Egon Elbre
4346cd060f
Implement mutex around satellitedb (#932) 2018-12-27 11:56:25 +02:00
Dylan Lott
7569b7d71b
Node lifecycle hooks (#920)
* wires up first draft of lifecycle methods

* creates interface on transport

* node lifecycle hooks works

* linter fixes

* adds error log at connection success

* chnages Observer interface to use context

* Makes Discovery take its own logger

* WIP

* linter fixes

* Test fixes

* adds in ConnFailure code for cache
2018-12-21 21:51:42 -07:00
Dylan Lott
2f6835a3be
Adds a logger to RoutingTable (#922)
* adds delete functions and tests for cache

* adds logger to routing table struct and passes it through to logger

* add loggers
2018-12-21 17:48:53 -07:00
Bill Thorp
3fa094f3da
Add the ability for Kademlia to do a refresh for each bucket (#908)
* added random id in range, added refresh
2018-12-20 16:45:06 -05:00
Egon Elbre
1feef7105b
Implement memory and file pipe (#894) 2018-12-20 16:51:39 +02:00
Egon Elbre
d9b9ae6ffa
Cleanup overlay methods and names. (#914) 2018-12-20 15:57:54 +02:00
Egon Elbre
d9a13667fc
Use fixed logger for Kademlia and make not being able to connect in discovery debug statement. (#899) 2018-12-18 17:13:32 +02:00
Natalie Villasana
17c60c1f06
moves node selection config setup from uplink to satellite (#891) 2018-12-17 16:05:05 -05:00
Cameron
0d17c21a1b
Add logger to piecestore server/node started log (#856)
* add logger to psserver struct

* node started log

* rename initialize to NewEndpoint

* return err from WriteBandwidthAllocToDB
2018-12-17 10:23:02 -05:00
Egon Elbre
d8df4b5f6b
Simplify provider tests (#862) 2018-12-17 17:09:52 +02:00
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830)
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
2018-12-14 15:17:30 -05:00
Dylan Lott
9bdee7b106
Discovery service (#805)
* WIP possible discovery service impl

* Adds discovery service to CaptPlanet

* Updates the config and server for discovery service

* updates testplanet to use discovery package

* update satellite imports

* Removes unnecessary cache test

* linter fixes

* adds discovery startup to captplanet

* invoke refresh

* updates to discovery refresh cycle

* Make implementation more consistent with previous implementation

* add wait before trying to upload

* sleep a bit more

* remove kademlia bootstrap

* updates

* remove comments
2018-12-14 12:23:54 -07:00
Bryan White
2016ce9fd6
Certificate revocation (#836)
* wip certificate revocation

* refactor tests

* wip testing

* testing

* review fixes

* integration fix attempt #1

* review fixes

* integration fix attempt #2

* linter fixes

* add copywrite

* integration fix attemp #3

* more testing

* more tests

* go mod tidy

* review fixes

* linter fixes
2018-12-13 21:01:43 +01:00
Egon Elbre
969fcabf72
Remove utils.LogClose from identity saving (#837) 2018-12-13 15:50:05 +02:00
Egon Elbre
7a80e7bf2f
Use testplanet in node tests (#841) 2018-12-12 17:40:33 +02:00
Kaloyan Raev
c025c13706
Fix deadlock on testplanet shutdown (#798) 2018-12-07 20:31:59 +02:00
Bryan White
228aa34ff6
TLS extension processing (#771) 2018-12-07 14:44:25 +01:00
Maximillian von Briesen
6c655d117b
Filter nodes by reputation and IP address (node selection) (#720)
*   Pulls statdb stats into overlay cache whenever cache.Put() is called
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to filter based on node stats
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to exclude duplicate IP addresses
2018-12-04 15:18:26 -05:00
Maximillian von Briesen
2ab15196d9
Get rid of statdb grpc stuff (#761)
* remove sdbclient

* do not register statdb grpc server/remove apikey from statdb

* start audit service after statdb in captplanet
2018-12-04 13:47:58 -05:00
Michal Niewrzal
06c196d28c
Allow config generation into empty directory (#752)
* Allow config generation into empty directory

* unifed messages

* move main method to fpath

* fix linter problems
2018-12-04 12:42:16 +01:00
Cameron
226cf48c41
move clients to internal/s3api (#749)
* move clients to internal/s3client

* move Config and Client into common.go
2018-12-03 14:50:05 -05:00
Michal Niewrzal
1206ef9126
Use OS related default setup directory (#741)
* Use OS related default setup directory

* added missing comment

* adjust test-captplanet.sh

* ApplicationDir moved to fpath
2018-12-03 16:51:56 +01:00
Kaloyan Raev
66476da1ae
metainfo: basic upload (#726) 2018-11-30 15:50:52 +02:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Egon Elbre
0ae05cf834
Ensure that we only choose storage nodes (#732) 2018-11-29 16:57:00 +02:00
Maximillian von Briesen
7cf16503b8
Add statdb to overlay cache (node selection) (#711)
* make overlay node rep consistent with statdb node rep

* add statdb to testplanet

* add statdb to overlay cache

* tests
2018-11-27 12:46:12 -05:00
Egon Elbre
ed86059829
Add stack-trace and remove testcase with testing (#716) 2018-11-27 17:50:40 +02:00
Dennis Coyle
e7e0d1daaa
Coyle/disable mock (#627)
* disabled mock overlay from captplanet
2018-11-20 11:54:52 -05:00
Egon Elbre
7d70842d53
Detect goroutines running for too long in testcontext (#685) 2018-11-20 11:32:18 +02:00
Egon Elbre
76af9f5171
Add overlay server to testplanet (#677) 2018-11-19 22:39:25 +02:00
Egon Elbre
d07433c150
Use testplanet to test Refresh (#678) 2018-11-19 16:40:01 +02:00
Egon Elbre
c4b90f84dd
Use better defaults and naming for postgres database (#659) 2018-11-15 20:36:57 +02:00