Commit Graph

80 Commits

Author SHA1 Message Date
Jess G
3c9d83dbfe
convert psserver dashboard into an inspector (#1407)
* Convert psserver dashboard into an inspector

* remove dashboard stream, update ps.pb.mock

* fixes for lint errs
2019-03-05 15:48:37 -05:00
Kaloyan Raev
54f68347ad
Use DialNode for requesting node info (#1394) 2019-03-02 09:34:08 +02:00
Dylan Lott
bacd683879
Adds DumpNodes method to Kademlia Inspector (#1328)
* Wiring up DumpNodes response for Inspector

* Finalize everything and test that it works

* remove changes to overlay inspector

* Replace list with iterate method in DumpNodes

* generate proto

* Merge in nodeinfo changes

* Update iterate function in DumpNodes

* Fix import order
2019-02-28 12:55:27 -07:00
Michal Niewrzal
4122c98cb7
Validate piece hash on satellite (#1359)
The satellite receives pieces signed hashes in Pointer. If signed hash cannot be validated then piece is removed from Pointer and not saved in DB.
2019-02-28 15:14:54 +01:00
Kaloyan Raev
1ec17653d4 Endpoint for local node info (#1355)
Adds a new `Info` method to the Kademlia endpoint that returns the following local node info:
* ID
* Type
* Metadata (email and wallet)
* Restrictions (free storage and bandwidth)

The new endpoint is exposed as `inspector kad node-info` command too.
2019-02-25 19:41:51 +01:00
Michal Niewrzal
6186b3f90a
Storage node hash calculation on upload (#1347)
Storage node is calculating hash of uploaded data and send it back to uplink with signature
2019-02-23 11:46:07 +01:00
Bill Thorp
373b301736
BWA aliases (#1333)
aliased RBAs and PBAs
2019-02-22 16:17:35 -05:00
Jennifer Li Johnson
b5447c6608
Routing table tests (#1270)
This PR includes a new package called testrouting, which implements a very algorithmically slow but hopefully easy-to-keep-operationally-correct in-memory routing table. The routing table also supports writing out its current structure as a DOT graph for visualization. testrouting is primarily meant to help in coming up with generic routing table integration tests.

This PR also includes a new routing table integration test suite that runs against all current routing table implementations. Our existing routing table passes a lot of the tests, but not all of them, still debugging why. I have confirmed the tests should pass with the visualization graphs though.
2019-02-22 13:39:29 -05:00
Natalie Villasana
c3d3f41d30 removes some SignedMessage use (#1258)
Removes most instances of pb.SignedMessage (there's more to take out but they shouldn't hurt anyone as is).

There used to be places in psserver where a PieceID was hmac'd with the SatelliteID, which was gotten from a SignedMessage. This PR makes it so some functions access the SatelliteID from the Payer Bandwidth Allocation instead.

This requires passing a SatelliteID into psserver functions where they weren't before, so the following proto messages have been changed:

 * PieceId - satellite_id field added
   This is so the psserver.Piece function has access to the SatelliteID when it needs to get the namespaced pieceID.
   This proto message should probably be renamed to PieceRequest, or a new PieceRequest message should be created so this isn't misnamed.

 * PieceDelete - satellite_id field added
   This is so the psserver.Delete function has access to the SatelliteID when receiving a request to Delete.
2019-02-19 23:36:08 -06:00
Bill Thorp
2c5716e874
pb.Equal fix (#1304)
* pb.Equal fix

* file capitalization

* oh snake case
2019-02-13 09:08:57 -05:00
Alexander Leitner
7c741c16d1
Readable sizes in dashboard (#1180)
* Readable sizes

* Use tabwriter

* Added more addresses to dashboard stats

* Use kad to look up addresses

* Move dashboard command to dashboard.go
2019-01-30 14:49:25 -05:00
Natalie Villasana
8d7944bcf8
new node selection in overlay (#1136) 2019-01-29 14:42:43 -05:00
Dylan Lott
bb2588e9ae
Refactors kademlia inspector to use FindNear (#1157)
* WIP

* update identity paths to get dump-nodes working

* Remove commented out code

* remove comment

* updates from code reviews

* updates linter
2019-01-28 15:53:37 -07:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's (#1152)
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Bill Thorp
8ba4b61e59
Make Bandwidth Agreements Secure / Trustable (#1117)
* Added cert chains and nodeid verification to bandwidth agreement
2019-01-25 13:05:21 -05:00
Jennifer Li Johnson
5397efe5c1
removes standalone payments service (#1134)
* remove standalone payments service - will integrate within satellite directly

* regenerate proto files
2019-01-25 11:23:25 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
70940b8710
Making sure payments works (#1122)
* byte hours

* updates comment

* removes payment test data methods

* wip

* adds tally and rollup to peer

* remove sim test for payment (for now)

* lint error
2019-01-24 13:44:14 -05:00
Jennifer Li Johnson
e6fbf63620
Generate Payments Report (#1079) 2019-01-17 13:34:13 -05:00
Dylan Lott
82b9b54695
storagenode dashboard is working and passing tests locally (#1072)
* storagenode dashboard is working and passing tests locally

* linter fixes

* linter fixes

* moar linter fixes god

* review fixes and updates

* linter fixes
2019-01-16 16:09:13 -05:00
JT Olio
87925789de pkg/inspector: split functionality to be per-service (#1029)
now kad inspector features exist on every server that has
kademlia running. likewise, overlay and statdb.

this means kad inspection features are now available on
storage nodes

Change-Id: I343c873552341de13302bfb7a5d79cccc84fc6b8
2019-01-14 11:47:22 -07:00
Bill Thorp
273c3924ca
continue agreement sender loop on error (#1020)
* continue agreement sender loop on error
* added REJECTED grpc status
2019-01-11 13:15:49 -05:00
Bill Thorp
fb4a11ebb0
Differentiate GET and PUT operations in the tally service (#965)
* draft of new bandwidth agreement types
* updated storagenode report for new types
* use correct pba types
2019-01-10 06:41:57 -05:00
JT Olio
f8ee5f88e5 cmd/bootstrap: a kademlia-only server (#1001) 2019-01-09 17:59:51 +02:00
Egon Elbre
756f49fea1
remove discovery endpoint (#1006) 2019-01-09 17:18:42 +02:00
aligeti
ce1db97ce6
fix manipulated paychecks V3-1023 (#961)
Satellite crashing on receiving a manipulated bandwidthagreement V3-1022
2019-01-04 11:26:26 -05:00
Jennifer Li Johnson
a2fa5c4c5a Proper NodeType Handling (#873)
* adds enums to nodetype

* updating nodetype todos

* ran pb updates

* reorder nodetypes

* adding checks

* wip

* wip

* wip

* bug in test-captplanet

* wip

* add values to storagenode, satellite, captplanet binaries

* Cleanup

* more cleanup

* wip

* lint

* lint

* wip

* fixes bug

* regenerate protos

Change-Id: Id270212e8c7479e52641058042cf23b5317ab773

* limit node type changes to kademlia

Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4

* dpanic

Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e

* review comments

Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
2019-01-02 11:47:34 -07:00
Bryan White
249244536a
CSR Service (part 2): cert signing rpc (#950)
* CSR Service:

+ implement certificate sign rpc method
+ implement certificate signer client/server
+ refactor `AuthorizationDB#Create`
+ refactor `NewTestIdentity`
+ add `AuthorizationDB#Claim`
+ add `Token#Equal`
+ fix `Authorizations#Marshal` when marshaling identities and certificates
+ tweak `Authorization#String` format
+ cert debugging improvements (jsondiff)
+ receive context arg in `NewTestIdentity`
+ misc. fixes
2019-01-02 12:39:17 -05:00
Bryan White
4efb5c0a75
Authorization improvements (#925) 2018-12-31 10:45:43 -05:00
Bryan White
398379b149
CSR service (part 1): authorizations (#906) 2018-12-20 19:29:05 +01:00
Natalie Villasana
d97e27ed80
removes a few todos (#867) 2018-12-17 13:11:29 -05:00
Michal Niewrzal
1d2523cf60
Payments API protobuf definition (#876) 2018-12-17 18:36:50 +01:00
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830)
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
2018-12-14 15:17:30 -05:00
Dylan Lott
9bdee7b106
Discovery service (#805)
* WIP possible discovery service impl

* Adds discovery service to CaptPlanet

* Updates the config and server for discovery service

* updates testplanet to use discovery package

* update satellite imports

* Removes unnecessary cache test

* linter fixes

* adds discovery startup to captplanet

* invoke refresh

* updates to discovery refresh cycle

* Make implementation more consistent with previous implementation

* add wait before trying to upload

* sleep a bit more

* remove kademlia bootstrap

* updates

* remove comments
2018-12-14 12:23:54 -07:00
Michal Niewrzal
725ed44ce0
Add gen.go for protobuf (#863) 2018-12-14 16:55:55 +01:00
Jennifer Li Johnson
6642f97142
Jj/accounting updates (#820)
* tallies up data stored on each node in pointerdb

* adds comments for data type enums

* changes Open to BeginTx because Go convention

* removes online status check from identify active nodes

* changes identifyactivenodes to calculatestaticdata

* updates accounting dbx names
2018-12-12 16:24:08 -05:00
Egon Elbre
7a80e7bf2f
Use testplanet in node tests (#841) 2018-12-12 17:40:33 +02:00
Dylan Lott
c6d790d58e
Improvements to Kad Lookups (#731)
* adds channel for getting node out of lookup

* WIP adding the channels to lookups

* WIP adding channel to node lookups

* Wires up FindNodes method with channels

* WIP adds a test suite for lookup - tests are still failing

* WIP wires up use of testplanet for kademlia lookup tests

* WIP merging in node id changes

* Merges in pkg/storj node type changes

* Tests passing

* Lookup node working via Inspector now

* updates

* WIP working on getting tests passing

* WIP getting tests passing

* FindNode works

* Linter fix

* Adds copyrights to lookup_test

* removes a fmt.Printf I missed

* Removes commented out lines
2018-12-04 14:39:28 -07:00
Maximillian von Briesen
6c655d117b
Filter nodes by reputation and IP address (node selection) (#720)
*   Pulls statdb stats into overlay cache whenever cache.Put() is called
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to filter based on node stats
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to exclude duplicate IP addresses
2018-12-04 15:18:26 -05:00
Maximillian von Briesen
2ab15196d9
Get rid of statdb grpc stuff (#761)
* remove sdbclient

* do not register statdb grpc server/remove apikey from statdb

* start audit service after statdb in captplanet
2018-12-04 13:47:58 -05:00
Bryan White
120d875e06
consolidate NodeRep and NodeStats (#744)
* consolidate `NodeRep` and `NodeStats`

* regenerate

* review fixes

* go travis
2018-12-03 14:23:12 +01:00
Egon Elbre
3e461df470
Linting protobuf files (#676) 2018-11-30 17:02:01 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Dylan Lott
a245aa7827
Inspector - Kad PING (#722)
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier

* Wiring up PING in kad inspector tools
2018-11-27 15:50:52 -07:00
Dylan Lott
fd42795456
updates dbx/gen.go to name generated package correctly (#723) 2018-11-27 13:52:19 -07:00
Maximillian von Briesen
7cf16503b8
Add statdb to overlay cache (node selection) (#711)
* make overlay node rep consistent with statdb node rep

* add statdb to testplanet

* add statdb to overlay cache

* tests
2018-11-27 12:46:12 -05:00
Michal Niewrzal
8dd669b37f
Bandwidth allocations need method designation (GET/PUT) (#708)
* Bandwidth allocations need method designation (GET/PUT)

* add method comment

* goimports
2018-11-26 19:21:44 +01:00
Maximillian von Briesen
b884ee4626
Update protos to fit style guide (easy review) (#702)
* fix overlay.proto and piecestore.proto to meet style guide

* update code to be consistent with regenerated protos
2018-11-23 21:46:53 -05:00
Maximillian von Briesen
06cf1c71ed
Statdb inspection tool (#626)
* begin adding stadb inspection tool

* add command for creating statdb nodes from csv

* add getCSVStats and create helper func for getting sdb client

* fix createcsvstats, add createstats

* fix linter errors
2018-11-21 17:01:09 -05:00
Dylan Lott
0d05cb26bf
Kademlia Inspector CLI (#657)
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier
2018-11-21 10:31:27 -07:00