Commit Graph

36 Commits

Author SHA1 Message Date
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830)
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
2018-12-14 15:17:30 -05:00
Bill Thorp
ba5f71810e
added tally to capt run.go (#751)
* added tally to capt run.go, ParseURL -> SplitDBURL
2018-12-12 08:15:34 -05:00
JT Olio
1c96db01ba
better database error handling (#784)
* better database error handling

Change-Id: I28dbd69cf6c2fa268e02405521ff6e6c1a68a702

* missing comments added

* missing comment added
2018-12-07 07:46:42 -07:00
Maximillian von Briesen
828a8b6907
Change statdb.FindValidNodes to FindInvalidNodes (#774)
* Remove statdb.FindValidNodes
* Add statdb.FindInvalidNodes for getting nodes that fall below a reputation threshold
2018-12-05 15:23:32 -05:00
Dylan Lott
c6d790d58e
Improvements to Kad Lookups (#731)
* adds channel for getting node out of lookup

* WIP adding the channels to lookups

* WIP adding channel to node lookups

* Wires up FindNodes method with channels

* WIP adds a test suite for lookup - tests are still failing

* WIP wires up use of testplanet for kademlia lookup tests

* WIP merging in node id changes

* Merges in pkg/storj node type changes

* Tests passing

* Lookup node working via Inspector now

* updates

* WIP working on getting tests passing

* WIP getting tests passing

* FindNode works

* Linter fix

* Adds copyrights to lookup_test

* removes a fmt.Printf I missed

* Removes commented out lines
2018-12-04 14:39:28 -07:00
Maximillian von Briesen
6c655d117b
Filter nodes by reputation and IP address (node selection) (#720)
*   Pulls statdb stats into overlay cache whenever cache.Put() is called
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to filter based on node stats
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to exclude duplicate IP addresses
2018-12-04 15:18:26 -05:00
Maximillian von Briesen
2ab15196d9
Get rid of statdb grpc stuff (#761)
* remove sdbclient

* do not register statdb grpc server/remove apikey from statdb

* start audit service after statdb in captplanet
2018-12-04 13:47:58 -05:00
Maximillian von Briesen
c6dd70541e
move sdb apikey to config struct (#753) 2018-12-03 13:40:03 -05:00
Bryan White
120d875e06
consolidate NodeRep and NodeStats (#744)
* consolidate `NodeRep` and `NodeStats`

* regenerate

* review fixes

* go travis
2018-12-03 14:23:12 +01:00
Egon Elbre
3e461df470
Linting protobuf files (#676) 2018-11-30 17:02:01 +02:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Egon Elbre
9ac3517432
Reduce noise in logging (#733) 2018-11-29 22:59:26 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Bill Thorp
b8b0d75cc6
made accounting use migrate (#688)
* made accounting use migrate
2018-11-26 16:49:55 -05:00
Maximillian von Briesen
5014a785a0
Node selection 0/4: Update statdb auth/functionality (#698)
* remove api key from statdb server reqs; add statdb UpdateUptime and UpdateAuditSuccess to server

* update api key authentication in statdb server

* add todos for future statdb updates

* add UpdateUptime and UpdateAuditSuccess to statdb server

* fix apikey stuff in config.go and statdb_test.go

* fix tests

* update sdbclient.NewClient call in audit package

* fix UpdateUptime and UpdateAuditSuccess in sdbclient

* set api key from statdb/config.go

* change package for statdb tests

* linter fixes

* remove todo comments

* fix sdbclient err checking

* move validate auth functionality to auth package

* update description for statdb api key

* remove import
2018-11-26 12:08:29 -05:00
Maximillian von Briesen
06cf1c71ed
Statdb inspection tool (#626)
* begin adding stadb inspection tool

* add command for creating statdb nodes from csv

* add getCSVStats and create helper func for getting sdb client

* fix createcsvstats, add createstats

* fix linter errors
2018-11-21 17:01:09 -05:00
Jennifer Li Johnson
93c5f385a8
Enable checker in captplanet and staging (#643)
* enable checker

* add option to use mock overlay in checker

* adds logs to checker

* appease linter
2018-11-20 10:54:22 -05:00
Maximillian von Briesen
2c36dcc2c8
Update statdb (#650)
* add sdbclient.UpdateUptime; update args for sdbclient.CreateEntryIfNotExists

* add auditcount to node stats; restructure statdb.CreateEntryIfNotExists

* add noop mock sdbclient

* add the ability to create a node in statdb without "default" stats

* update statdb.CreateEntryIfNotExists

* take fewer args for sdbclient.CreateWithStats/FindValidNodes

* add sdbclient.UpdateAuditSuccess

* update sdbclient.Update so that all fields are updated when called (reduce args)

* update error checking in statdb.Create
2018-11-14 19:03:19 -05:00
Egon Elbre
6066ae7567
Check schemas while creating tables (#609) 2018-11-10 00:15:35 +02:00
Maximillian von Briesen
741a6bc8c3
update dbx generation script for statdb (#602) 2018-11-09 16:01:51 -05:00
Egon Elbre
6d401a4351
Check for go.mod validity (#605) 2018-11-09 15:32:35 +02:00
Natalie Villasana
3b572264ca
adds audit job (#521) 2018-11-07 01:16:43 +00:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574)
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
2018-11-06 18:49:17 +01:00
Maximillian von Briesen
40b210f604
Fix statdb query to avoid sql injection (#562)
* fix statdb query to avoid sql injection

* use byte array instead of string for node ids in statdb
2018-10-31 12:18:51 -04:00
Maximillian von Briesen
0606b38d68
Add FindValidNodes() to statdb (#530)
* add FindValidNodes() to statdb and update overlay node selection options

* add logic for FindValidNodes

* add FindValidNodes test

* fix findvalidnodes query

* fix linter errors

* clean up comments
2018-10-30 13:11:22 -04:00
Natalie Villasana
1687111cca
adds Reporter to the audit package to record failed audits (#467) 2018-10-16 13:40:34 -04:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Maximillian von Briesen
81b106fb72 Add StatDB to Satellite (#396) 2018-10-08 18:15:54 -04:00
Dennis Coyle
dee2c137c8
Remove BKAD dependency from pkg/kademlia (#294)
* slowly but surely

* hardcode ID for tests so we can get predictable results

* skipping bad test

* removing tests of bkad

* wip

* new algorithm for worker

* clean up

* remove skipped test

* changes

* uncomment

* fixed conflicts

* maybe done ?

* cleanup

* boot bkad

* wip

* cleanup

* undo change

* fixes

* wip

* wip

* moving nodeID around

* wip

* wip

* fixes

* fixes after merge

* added TODO

* fixed tests post identity

* linter fixes

* wip

* PR review comments

* wip

* fixing tests

* fix tests

* force db directory

* bad test

* fixes race condition

* small cleanups

* adding db folder

* testing

* wip

* cleanup

* cleanup

* linters

* export Restrict

* add timeout

* testing

* linters

* forgot one

* moar fixes from master merge

* PR comments

* moar PR comments

* removed stun flag

* remove duplicate declaration

* remove old tests

* remove timeout

* fix tests

* missed one

* changed StringToID >> IDFromString

* PR comments

* stupid linter

* moevd overlay mock

* fixed merge conflicts

* fixes

* linter
2018-10-08 11:09:37 -04:00
Dylan Lott
ae38817fa3
Pointer lookup (#407)
* Loads cache from context for PointerDB access

* WIP adds overlay lookups to pointerdb requests

* Pointer lookup code is added for Get

* adds feature flag for pointerdb return

* refactors pointerdb code

* removes some unnecessary debug logs

* Fixes indent in config

* adds early return for non-remote pointers

* formats code, removes some comments

* Fixes tests broken by pointer proto changes

* adds error check and merges variable declaration

* removes commented out proto import

* adds error check to pdbclient
2018-10-04 15:00:19 -06:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Egon Elbre
7a906285e3 Small cleanup at top-level (#336)
* cleanup some top-level packages

* remove unused index.html

* update indirect tags
2018-09-10 07:23:08 -07:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
aligeti
09da23737a
rename farmers to storagenode (#275)
* rename farmers to storagenode

* review changes

* merge conflicts
2018-08-24 21:52:58 -04:00
Jennifer Li Johnson
3230762041
Kademlia Routing Table (#164)
* adds comment

* runs deps

* creates boltdb kademlia routing table

* protobuf updates

* adds reverselist to mockkeyvaluestore interface

* xor wip

* xor wip

* fixes xor sort

* runs go fmt

* fixes

* goimports again

* trying to fix travis tests

* fixes mock tests
2018-07-30 15:25:18 -04:00
Maximillian von Briesen
36c394c295
StatDB (#144)
* add statdb proto and example client

* server logic

* update readme

* remove boltdb from service.go

* sqlite3

* add statdb server executable file

* create statdb node table if it does not exist already

* get UpdateBatch working

* update based on jt review

* remove some commented lines

* fix linting issues

* reformat

* apiKey -> APIKey

* update statdb client apiKey->APIKey
2018-07-27 17:11:44 -04:00