Michal Niewrzal
eb2d858ffc
Use pointerdb.Service instead of pointerdb.Server ( #1099 )
2019-01-19 13:58:53 -05:00
Egon Elbre
78dc02b758
Satellite Peer ( #1034 )
...
* add satellite peer
* Add overlay
* reorganize kademlia
* add RunRefresh
* add refresh to storagenode.Peer
* add discovery
* add agreements and metainfo
* rename
* add datarepair checker
* add repair
* add todo notes for audit
* add testing interface
* add into testplanet
* fixes
* fix compilation errors
* fix compilation errors
* make testplanet run
* remove audit refrences
* ensure that audit tests run
* dev
* checker tests compilable
* fix discovery
* fix compilation
* fix
* fix
* dev
* fix
* disable auth
* fixes
* revert go.mod/sum
* fix linter errors
* fix
* fix copyright
* Add address param for SN dashboard (#1076 )
* Rename storj-sdk to storj-sim (#1078 )
* Storagenode logs and config improvements (#1075 )
* Add more info to SN logs
* remove config-dir from user config
* add output where config was stored
* add message for successful connection
* fix linter
* remove storage.path from user config
* resolve config path
* move success message to info
* log improvements
* Remove captplanet (#1070 )
* pkg/server: include production cert (#1082 )
Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792
* Generate Payments Report (#1079 )
* memory.Size: autoformat sizes based on value entropy (#1081 )
* Jj/bytes (#1085 )
* run tally and rollup
* sets dev default tally and rollup intervals
* nonessential storj-sim edits (#1086 )
* Closing context doesn't stop storage node (#1084 )
* Print when cancelled
* Close properly
* Don't log nil
* Don't print error when closing dashboard
* Fix panic in inspector if ping fails (#1088 )
* Consolidate identity management to identity cli commands (#1083 )
* Consolidate identity management:
Move identity cretaion/signing out of storagenode setup command.
* fixes
* linters
* Consolidate identity management:
Move identity cretaion/signing out of storagenode setup command.
* fixes
* sava backups before saving signed certs
* add "-prebuilt-test-cmds" test flag
* linters
* prepare cli tests for travis
* linter fixes
* more fixes
* linter gods
* sp/sdk/sim
* remove ca.difficulty
* remove unused difficulty
* return setup to its rightful place
* wip travis
* Revert "wip travis"
This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.
* typo in travis.yaml
* remove tests
* remove more
* make it only create one identity at a time for consistency
* add config-dir for consitency
* add identity creation to storj-sim
* add flags
* simplify
* fix nolint and compile
* prevent overwrite and pass difficulty, concurrency, and parent creds
* goimports
2019-01-18 08:54:08 -05:00
Bill Thorp
342dc857f5
rollup query ( #1056 )
...
* implemention notes
* more notes
* starting rollup query
* not working yet
* fixed build
* fixed cfg bug
* change context cancelled errs to debugs
* using byte hours for at rest tally
* revert changes to go.mod
* comment fixes
* prevent double recording tallies in rollup
* linting
* stop leaking dbx
* nodeid changes
* fix build
2019-01-16 14:30:33 -05:00
Alexander Leitner
bfde515391
Clean up Storage node setup ( #1013 )
...
* Edit config on Setup
* Default to 1TiB storage space and 500GiB bandwidth
* Use human readable formats
* Use memory
* units of 1024 are measured with KiB/MiB etc
* pkg/cfgstruct: allow values to be configured with human readable sizes
Change-Id: Ic4e9ae461516d1d26fb81f6e44c5ac5cfccf777f
* Modify tests
* Removed comments
* More merge conflict stuff resolved
* Fix lint
* test fixin
Change-Id: I3a008206bf03a4446da19f642a2f9c1f9acaae36
* Remove commented code but secretly leave it in the histroy forever
* Move flag definition to struct
2019-01-14 16:19:15 -05:00
Egon Elbre
0ca03b41e2
Use generated locking database implementation ( #947 )
2019-01-02 19:53:27 +02:00
Michal Niewrzal
b712fbcbb0
Fix 'empty queue' error when satellite starts ( #939 )
2019-01-02 17:00:32 +01:00
Egon Elbre
4346cd060f
Implement mutex around satellitedb ( #932 )
2018-12-27 11:56:25 +02:00
Cameron
f70b826fd4
repair queue masterDB support ( #865 )
...
* add injuredsegment model to satellitedb.dbx
* add context to queue.RepairQueue interface
* use queue.RepairQueue interface, use masterdb
2018-12-21 10:11:19 -05:00
Bryan White
398379b149
CSR service (part 1): authorizations ( #906 )
2018-12-20 19:29:05 +01:00
Egon Elbre
d9b9ae6ffa
Cleanup overlay methods and names. ( #914 )
2018-12-20 15:57:54 +02:00
Maximillian von Briesen
f18c38628e
Refactor statdb ( #887 )
...
Update statdb args/return values to minimize structs
Simplify statdb.Update() to update all stats instead of an arbitrary subset determined by flags
Remove CreateIfNotExists logic from statdb.Update()
Simplify audit code structure
2018-12-19 13:44:03 -05:00
Natalie Villasana
17c60c1f06
moves node selection config setup from uplink to satellite ( #891 )
2018-12-17 16:05:05 -05:00
aligeti
5e1b02ca8b
Statdb master db v3 848 ( #830 )
...
* intial changes to migrate statdb to masterdb framework
* statdb refactor compiles
* added TestCreateDoesNotExist testcase
* Initial port of statdb to masterdb framework working
* refactored statdb proto def to pkg/statdb
* removed statdb/proto folder
* moved pb.Node to storj.NodeID
* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID
* moved the fields from pb.Node to statdb.UpdateRequest
ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
2018-12-14 15:17:30 -05:00
Michal Niewrzal
db5a3e0d0a
Parameters cleanup ( #864 )
...
* Parameters cleanup
* goimports
2018-12-14 16:55:45 +01:00
Kaloyan Raev
2a8fef8062
Data repairer should use the redundancy strategy from segment's pointer ( #838 )
2018-12-13 09:12:36 +02:00
Jennifer Li Johnson
6642f97142
Jj/accounting updates ( #820 )
...
* tallies up data stored on each node in pointerdb
* adds comments for data type enums
* changes Open to BeginTx because Go convention
* removes online status check from identify active nodes
* changes identifyactivenodes to calculatestaticdata
* updates accounting dbx names
2018-12-12 16:24:08 -05:00
Egon Elbre
c56307f5b9
datarepair: remove unneeded debug message ( #839 )
...
* datarepair: remove unneeded debug message
* Make overlay-cache and routing table logs more specific
2018-12-12 17:39:16 +02:00
JT Olio
ceb590fa67
capt: reduce nodes to 10 ( #793 )
...
Change-Id: Ief380fe29e3043657705cd7505c266fd774181a4
2018-12-11 11:40:54 -07:00
Natalie Villasana
9e1ec97b31
adds node selection config ( #782 )
2018-12-11 12:30:14 -05:00
Michal Niewrzal
7e2f66ee24
Small fixes around master db ( #822 )
2018-12-11 10:30:09 +01:00
aligeti
d27bcddaab
Merge the irreparable DB into the master database ( #777 )
...
* initial refactoring to merge irreparabledb to master db
* refactor
* code review comment updates
2018-12-10 14:08:45 -05:00
Egon Elbre
c7cf97cc23
Fix sqlite registration ( #806 )
2018-12-10 10:32:49 +02:00
JT Olio
1c96db01ba
better database error handling ( #784 )
...
* better database error handling
Change-Id: I28dbd69cf6c2fa268e02405521ff6e6c1a68a702
* missing comments added
* missing comment added
2018-12-07 07:46:42 -07:00
Jennifer Li Johnson
dac79fd762
removes check for mock overlay in data repair checker config ( #794 )
2018-12-07 00:42:16 -05:00
Alexander Leitner
04ff6b8e11
Use audit results in checker ( #754 )
...
* Use statdb to get audit results
* Find invalid nodes
2018-12-06 13:51:23 -05:00
Egon Elbre
fc8e9c6572
Move irreparabledb under datarepair ( #764 )
2018-12-06 14:32:21 +02:00
aligeti
9f6e5d6731
Keep track of 'irreparable' segments in a database ( #686 )
...
* initial irreparable development
* added the Open, Insert, Update, Get, Delete, Close support
* added the Unit test cases
* code review changes
2018-12-04 11:26:30 -05:00
Egon Elbre
9ac3517432
Reduce noise in logging ( #733 )
2018-11-29 22:59:26 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type ( #693 )
...
* preparing for use of `customtype` gogo extension with `NodeID` type
* review changes
* preparing for use of `customtype` gogo extension with `NodeID` type
* review changes
* wip
* tests passing
* wip fixing tests
* more wip test fixing
* remove NodeIDList from proto files
* linter fixes
* linter fixes
* linter/review fixes
* more freaking linter fixes
* omg just kill me - linterrrrrrrr
* travis linter, i will muder you and your family in your sleep
* goimports everything - burn in hell travis
* goimports update
* go mod tidy
2018-11-29 19:39:27 +01:00
Egon Elbre
0ae05cf834
Ensure that we only choose storage nodes ( #732 )
2018-11-29 16:57:00 +02:00
Michal Niewrzal
376bd74bed
Stops logging repairer errors for empty queue ( #717 )
2018-11-27 16:57:51 +01:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf ( #692 )
...
* swap golang/protobuf out for gogo/protobuf
* revert go.mod and go.sum changes
* linter fixes
2018-11-20 19:29:07 +01:00
Jennifer Li Johnson
93c5f385a8
Enable checker in captplanet and staging ( #643 )
...
* enable checker
* add option to use mock overlay in checker
* adds logs to checker
* appease linter
2018-11-20 10:54:22 -05:00
aligeti
7958994ae2
Diagnostic tool to inspect repair queue ( #656 )
...
* initial repair queue diag tool development
* fixes linter warnings
* code review updates
2018-11-16 08:31:33 -05:00
Jennifer Li Johnson
e678e52229
Creates Accounting Pkg to tally at rest node storage ( #568 )
...
* creates accounting package with tally service
* adds cancel on context
* test online nodes
2018-11-08 11:18:28 -05:00
Bill Thorp
07ed38c930
Added distqueue interface and redis and test queue ( #555 )
2018-11-08 08:53:27 -05:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients ( #574 )
...
* wip
* linter fixes
* linter fixes
* test fixes
* linter fixes
* fix merge + restructure piecestore packages
* review feedback
* linter fixes
* linter fixes
* remove unnecessary aliases to piecestore
* more merge fixing
2018-11-06 18:49:17 +01:00
Cameron
de46a999bc
Integrate SegmentStore Repair method with repair service ( #582 )
...
* add storeConfig struct and getSegmentStore helper for creating a segment store
* implement segment store in repairer, remove unnecessary repairer Repair method
* change repair method parameter from int to int32 to match type being passed in
* implement repairer service in captplanet
* rework Config, set Config defaults in captplanet/setup
2018-11-06 09:52:11 -05:00
Egon Elbre
2a8b681c4d
Run repairer and checker early ( #565 )
...
* Run repairers, checker, auditors first time they run to detect potential setup problems.
* Fix error handling in audit.Service
2018-11-01 16:03:45 +02:00
Dennis Coyle
a3becb8a7b
Add repairer & checker to Satelite ( #561 )
...
* Added repairer & checker to Satellite
* fixed repairer and checker configs
2018-10-31 12:22:35 -04:00
Jennifer Li Johnson
7ae2fa3575
moves bulk of code from ticker case to outside for indentation's sake ( #559 )
...
* moves bulk of code from ticker case to outside for indentation's sake
* adds whitespace
* removes break
2018-10-30 16:14:15 -04:00
Jennifer Li Johnson
1fb96689b8
creates run loop for data repair checker ( #490 )
...
* creates run loop for data repair checker
* moves actual checking and repairing under ticker case
* fixes mismatched queueaddrs
2018-10-30 15:16:40 -04:00
Cameron
f7828e73ea
remove ctx from repairer struct ( #535 )
2018-10-25 14:59:36 -04:00
Egon Elbre
8efb4f0e89
Fix repairing run ( #523 )
...
* Fix repairing run
* Fix concurrency bugs
* Add sync2.Limiter concurrency primitive
2018-10-24 15:35:59 +03:00
Alexander Leitner
3e1b16ea99
Start redis ( #470 )
...
* Start miniredis, repairer, and checker with captplanet
2018-10-12 14:04:16 -04:00
Jennifer Li Johnson
0e7f6358fb
creates configs for data repair package ( #463 )
...
* creates configs
2018-10-12 13:49:49 -04:00
Egon Elbre
db42772b4d
Enable goimports and fix formatting. ( #459 )
2018-10-11 23:25:54 +03:00
Jennifer Li Johnson
ea6fc3c532
use utils.combinederrors ( #461 )
2018-10-11 14:42:32 -04:00
Michal Niewrzal
351dc523d6
Satellite signature generation ( #453 )
...
* Satellite signature generation
* Add unit test
* remove unused var
* remove base64 encoding from signature generation
2018-10-11 16:35:55 +02:00
Jennifer Li Johnson
6fb13896fb
Method to identify injured segments to repair ( #398 )
...
* creates checker
* tests offline nodes
* test id injured segs:
* Adds healthy pieces to injured segment struct
* changes inequality
* creates common files
* adds checker benchmarking
* creates more common files
* Replaces pointedb direct db with api call to a new iterate method on pointerdb
* move monkit
* removes identifyrequest proto
* remove healthypieces
* adds benchmarking
creates common file for datarepair
* recreates proto file
* api key on ctx
2018-10-09 12:09:33 -04:00
