Commit Graph

39 Commits

Author SHA1 Message Date
Kaloyan Raev
964c87c476 Fix checks around repair threshold (#2246) 2019-06-19 22:13:11 +02:00
Kaloyan Raev
ebd9b375fc
Repair should not corrupt files (#2194) 2019-06-14 12:16:31 +03:00
ethanadams
8f2dca8437 Re-enabling and fixing repairer tests (#2099)
* Disabled discovery service by changiing from Stop() to Pause()

Paused to solve race condition.  If discovery is running, it may mark a node "up" after they've been manually marked "down" in this test.

* Extend to the repair timeout

Fixes intermittent test failures when repairs were taking more than 2 seconds.

* Re-enabled test. Disabled discovery service by changiing from Stop() to Pause()

* Changed back to Stop.

* Revert "Changed back to Stop."

This reverts commit 46d410e72dfae63e0c44915be42784cc9a7b5abf.

* re-enabling TestIdentifyInjuredSegments

* Changed Pause to Stop.  Commented on timeout change

* testing...

* temporarily skipping audit tests

* changing back to discover Stop for testing via jenkins

* Revert "changing back to discover Stop for testing via jenkins"

This reverts commit 6aa8558b11a0053c30e0c8b2dbf0d6c0cb34ee6c.

* Changing back to Stop().  Depends on PR 2137

* Revert "temporarily skipping audit tests"

This reverts commit 1940ed9b315d663a0eb6c95521780cbcb48cb121.

* Removed reference to Graveyard since its been removed
2019-06-10 09:06:21 +02:00
JT Olio
f1641af802 storage: add monkit task to missing places (#2122)
* storage: add monkit task to missing places

Change-Id: I9e17a6b14f7c25bbf698eeecf32785e9add3f26e

* fix tests

Change-Id: Id078276fa3de61a28eb3d01d4e751732ecbb173f

* import order

Change-Id: I814e33755b9f10b5219af37cd828cd75eb3da1a4

* remove part of other commit

Change-Id: Idaa4c95cd65e97567fb466de49718db8203cfbe1
2019-06-05 16:23:10 +02:00
aligeti
934ebf9cbf
Added the irreparable repair functionality (#1955)
* Added the irreparable repair functionality
2019-05-30 11:18:20 -04:00
Bill Thorp
4002ed4463 unskip TestIdentifyIrreparableSegments (#1927) 2019-05-09 15:55:34 +03:00
Natalie Villasana
b48f584cea
repair checker resumes iterating where left off (#1879) 2019-05-08 13:59:50 -04:00
Michal Niewrzal
fe3dfc1587
Move pointerdb.Service to satellite (#1826) 2019-04-25 10:46:32 +02:00
Bill Thorp
17a227e6e9
refactor injuredsegments db so that we can't have duplicates (#1717)
made repairqueue not use a true queue, forbid duplicates
2019-04-16 14:14:09 -04:00
JT Olio
ffdb2e7728
actually skip the data repair test (#1728)
Change-Id: I76286fc6cc5129d8be50d45a684a3e0dce9c0cc6
2019-04-09 23:29:05 -06:00
Egon Elbre
94e79eda6d
remove overlay endpoint (#1521) 2019-03-23 10:06:11 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Cameron
c7ffbe1c28
Add ability to view irreparable segments on satellite (#1448)
* define irreparable inspector protobuf

* add IrreparableDB method GetLimited

* fill out irreparable inspector API

* add IrreparableInspector server to satellite, fix small error

* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
2019-03-15 16:21:52 -04:00
Bill Thorp
9b580c5fb6 Repair checker is checking the same 1000 elements all the time (#1297)
* removed limit on repair, now using cycle

* added BatchIteratorOptions

* consolidated boltdb common.go

* PR feedback cleanup
2019-02-14 13:33:41 +01:00
Michal Niewrzal
b2f9453184
Disable Checker subsystem in tests (#1279)
* Disable Checker subsystem in tests

* rename field

* remove sleeps and errgroup.Group
2019-02-11 22:06:39 +01:00
Michal Niewrzal
aa3836c06a
Add checker irreparable test (#1246)
* Add checker irreparable test

* use RepairUnixSec

* wait for second check

* use storagenode.ID()

* Update comments
2019-02-06 14:03:38 +01:00
Egon Elbre
39c1e5ccec
Ensure checker tests run against postgres (#1229) 2019-02-05 18:00:52 +02:00
Michal Niewrzal
14434594c4
Re-enable repair checker tests (#1172)
* Re-enable repair checker tests

* simpler NodeID construction

* adjust benchmark
2019-01-30 19:34:44 +01:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's (#1152)
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Egon Elbre
78dc02b758 Satellite Peer (#1034)
* add satellite peer

* Add overlay

* reorganize kademlia

* add RunRefresh

* add refresh to storagenode.Peer

* add discovery

* add agreements and metainfo

* rename

* add datarepair checker

* add repair

* add todo notes for audit

* add testing interface

* add into testplanet

* fixes

* fix compilation errors

* fix compilation errors

* make testplanet run

* remove audit refrences

* ensure that audit tests run

* dev

* checker tests compilable

* fix discovery

* fix compilation

* fix

* fix

* dev

* fix

* disable auth

* fixes

* revert go.mod/sum

* fix linter errors

* fix

* fix copyright

* Add address param for SN dashboard (#1076)

* Rename storj-sdk to storj-sim (#1078)

* Storagenode logs and config improvements  (#1075)

* Add more info to SN logs

* remove config-dir from user config

* add output where config was stored

* add message for successful connection

* fix linter

* remove storage.path from user config

* resolve config path

* move success  message to info

* log improvements

* Remove captplanet (#1070)

* pkg/server: include production cert (#1082)

Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792

* Generate Payments Report (#1079)

* memory.Size: autoformat sizes based on value entropy (#1081)

* Jj/bytes (#1085)

* run tally and rollup

* sets dev default tally and rollup intervals

* nonessential storj-sim edits (#1086)

* Closing context doesn't stop storage node (#1084)

* Print when cancelled

* Close properly

* Don't log nil

* Don't print error when closing dashboard

* Fix panic in inspector if ping fails (#1088)

* Consolidate identity management to identity cli commands (#1083)

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
2019-01-18 08:54:08 -05:00
Egon Elbre
4346cd060f
Implement mutex around satellitedb (#932) 2018-12-27 11:56:25 +02:00
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830)
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
2018-12-14 15:17:30 -05:00
Michal Niewrzal
7e2f66ee24
Small fixes around master db (#822) 2018-12-11 10:30:09 +01:00
aligeti
d27bcddaab
Merge the irreparable DB into the master database (#777)
* initial refactoring to merge irreparabledb to master db

* refactor

* code review comment updates
2018-12-10 14:08:45 -05:00
Alexander Leitner
04ff6b8e11
Use audit results in checker (#754)
* Use statdb to get audit results

* Find invalid nodes
2018-12-06 13:51:23 -05:00
Egon Elbre
fc8e9c6572
Move irreparabledb under datarepair (#764) 2018-12-06 14:32:21 +02:00
aligeti
9f6e5d6731
Keep track of 'irreparable' segments in a database (#686)
* initial irreparable development

* added the Open, Insert, Update, Get, Delete, Close  support

* added the Unit test cases

* code review changes
2018-12-04 11:26:30 -05:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Egon Elbre
0ae05cf834
Ensure that we only choose storage nodes (#732) 2018-11-29 16:57:00 +02:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf (#692)
* swap golang/protobuf out for gogo/protobuf

* revert go.mod and go.sum changes

* linter fixes
2018-11-20 19:29:07 +01:00
Jennifer Li Johnson
e678e52229
Creates Accounting Pkg to tally at rest node storage (#568)
* creates accounting package with tally service

* adds cancel on context

* test online nodes
2018-11-08 11:18:28 -05:00
Bill Thorp
07ed38c930
Added distqueue interface and redis and test queue (#555) 2018-11-08 08:53:27 -05:00
Jennifer Li Johnson
1fb96689b8
creates run loop for data repair checker (#490)
* creates run loop for data repair checker

* moves actual checking and repairing under ticker case

* fixes mismatched queueaddrs
2018-10-30 15:16:40 -04:00
Jennifer Li Johnson
0e7f6358fb
creates configs for data repair package (#463)
* creates configs
2018-10-12 13:49:49 -04:00
Egon Elbre
db42772b4d
Enable goimports and fix formatting. (#459) 2018-10-11 23:25:54 +03:00
Michal Niewrzal
351dc523d6
Satellite signature generation (#453)
* Satellite signature generation

* Add unit test

* remove unused var

* remove base64 encoding from signature generation
2018-10-11 16:35:55 +02:00
Jennifer Li Johnson
6fb13896fb
Method to identify injured segments to repair (#398)
* creates checker

* tests offline nodes

* test id injured segs:

* Adds healthy pieces to injured segment struct

* changes inequality

* creates common files

* adds checker benchmarking

* creates more common files

* Replaces pointedb direct db with api call to a new iterate method on pointerdb

* move monkit

* removes identifyrequest proto

* remove healthypieces

* adds benchmarking

creates common file for datarepair

* recreates proto file

* api key on ctx
2018-10-09 12:09:33 -04:00
Cameron
027e4045c6
setup repairer loop (#378)
* setup repairer loop

* added read from queue

* Refactor to make things easier to import

* add more control flow to repairer

* add comment

* basic interval structure for running check/repair

* change function name GetNext to Dequeue

* better increment/decrement syntax

* export Repairer struct

* delete 'unreachable code'

* add mon.Task() to Repairer.Repair

* remove 24 hour interval

* set maxRepair on Config as well as Repairer

* add comment for Repairer struct, check err

* comment out runCfg.Repair in cmd/satellite/main.go because it is NI yet
2018-10-02 15:46:29 -04:00