Commit Graph

345 Commits

Author SHA1 Message Date
Yehor Butko
5352778b3c
refactored byte to uuid converting (#610)
* refactored byte to uuid converting

* linter fixed

* error messages updated
2018-11-12 09:14:16 +00:00
Egon Elbre
6066ae7567
Check schemas while creating tables (#609) 2018-11-10 00:15:35 +02:00
Dennis Coyle
e4baec8fce
Coyle/tls verify (#612)
* add nodeID to dial opts

* fix tests to use real identity
2018-11-09 17:08:33 -05:00
Maximillian von Briesen
741a6bc8c3
update dbx generation script for statdb (#602) 2018-11-09 16:01:51 -05:00
Jennifer Li Johnson
ad4df839a8
Data at rest accounting database (#601)
* starts at rest storage db

* generates aggregate and granular models with dbx

* reverts gomod

* reverts gosum
2018-11-09 15:02:28 -05:00
Dylan Lott
c77ca2b161
[WIP] MVP of Kad inspection tools (#596)
* WIP starting to wire up the kademlia CLI tool

* WIP wiring up kad cli tools

* WIP starting to wire up the kademlia CLI tool

* WIP wiring up kad cli tools

* Got everything wired up

* WIP starting to wire up the kademlia CLI tool

* WIP wiring up kad cli tools

* merge in upstream

* WIP wiring up kad cli tools

* Got everything wired up

* WIP trying to get CLI to connect

* Inspector connects to overlay now

* Some refactoring

* Linter fixes

* Linter fixes

* Switch to pkg/process instead of using rootCmd.Execute
2018-11-09 11:40:15 -07:00
Egon Elbre
6d401a4351
Check for go.mod validity (#605) 2018-11-09 15:32:35 +02:00
Yehor Butko
7dcbba2541
V3-665 Creating Companies repository (#606) 2018-11-09 14:05:24 +02:00
Natalie Villasana
f11f4653e4
removes audit erroring when no stripe is given (#603) 2018-11-08 22:36:44 +00:00
Michal Niewrzal
45d4234749 Logs cleanup (uplink, storage node) (#600)
* Logs cleanup (uplink, storage node)

* fix unit tests

* more logs cleanups
2018-11-08 15:27:07 -05:00
Jennifer Li Johnson
e678e52229
Creates Accounting Pkg to tally at rest node storage (#568)
* creates accounting package with tally service

* adds cancel on context

* test online nodes
2018-11-08 11:18:28 -05:00
Alexander Leitner
320c93368a
Add creation time to Payer Bandwidth Allocation (#581)
* Add creation time to Payer Bandwidth Allocation

* goimports
2018-11-08 11:04:52 -05:00
Yehor Butko
54cd9491c6
V3-664 Extend User repository. Reimplemented with dbx (#591)
* V3-664 Extend User repository. Reimplemented with dbx

* structure updated

* redundant packages removed, structure simplified

* fixing goimports

* removed constuctor from user struct

* separated types declarations from database file

* test file renamed

* fixes according to review

* fixing goimports

* fixing goimports
2018-11-08 16:19:42 +02:00
Bill Thorp
07ed38c930
Added distqueue interface and redis and test queue (#555) 2018-11-08 08:53:27 -05:00
Kaloyan Raev
f62397e91d
Don't encrypt empty content for buckets and 0-byte objects (#583) 2018-11-08 15:22:38 +02:00
aligeti
285b70048f
Diagnostic tool for Satellite (#586)
* Satellite diagnostic tool initial development

* changes for code review and lint warning fixes
2018-11-08 08:20:23 -05:00
littleskunk
810835e6a9
pretty print logging (#598) 2018-11-08 11:28:30 +01:00
Egon Elbre
cebebb1e07
Add redundancy share information (#589) 2018-11-08 10:45:48 +02:00
Kaloyan Raev
3501d983ec
Don't count current channel when testing for slowness (#584) 2018-11-08 09:40:07 +02:00
Maximillian von Briesen
66ee332446
Fix audit stuff breaking test-captplanet (#594)
* fix audit stripe selector to work if last segment is smaller than stripe size

* fix audit bug related to indexing an incomplete list of nodes returned by overlay
2018-11-07 16:23:05 -05:00
Egon Elbre
d5b3d5f395
Don't panic in case of a 0-size pointer (#590) 2018-11-07 18:51:36 +02:00
Kaloyan Raev
18f295cbab
Allow S3 clients to reupload stale parts (#588) 2018-11-07 17:13:40 +02:00
Natalie Villasana
3b572264ca
adds audit job (#521) 2018-11-07 01:16:43 +00:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574)
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
2018-11-06 18:49:17 +01:00
Michal Niewrzal
deb015970d
Use Nodes array from pdb.Get (#578)
* Use Nodes array from pdb.Get

* fix problems with captplanet tests

* better comments
2018-11-06 18:03:11 +01:00
Cameron
de46a999bc
Integrate SegmentStore Repair method with repair service (#582)
* add storeConfig struct and getSegmentStore helper for creating a segment store

* implement segment store in repairer, remove unnecessary repairer Repair method

* change repair method parameter from int to int32 to match type being passed in

* implement repairer service in captplanet

* rework Config, set Config defaults in captplanet/setup
2018-11-06 09:52:11 -05:00
Egon Elbre
eb5e46e336
metainfo readonly stream (#580) 2018-11-06 13:40:06 +02:00
aligeti
a760e48d4d
Satellite bw usage v3 121 (#547)
* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Linter errors

* Read agreements from PSDB

* Try writing message to server

* Cleanup

* Basic functionality

* Better error handelling

* Fix test

* setup config and server structure for receiving bandwidth agreements

* Resolve linter issues

* Optional commit for if we want to handle deletes all at once

* add identity to Server, add logic for receiving bandwidth messsages

* Bandwidth agreement DBX creation and integration with bw agreement endpoint

Co-authored-by: Kishore <kishore@storj.io>
Co-authored-by: Cam <cameron@storj.io>

* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Linter errors

* Read agreements from PSDB

* Try writing message to server

* Cleanup

* Basic functionality

* Better error handelling

* Fix test

* setup config and server structure for receiving bandwidth agreements

* Resolve linter issues

* Optional commit for if we want to handle deletes all at once

* add identity to Server, add logic for receiving bandwidth messsages

* Bandwidth agreement DBX creation and integration with bw agreement endpoint

Co-authored-by: Kishore <kishore@storj.io>
Co-authored-by: Cam <cameron@storj.io>

* added postgres create/read/delete test function

Co-authored-by: kishore <kishore@storj.io
Co-authored-by: cam <cameron@storj.io>

* edit comment

* removed sqlite3 driver from dbx

* remove generated sqlite code, add dbx read limitoffset

* remove getServerAndDB function, rename getDBPath to getPSQLInfo

* WIP writing server endpoint test

* code review changes
2018-11-05 10:23:54 -05:00
Michal Niewrzal
1129ead6ef
Fix 'message not signed by the satellite' problem (#560)
* Fix 'message not signed by the satellite' problem

* fixed build

* fix errors
2018-11-05 16:12:19 +01:00
Egon Elbre
c765dd98de
internal/testplanet (#528) 2018-11-03 14:17:14 +02:00
Jennifer Li Johnson
7de84a0a40
adds MaxNodes and Start to FindStorageNodes (#569)
* adds MaxNodes and Start params to FindStorageNodesRequest

* use Gets on protos
2018-11-02 14:50:28 -04:00
Egon Elbre
fe36d5cd7d
Optimize sortByXOR (#571) 2018-11-02 18:46:59 +02:00
aligeti
771c7e6ef9
download with specific number of nodes V3-353 (#564)
* download with specific number of nodes

* changes per code review comments
2018-11-02 11:22:01 -04:00
Egon Elbre
830241d845
Remove debug logging for routing table (#572) 2018-11-02 16:55:14 +02:00
Bryan White
57572cdeed
Use sync.Cond with kademlia workers (#512)
* wip - have to take a break; crying baby

* linter fixes

* bugfix

* responding to review feedback

* linter fixes?

* linter fixes

* feedback fixes

* feedback fixes

* linter fixes

* linter fixes

* linter fixes
2018-11-01 18:03:46 +01:00
Cameron
856c3a779f
Agreement Receiver service (#546)
* setup config and server structure for receiving bandwidth agreements

* add error channel
2018-11-01 12:51:19 -04:00
Alexander Leitner
67306d296b Bandwidth fix (#567)
* Don't scan into nil
2018-11-01 12:40:26 -04:00
Bryan White
8b9711cb5e
better waitlist-gating (#557)
* better waitlist-gating

(cherry picked from commit 490fe02b7c3558da18678dfb651c92ec9c4a75b5)

* fix broken test

* linter fixes

* linter fixes

* make extension verification optional

* add certifcate gating script for captplanet

* fixing tests

* linter fixes

* linter fixes?

* moar linter fixes

* Revert "moar linter fixes"

This reverts commit 8139ccbd73cbbead987b7667567844f50f7df2c8.

* just kill me

* refactor

* refactor tests

* liniter...

* cleanup
2018-11-01 16:48:43 +01:00
Egon Elbre
2a8b681c4d
Run repairer and checker early (#565)
* Run repairers, checker, auditors first time they run to detect potential setup problems.
* Fix error handling in audit.Service
2018-11-01 16:03:45 +02:00
Alexander Leitner
db80358f32
bandwidth agreements protobuf (#505)
* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Renamed payer to satellite in psdb
2018-10-31 14:47:25 -04:00
Dennis Coyle
a3becb8a7b
Add repairer & checker to Satelite (#561)
* Added repairer & checker to Satellite

* fixed repairer and checker configs
2018-10-31 12:22:35 -04:00
Maximillian von Briesen
40b210f604
Fix statdb query to avoid sql injection (#562)
* fix statdb query to avoid sql injection

* use byte array instead of string for node ids in statdb
2018-10-31 12:18:51 -04:00
Jennifer Li Johnson
7ae2fa3575
moves bulk of code from ticker case to outside for indentation's sake (#559)
* moves bulk of code from ticker case to outside for indentation's sake

* adds whitespace

* removes break
2018-10-30 16:14:15 -04:00
Jennifer Li Johnson
1fb96689b8
creates run loop for data repair checker (#490)
* creates run loop for data repair checker

* moves actual checking and repairing under ticker case

* fixes mismatched queueaddrs
2018-10-30 15:16:40 -04:00
Alexander Leitner
7e4149138a
Piecestore proto change (#556)
* add action and storage node id to piecestore.proto. renamed payer and renter to satellite_id and uplink_id
2018-10-30 15:03:41 -04:00
aligeti
7ce94627f1
Implements Data Repair Interfaces in Stream Store (#455)
* add filter field into OverlayOptions message

* chooseFiltered method, add excluded parameter in populate method

* change excluded type to []dht.NodeID in ChooseFiltered, change comment

* change name filter to excluded_nodes in proto

* implement helper function contains

* delete ChooseFiltered and add its functionality into Choose method to keep original author's history, add excluded argument into Choose calls

* regenerate mock_client.go

* regenerate protobuf

* adding the repair() func

* update test case to use new IDFromString function

* modified the repair() and updated streams mock

* modified the repair() and updated streams mock

* Options struct

* adding the repair() func

* modified the repair() and updated streams mock

* modified the repair() and updated streams mock

* integrating the segment repair()

* development repair with hack working

* repair segment changes

* integrated with mini hacks and rigged up test case with dev debug info

* integrated with ec and overlay

* added repair test case

* made the getNewUniqueNodes() to recursively go thru choose() to find get the required number of unique nodes

* cleaned up code
2018-10-30 14:06:12 -04:00
Dennis Coyle
d2933840d2
RPC Ping Request/Response (#544)
* Creates an RPC Ping method on the Node server

* adding Ping method to mockNodeServer implementation

* Remove the line skipping the Ping Test
2018-10-30 14:04:29 -04:00
Maximillian von Briesen
0606b38d68
Add FindValidNodes() to statdb (#530)
* add FindValidNodes() to statdb and update overlay node selection options

* add logic for FindValidNodes

* add FindValidNodes test

* fix findvalidnodes query

* fix linter errors

* clean up comments
2018-10-30 13:11:22 -04:00
Egon Elbre
bd67288071
Add inmemory psdb (#543) 2018-10-30 18:43:09 +02:00
Michal Niewrzal
0d03f2fbb5
Send bandwidth alloc from satellite to storage node (#538)
* Send bandwidth alloc from satellite to storage node

* Remove unecessary nil checks

* set Renter field

* fix tests

* goimports

* Update README.md

* Update README.md

* Update README.md

* Update README.md (#550)

* Skip flaky TestPing (#552)
2018-10-30 17:24:46 +01:00