Commit Graph

442 Commits

Author SHA1 Message Date
Michal Niewrzal
8dd669b37f
Bandwidth allocations need method designation (GET/PUT) (#708)
* Bandwidth allocations need method designation (GET/PUT)

* add method comment

* goimports
2018-11-26 19:21:44 +01:00
Maximillian von Briesen
5014a785a0
Node selection 0/4: Update statdb auth/functionality (#698)
* remove api key from statdb server reqs; add statdb UpdateUptime and UpdateAuditSuccess to server

* update api key authentication in statdb server

* add todos for future statdb updates

* add UpdateUptime and UpdateAuditSuccess to statdb server

* fix apikey stuff in config.go and statdb_test.go

* fix tests

* update sdbclient.NewClient call in audit package

* fix UpdateUptime and UpdateAuditSuccess in sdbclient

* set api key from statdb/config.go

* change package for statdb tests

* linter fixes

* remove todo comments

* fix sdbclient err checking

* move validate auth functionality to auth package

* update description for statdb api key

* remove import
2018-11-26 12:08:29 -05:00
Yehor Butko
1f1f9f5875
Satellitedb company-user relation changed to 1-to-1 (#706)
* Satellitedb company-user relation changed to 1-to-1

* fixing comments
2018-11-26 14:57:55 +00:00
Yehor Butko
a998665491
Satellite projects api (#703)
* V3-645 Create Project API

* fixing linter

* update project refactored
2018-11-26 10:47:23 +00:00
Kaloyan Raev
67db26b36e
Metainfo stream download (#697) 2018-11-26 09:39:05 +02:00
Maximillian von Briesen
b884ee4626
Update protos to fit style guide (easy review) (#702)
* fix overlay.proto and piecestore.proto to meet style guide

* update code to be consistent with regenerated protos
2018-11-23 21:46:53 -05:00
Yaroslav Vorobiov
bdebe36ae0
updated login api (#699) 2018-11-22 12:38:58 +02:00
Maximillian von Briesen
06cf1c71ed
Statdb inspection tool (#626)
* begin adding stadb inspection tool

* add command for creating statdb nodes from csv

* add getCSVStats and create helper func for getting sdb client

* fix createcsvstats, add createstats

* fix linter errors
2018-11-21 17:01:09 -05:00
Dylan Lott
0d05cb26bf
Kademlia Inspector CLI (#657)
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier
2018-11-21 10:31:27 -07:00
Yaroslav Vorobiov
c829835dc1
updated createuser api (#687) 2018-11-21 17:51:43 +02:00
Michal Niewrzal
f72832ee69
Add metadata to pb.Node (email and wallet) (#671)
* Add metadata to pb.Node (email and wallet)

* goimports

* reorganize config

* fix typo

* remove 'foo' id

* add basic test
2018-11-21 16:07:18 +01:00
Kaloyan Raev
aec2c41911
Metainfo remote segment test (#696) 2018-11-21 16:35:53 +02:00
Kaloyan Raev
8c56198090
Use designated error classes for common bucket and object errors (#691) 2018-11-21 13:17:28 +02:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf (#692)
* swap golang/protobuf out for gogo/protobuf

* revert go.mod and go.sum changes

* linter fixes
2018-11-20 19:29:07 +01:00
Bryan White
54ccb460a9
rename size fields in protobuf messages for gogo compatibility (#690)
* rename `size` fields in protobuf messages for gogo compatibility

* linter fixes
2018-11-20 18:09:35 +01:00
Dennis Coyle
e7e0d1daaa
Coyle/disable mock (#627)
* disabled mock overlay from captplanet
2018-11-20 11:54:52 -05:00
Jennifer Li Johnson
93c5f385a8
Enable checker in captplanet and staging (#643)
* enable checker

* add option to use mock overlay in checker

* adds logs to checker

* appease linter
2018-11-20 10:54:22 -05:00
Yehor Butko
4734a7447b
V3-805 Extend satellitedb with Project members entity (#683)
* V3-805 Extend satellitedb with Project members entity

* fixing linter

* small refactoring

* tests refactoring
2018-11-20 14:50:47 +00:00
Egon Elbre
76af9f5171
Add overlay server to testplanet (#677) 2018-11-19 22:39:25 +02:00
Egon Elbre
832317b0ee
Remove unused workers (#640) 2018-11-19 17:07:24 +02:00
Egon Elbre
d07433c150
Use testplanet to test Refresh (#678) 2018-11-19 16:40:01 +02:00
Kaloyan Raev
bb1cf151e3
Metainfo handle inline segments in ReadOnlyStream (#674) 2018-11-19 13:08:28 +02:00
Egon Elbre
6fe16e48ba
Simplify and fix tests (#660) 2018-11-16 18:31:14 +02:00
Yaroslav Vorobiov
3e99861e04
operationName and variables fields added to graphql json parsing (#668) 2018-11-16 16:34:35 +02:00
Kaloyan Raev
f730ce451a
Telemetry in metainfo implementation (#670) 2018-11-16 15:59:27 +02:00
aligeti
7958994ae2
Diagnostic tool to inspect repair queue (#656)
* initial repair queue diag tool development

* fixes linter warnings

* code review updates
2018-11-16 08:31:33 -05:00
Alexander Leitner
5e9377524f
Refactor basystem (#641)
* Signature verification

* Clean up agreement sender to have less errors

* overlay address in captnplanet

* Refactor bandwidth.proto to not use streams

* Make sure the send worked

* Handle connection to satellite

* Save renter public key inside of renter bandwidth allocations

* Default diag to sqlite. Make configurable

* Separate bw server and dbm; regenerate dbx files

* Make sure test uses protobufs

* Demonstrate creating bandwidth allocations
2018-11-15 14:06:09 -05:00
Egon Elbre
c4b90f84dd
Use better defaults and naming for postgres database (#659) 2018-11-15 20:36:57 +02:00
Kaloyan Raev
0357e61bbd
metainfo objects tests (#662) 2018-11-15 17:31:33 +02:00
Yaroslav Vorobiov
92b1bceca7
added copyright to files in satelliteauth pkg (#663) 2018-11-15 14:25:04 +02:00
Yaroslav Vorobiov
3f2ce11315
added copyright to every file in satellite pkg (#661) 2018-11-15 14:00:08 +02:00
Egon Elbre
21a102a05f
testplanet: add logger (#649) 2018-11-15 10:57:47 +02:00
Maximillian von Briesen
2c36dcc2c8
Update statdb (#650)
* add sdbclient.UpdateUptime; update args for sdbclient.CreateEntryIfNotExists

* add auditcount to node stats; restructure statdb.CreateEntryIfNotExists

* add noop mock sdbclient

* add the ability to create a node in statdb without "default" stats

* update statdb.CreateEntryIfNotExists

* take fewer args for sdbclient.CreateWithStats/FindValidNodes

* add sdbclient.UpdateAuditSuccess

* update sdbclient.Update so that all fields are updated when called (reduce args)

* update error checking in statdb.Create
2018-11-14 19:03:19 -05:00
Bill Thorp
d3c347a0ac
Verify overlay cache is accurately and fully tested (#540)
* added intro node id, moved port range to 1024, listen in go routines
2018-11-14 16:30:07 -05:00
Yaroslav Vorobiov
f28af4cdfb
added creation date to graphql user and removed redundant check from graphql api handler (#654) 2018-11-14 17:13:13 +02:00
Yehor Butko
8990fea63c
Satellitedb refactoring (#647) 2018-11-14 12:45:49 +00:00
Yaroslav Vorobiov
c442205b3a
account api: infrastructure, login, register, getUser (#611) 2018-11-14 12:50:15 +02:00
Egon Elbre
17519a7532 metainfo: Fix listing paths (#645) 2018-11-14 12:40:53 +02:00
Kaloyan Raev
5cdb03515f
Configure path encryption in metainfo API (#644) 2018-11-14 11:26:18 +02:00
Jennifer Li Johnson
377832c705
Create Tally and Rollup pkgs in accounting (#642)
* creates separate tally and rollup packages and writes skeleton for rollup

* TODO add rollupDB and rawDB to rollup struct

* TODO add rawDB to tally struct
2018-11-13 20:22:18 -05:00
Alexander Leitner
50d657af11
Cleanup agreement sender (#630)
* Signature verification

* code review comments & lint warnings

* Clean up agreement sender to have less errors

* overlay address in captnplanet
2018-11-13 10:37:49 -05:00
Kaloyan Raev
514ca50611
Configurable encryption type for paths (#637) 2018-11-13 14:21:52 +02:00
Yehor Butko
75213fa633
V3-663 Extend database with Project entity (#622)
* V3-663 Extend database with Project entity

* fixing linter
2018-11-13 10:27:42 +02:00
aligeti
6576d31b29 Signature verification (#604)
* Signature verification

* code review comments & lint warnings
2018-11-12 16:59:30 -05:00
Bill Thorp
ad39201ba9
Issues in overlay rebuild allowed to continue (#625)
* Issues in overlay rebuild allowed to continue
2018-11-12 14:53:20 -05:00
Cameron
ace82bc834
remove bad nodes from originalNodes list when repairing (#597)
* remove bad nodes from list

* create slice for healthy nodes

* add contains helper, clean up loop
2018-11-12 11:10:44 -05:00
Kaloyan Raev
9afa9b6b52
Metainfo buckets tests (#607) 2018-11-12 15:23:19 +02:00
Yehor Butko
5352778b3c
refactored byte to uuid converting (#610)
* refactored byte to uuid converting

* linter fixed

* error messages updated
2018-11-12 09:14:16 +00:00
Egon Elbre
6066ae7567
Check schemas while creating tables (#609) 2018-11-10 00:15:35 +02:00
Dennis Coyle
e4baec8fce
Coyle/tls verify (#612)
* add nodeID to dial opts

* fix tests to use real identity
2018-11-09 17:08:33 -05:00
Maximillian von Briesen
741a6bc8c3
update dbx generation script for statdb (#602) 2018-11-09 16:01:51 -05:00
Jennifer Li Johnson
ad4df839a8
Data at rest accounting database (#601)
* starts at rest storage db

* generates aggregate and granular models with dbx

* reverts gomod

* reverts gosum
2018-11-09 15:02:28 -05:00
Dylan Lott
c77ca2b161
[WIP] MVP of Kad inspection tools (#596)
* WIP starting to wire up the kademlia CLI tool

* WIP wiring up kad cli tools

* WIP starting to wire up the kademlia CLI tool

* WIP wiring up kad cli tools

* Got everything wired up

* WIP starting to wire up the kademlia CLI tool

* WIP wiring up kad cli tools

* merge in upstream

* WIP wiring up kad cli tools

* Got everything wired up

* WIP trying to get CLI to connect

* Inspector connects to overlay now

* Some refactoring

* Linter fixes

* Linter fixes

* Switch to pkg/process instead of using rootCmd.Execute
2018-11-09 11:40:15 -07:00
Egon Elbre
6d401a4351
Check for go.mod validity (#605) 2018-11-09 15:32:35 +02:00
Yehor Butko
7dcbba2541
V3-665 Creating Companies repository (#606) 2018-11-09 14:05:24 +02:00
Natalie Villasana
f11f4653e4
removes audit erroring when no stripe is given (#603) 2018-11-08 22:36:44 +00:00
Michal Niewrzal
45d4234749 Logs cleanup (uplink, storage node) (#600)
* Logs cleanup (uplink, storage node)

* fix unit tests

* more logs cleanups
2018-11-08 15:27:07 -05:00
Jennifer Li Johnson
e678e52229
Creates Accounting Pkg to tally at rest node storage (#568)
* creates accounting package with tally service

* adds cancel on context

* test online nodes
2018-11-08 11:18:28 -05:00
Alexander Leitner
320c93368a
Add creation time to Payer Bandwidth Allocation (#581)
* Add creation time to Payer Bandwidth Allocation

* goimports
2018-11-08 11:04:52 -05:00
Yehor Butko
54cd9491c6
V3-664 Extend User repository. Reimplemented with dbx (#591)
* V3-664 Extend User repository. Reimplemented with dbx

* structure updated

* redundant packages removed, structure simplified

* fixing goimports

* removed constuctor from user struct

* separated types declarations from database file

* test file renamed

* fixes according to review

* fixing goimports

* fixing goimports
2018-11-08 16:19:42 +02:00
Bill Thorp
07ed38c930
Added distqueue interface and redis and test queue (#555) 2018-11-08 08:53:27 -05:00
Kaloyan Raev
f62397e91d
Don't encrypt empty content for buckets and 0-byte objects (#583) 2018-11-08 15:22:38 +02:00
aligeti
285b70048f
Diagnostic tool for Satellite (#586)
* Satellite diagnostic tool initial development

* changes for code review and lint warning fixes
2018-11-08 08:20:23 -05:00
littleskunk
810835e6a9
pretty print logging (#598) 2018-11-08 11:28:30 +01:00
Egon Elbre
cebebb1e07
Add redundancy share information (#589) 2018-11-08 10:45:48 +02:00
Kaloyan Raev
3501d983ec
Don't count current channel when testing for slowness (#584) 2018-11-08 09:40:07 +02:00
Maximillian von Briesen
66ee332446
Fix audit stuff breaking test-captplanet (#594)
* fix audit stripe selector to work if last segment is smaller than stripe size

* fix audit bug related to indexing an incomplete list of nodes returned by overlay
2018-11-07 16:23:05 -05:00
Egon Elbre
d5b3d5f395
Don't panic in case of a 0-size pointer (#590) 2018-11-07 18:51:36 +02:00
Kaloyan Raev
18f295cbab
Allow S3 clients to reupload stale parts (#588) 2018-11-07 17:13:40 +02:00
Natalie Villasana
3b572264ca
adds audit job (#521) 2018-11-07 01:16:43 +00:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574)
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
2018-11-06 18:49:17 +01:00
Michal Niewrzal
deb015970d
Use Nodes array from pdb.Get (#578)
* Use Nodes array from pdb.Get

* fix problems with captplanet tests

* better comments
2018-11-06 18:03:11 +01:00
Cameron
de46a999bc
Integrate SegmentStore Repair method with repair service (#582)
* add storeConfig struct and getSegmentStore helper for creating a segment store

* implement segment store in repairer, remove unnecessary repairer Repair method

* change repair method parameter from int to int32 to match type being passed in

* implement repairer service in captplanet

* rework Config, set Config defaults in captplanet/setup
2018-11-06 09:52:11 -05:00
Egon Elbre
eb5e46e336
metainfo readonly stream (#580) 2018-11-06 13:40:06 +02:00
aligeti
a760e48d4d
Satellite bw usage v3 121 (#547)
* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Linter errors

* Read agreements from PSDB

* Try writing message to server

* Cleanup

* Basic functionality

* Better error handelling

* Fix test

* setup config and server structure for receiving bandwidth agreements

* Resolve linter issues

* Optional commit for if we want to handle deletes all at once

* add identity to Server, add logic for receiving bandwidth messsages

* Bandwidth agreement DBX creation and integration with bw agreement endpoint

Co-authored-by: Kishore <kishore@storj.io>
Co-authored-by: Cam <cameron@storj.io>

* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Linter errors

* Read agreements from PSDB

* Try writing message to server

* Cleanup

* Basic functionality

* Better error handelling

* Fix test

* setup config and server structure for receiving bandwidth agreements

* Resolve linter issues

* Optional commit for if we want to handle deletes all at once

* add identity to Server, add logic for receiving bandwidth messsages

* Bandwidth agreement DBX creation and integration with bw agreement endpoint

Co-authored-by: Kishore <kishore@storj.io>
Co-authored-by: Cam <cameron@storj.io>

* added postgres create/read/delete test function

Co-authored-by: kishore <kishore@storj.io
Co-authored-by: cam <cameron@storj.io>

* edit comment

* removed sqlite3 driver from dbx

* remove generated sqlite code, add dbx read limitoffset

* remove getServerAndDB function, rename getDBPath to getPSQLInfo

* WIP writing server endpoint test

* code review changes
2018-11-05 10:23:54 -05:00
Michal Niewrzal
1129ead6ef
Fix 'message not signed by the satellite' problem (#560)
* Fix 'message not signed by the satellite' problem

* fixed build

* fix errors
2018-11-05 16:12:19 +01:00
Egon Elbre
c765dd98de
internal/testplanet (#528) 2018-11-03 14:17:14 +02:00
Jennifer Li Johnson
7de84a0a40
adds MaxNodes and Start to FindStorageNodes (#569)
* adds MaxNodes and Start params to FindStorageNodesRequest

* use Gets on protos
2018-11-02 14:50:28 -04:00
Egon Elbre
fe36d5cd7d
Optimize sortByXOR (#571) 2018-11-02 18:46:59 +02:00
aligeti
771c7e6ef9
download with specific number of nodes V3-353 (#564)
* download with specific number of nodes

* changes per code review comments
2018-11-02 11:22:01 -04:00
Egon Elbre
830241d845
Remove debug logging for routing table (#572) 2018-11-02 16:55:14 +02:00
Bryan White
57572cdeed
Use sync.Cond with kademlia workers (#512)
* wip - have to take a break; crying baby

* linter fixes

* bugfix

* responding to review feedback

* linter fixes?

* linter fixes

* feedback fixes

* feedback fixes

* linter fixes

* linter fixes

* linter fixes
2018-11-01 18:03:46 +01:00
Cameron
856c3a779f
Agreement Receiver service (#546)
* setup config and server structure for receiving bandwidth agreements

* add error channel
2018-11-01 12:51:19 -04:00
Alexander Leitner
67306d296b Bandwidth fix (#567)
* Don't scan into nil
2018-11-01 12:40:26 -04:00
Bryan White
8b9711cb5e
better waitlist-gating (#557)
* better waitlist-gating

(cherry picked from commit 490fe02b7c3558da18678dfb651c92ec9c4a75b5)

* fix broken test

* linter fixes

* linter fixes

* make extension verification optional

* add certifcate gating script for captplanet

* fixing tests

* linter fixes

* linter fixes?

* moar linter fixes

* Revert "moar linter fixes"

This reverts commit 8139ccbd73cbbead987b7667567844f50f7df2c8.

* just kill me

* refactor

* refactor tests

* liniter...

* cleanup
2018-11-01 16:48:43 +01:00
Egon Elbre
2a8b681c4d
Run repairer and checker early (#565)
* Run repairers, checker, auditors first time they run to detect potential setup problems.
* Fix error handling in audit.Service
2018-11-01 16:03:45 +02:00
Alexander Leitner
db80358f32
bandwidth agreements protobuf (#505)
* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Renamed payer to satellite in psdb
2018-10-31 14:47:25 -04:00
Dennis Coyle
a3becb8a7b
Add repairer & checker to Satelite (#561)
* Added repairer & checker to Satellite

* fixed repairer and checker configs
2018-10-31 12:22:35 -04:00
Maximillian von Briesen
40b210f604
Fix statdb query to avoid sql injection (#562)
* fix statdb query to avoid sql injection

* use byte array instead of string for node ids in statdb
2018-10-31 12:18:51 -04:00
Jennifer Li Johnson
7ae2fa3575
moves bulk of code from ticker case to outside for indentation's sake (#559)
* moves bulk of code from ticker case to outside for indentation's sake

* adds whitespace

* removes break
2018-10-30 16:14:15 -04:00
Jennifer Li Johnson
1fb96689b8
creates run loop for data repair checker (#490)
* creates run loop for data repair checker

* moves actual checking and repairing under ticker case

* fixes mismatched queueaddrs
2018-10-30 15:16:40 -04:00
Alexander Leitner
7e4149138a
Piecestore proto change (#556)
* add action and storage node id to piecestore.proto. renamed payer and renter to satellite_id and uplink_id
2018-10-30 15:03:41 -04:00
aligeti
7ce94627f1
Implements Data Repair Interfaces in Stream Store (#455)
* add filter field into OverlayOptions message

* chooseFiltered method, add excluded parameter in populate method

* change excluded type to []dht.NodeID in ChooseFiltered, change comment

* change name filter to excluded_nodes in proto

* implement helper function contains

* delete ChooseFiltered and add its functionality into Choose method to keep original author's history, add excluded argument into Choose calls

* regenerate mock_client.go

* regenerate protobuf

* adding the repair() func

* update test case to use new IDFromString function

* modified the repair() and updated streams mock

* modified the repair() and updated streams mock

* Options struct

* adding the repair() func

* modified the repair() and updated streams mock

* modified the repair() and updated streams mock

* integrating the segment repair()

* development repair with hack working

* repair segment changes

* integrated with mini hacks and rigged up test case with dev debug info

* integrated with ec and overlay

* added repair test case

* made the getNewUniqueNodes() to recursively go thru choose() to find get the required number of unique nodes

* cleaned up code
2018-10-30 14:06:12 -04:00
Dennis Coyle
d2933840d2
RPC Ping Request/Response (#544)
* Creates an RPC Ping method on the Node server

* adding Ping method to mockNodeServer implementation

* Remove the line skipping the Ping Test
2018-10-30 14:04:29 -04:00
Maximillian von Briesen
0606b38d68
Add FindValidNodes() to statdb (#530)
* add FindValidNodes() to statdb and update overlay node selection options

* add logic for FindValidNodes

* add FindValidNodes test

* fix findvalidnodes query

* fix linter errors

* clean up comments
2018-10-30 13:11:22 -04:00
Egon Elbre
bd67288071
Add inmemory psdb (#543) 2018-10-30 18:43:09 +02:00
Michal Niewrzal
0d03f2fbb5
Send bandwidth alloc from satellite to storage node (#538)
* Send bandwidth alloc from satellite to storage node

* Remove unecessary nil checks

* set Renter field

* fix tests

* goimports

* Update README.md

* Update README.md

* Update README.md

* Update README.md (#550)

* Skip flaky TestPing (#552)
2018-10-30 17:24:46 +01:00
Egon Elbre
7145d2a255 Skip flaky TestPing (#552) 2018-10-30 04:33:38 -06:00
Egon Elbre
47d056d627
testcontext package (#545) 2018-10-29 16:16:36 +02:00
Egon Elbre
68796d7964
Ensure we can create kademlia with custom storage (#533) 2018-10-26 19:54:00 +03:00