Commit Graph

517 Commits

Author SHA1 Message Date
aligeti
771c7e6ef9
download with specific number of nodes V3-353 (#564)
* download with specific number of nodes

* changes per code review comments
2018-11-02 11:22:01 -04:00
Egon Elbre
830241d845
Remove debug logging for routing table (#572) 2018-11-02 16:55:14 +02:00
Bryan White
57572cdeed
Use sync.Cond with kademlia workers (#512)
* wip - have to take a break; crying baby

* linter fixes

* bugfix

* responding to review feedback

* linter fixes?

* linter fixes

* feedback fixes

* feedback fixes

* linter fixes

* linter fixes

* linter fixes
2018-11-01 18:03:46 +01:00
Cameron
856c3a779f
Agreement Receiver service (#546)
* setup config and server structure for receiving bandwidth agreements

* add error channel
2018-11-01 12:51:19 -04:00
littleskunk
8c740f8390 reproduce repair error (#566) 2018-11-01 12:46:14 -04:00
Alexander Leitner
67306d296b Bandwidth fix (#567)
* Don't scan into nil
2018-11-01 12:40:26 -04:00
Bryan White
8b9711cb5e
better waitlist-gating (#557)
* better waitlist-gating

(cherry picked from commit 490fe02b7c3558da18678dfb651c92ec9c4a75b5)

* fix broken test

* linter fixes

* linter fixes

* make extension verification optional

* add certifcate gating script for captplanet

* fixing tests

* linter fixes

* linter fixes?

* moar linter fixes

* Revert "moar linter fixes"

This reverts commit 8139ccbd73cbbead987b7667567844f50f7df2c8.

* just kill me

* refactor

* refactor tests

* liniter...

* cleanup
2018-11-01 16:48:43 +01:00
Egon Elbre
2a8b681c4d
Run repairer and checker early (#565)
* Run repairers, checker, auditors first time they run to detect potential setup problems.
* Fix error handling in audit.Service
2018-11-01 16:03:45 +02:00
Dennis Coyle
1d3367bb09
bypass data checker and repairer in captplanet (#563) 2018-10-31 20:46:47 -04:00
Michal Niewrzal
a62b413de2
Uplink mount better prefix handling (#541)
* Uplink mount better prefix handling

* fix formatting
2018-10-31 21:38:32 +01:00
Egon Elbre
fd99de5834
Don't try to remove empty directory (#553) 2018-10-31 20:55:14 +02:00
Alexander Leitner
db80358f32
bandwidth agreements protobuf (#505)
* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Renamed payer to satellite in psdb
2018-10-31 14:47:25 -04:00
Dennis Coyle
a3becb8a7b
Add repairer & checker to Satelite (#561)
* Added repairer & checker to Satellite

* fixed repairer and checker configs
2018-10-31 12:22:35 -04:00
Maximillian von Briesen
40b210f604
Fix statdb query to avoid sql injection (#562)
* fix statdb query to avoid sql injection

* use byte array instead of string for node ids in statdb
2018-10-31 12:18:51 -04:00
JT Olio
a42973f089
captplanet: fix output line name (#549)
Change-Id: I37fbb76586ee3c0a40d24aad72aff2fcfc039a25
2018-10-31 10:00:58 -06:00
Jennifer Li Johnson
7ae2fa3575
moves bulk of code from ticker case to outside for indentation's sake (#559)
* moves bulk of code from ticker case to outside for indentation's sake

* adds whitespace

* removes break
2018-10-30 16:14:15 -04:00
Jennifer Li Johnson
1fb96689b8
creates run loop for data repair checker (#490)
* creates run loop for data repair checker

* moves actual checking and repairing under ticker case

* fixes mismatched queueaddrs
2018-10-30 15:16:40 -04:00
Alexander Leitner
7e4149138a
Piecestore proto change (#556)
* add action and storage node id to piecestore.proto. renamed payer and renter to satellite_id and uplink_id
2018-10-30 15:03:41 -04:00
aligeti
7ce94627f1
Implements Data Repair Interfaces in Stream Store (#455)
* add filter field into OverlayOptions message

* chooseFiltered method, add excluded parameter in populate method

* change excluded type to []dht.NodeID in ChooseFiltered, change comment

* change name filter to excluded_nodes in proto

* implement helper function contains

* delete ChooseFiltered and add its functionality into Choose method to keep original author's history, add excluded argument into Choose calls

* regenerate mock_client.go

* regenerate protobuf

* adding the repair() func

* update test case to use new IDFromString function

* modified the repair() and updated streams mock

* modified the repair() and updated streams mock

* Options struct

* adding the repair() func

* modified the repair() and updated streams mock

* modified the repair() and updated streams mock

* integrating the segment repair()

* development repair with hack working

* repair segment changes

* integrated with mini hacks and rigged up test case with dev debug info

* integrated with ec and overlay

* added repair test case

* made the getNewUniqueNodes() to recursively go thru choose() to find get the required number of unique nodes

* cleaned up code
2018-10-30 14:06:12 -04:00
Dennis Coyle
d2933840d2
RPC Ping Request/Response (#544)
* Creates an RPC Ping method on the Node server

* adding Ping method to mockNodeServer implementation

* Remove the line skipping the Ping Test
2018-10-30 14:04:29 -04:00
Maximillian von Briesen
0606b38d68
Add FindValidNodes() to statdb (#530)
* add FindValidNodes() to statdb and update overlay node selection options

* add logic for FindValidNodes

* add FindValidNodes test

* fix findvalidnodes query

* fix linter errors

* clean up comments
2018-10-30 13:11:22 -04:00
Egon Elbre
bd67288071
Add inmemory psdb (#543) 2018-10-30 18:43:09 +02:00
Michal Niewrzal
0d03f2fbb5
Send bandwidth alloc from satellite to storage node (#538)
* Send bandwidth alloc from satellite to storage node

* Remove unecessary nil checks

* set Renter field

* fix tests

* goimports

* Update README.md

* Update README.md

* Update README.md

* Update README.md (#550)

* Skip flaky TestPing (#552)
2018-10-30 17:24:46 +01:00
JT Olio
be6164c506 Update README.md (#550)
Change-Id: I2003abd8a98e3ce0a9336c1cfb00b4095a63f4e2
2018-10-30 04:34:00 -06:00
Richard Littauer
b3fcb37948 Update note about GOPATH (#551)
Ping @egonelbre
2018-10-30 04:33:38 -06:00
Egon Elbre
7145d2a255 Skip flaky TestPing (#552) 2018-10-30 04:33:38 -06:00
JT Olio
08ec36dae7 Update README.md 2018-10-30 04:33:38 -06:00
Richard Littauer
01a98a58b1 Some README edits (#548)
* Readme edits

Mostly language changes, but some code shifts as uplink was renamed.

* Edit intro contributing paragraph

Co-Authored-By: RichardLitt <richard.littauer@gmail.com>

* Add dev note

Co-Authored-By: RichardLitt <richard.littauer@gmail.com>
2018-10-29 18:36:17 -06:00
Egon Elbre
2328bf1b06
Add tparse to tests (#542) 2018-10-29 16:21:23 +02:00
Egon Elbre
47d056d627
testcontext package (#545) 2018-10-29 16:16:36 +02:00
Egon Elbre
68796d7964
Ensure we can create kademlia with custom storage (#533) 2018-10-26 19:54:00 +03:00
Dennis Coyle
3b7b2afb1f
Added ping support to node client (#491)
* added ping support to node client

* Added tests to Ping

* Added connection creation responsibility to Connection Pool
2018-10-26 12:38:22 -04:00
Michal Niewrzal
79e1ed5d1f
Fix uplink mount for latest api (#539)
Fix uplink mount for changes in store.List
2018-10-26 16:22:20 +02:00
Jennifer Li Johnson
8d779d3d3e Disconnects nodeclient, routing table dbs when done with kademlia (#507)
* disconnect from nodeclient

* cleanup connections in tests

* kademlia disconnects from nodeclient

* updating disconnect method for mocks

* creates separate disconnect and removeAll methods for tests

* adds init to connection pool

* fix folder cleanup and disconnect

* creates and cleans up test db files and disconnects kad

* removes db/.keep

* includes disconnect within cleanup methods

* creates public init method on connection pool to handle mutex copy issues

* remove all after disconnect

* pair creation and destruction

* checks disconnect error

* remove ctx

* fixes mock kad
2018-10-26 10:07:02 -04:00
Bryan White
df1f7a6214 Wait-list gating (#534) 2018-10-26 09:52:37 -04:00
JT Olio
fdfb5f1da9
uplink: fix mount and cli/gateway logical merge conflict (#537)
Change-Id: Ib1661e1c9d3f06783854b2293579ab715d0f1c20
2018-10-25 16:00:15 -06:00
JT Olio
cd51787827 split gateway from uplink
Change-Id: I7504a630cb85539750e32780c1c2347742a9c9c4
Reviewed-on: https://review.gerrithub.io/430644
Reviewed-by: Kaloyan Raev <kaloyan@storj.io>
Tested-by: Kaloyan Raev <kaloyan@storj.io>
2018-10-25 15:01:44 -06:00
Kaloyan Raev
99640225fd
Refactor Path type (#522)
The old paths.Path type is now replaced with the new storj.Path.

storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().

As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:

    storj.SplitPath
    storj.JoinPaths
    encryption.EncryptPath
    encryption.DecryptPath
    encryption.DerivePathKey
    encryption.DeriveContentKey

All code in master is migrated to the new storj.Path type.

The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go

This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
2018-10-25 23:28:16 +03:00
Cameron
f7828e73ea
remove ctx from repairer struct (#535) 2018-10-25 14:59:36 -04:00
paul cannon
e2c0dd437a
offer PostgreSQL storage for pointerdb (#440)
..although it ought to work for other storage.KeyValueStore needs as
well. it's just optimized to work pretty well for a largish hierarchy of
paths.

This includes the addition of "long benchmarks" for KeyValueStore
testing. These will only be run when -test-bench-long is added to the
test flags. In these benchmarks, a large corpus of paths matching a
natural ("real-life") hierarchy is read from paths.data.gz (which you
can get from https://github.com/storj/path-test-corpus) and imported
into a particular KeyValueStore. Recursive and non-recursive queries are
run on it to detect performance problems that arise only at scale.

This also includes alternate implementation of the postgreskv client,
which works in a less-bizarre way for non-recursive queries, but suffers
from poor performance in tests such as the long benchmarks. Once this
alternate impl is committed to the tree, we can remove it again; I just
want it to be available for future reference.
2018-10-25 12:11:28 -05:00
Egon Elbre
6e148e6249
Implement buckets metainfo as a wrapper over storage/buckets (#482) 2018-10-25 19:38:53 +03:00
Bill Thorp
cb454638d9
moved poll, removed ctx params (#531)
* moved poll, removed ctx params
2018-10-25 12:11:50 -04:00
Michal Niewrzal
b3c1ea1852
Add 'mount' command for uplink cli (read-only access) (#345)
* Add 'mount' command for uplink cli

* limit build to unix

* small changes

* Removed in-memory files map, fix in read function

* unmount on interrupt

* add delete support

* some Read improvements

* update to latest api version

* removed Println and better error handling

* fix linter errors

* fix linter errors

* disable async read + review comments

* minor fixes

* remove mutex

* code cleanup

* add prefix support

* simplifying if statement
2018-10-25 12:44:34 +02:00
Egon Elbre
61632095e9
Flatten internal/pkg and remove unused internal/pkg/node (#526) 2018-10-25 11:24:39 +03:00
Shawn Wilkinson
c8376a3eb7 Fix README whitespace (#532) 2018-10-25 10:02:52 +03:00
Richard Littauer
49b6972ad9 Fix logo in README (#527) 2018-10-25 10:01:23 +03:00
Kaloyan Raev
6413307edc Remove EncryptionScheme from pointerdb protobuf
This is an old definition from the very early stage of development. It
is not used anymore.

Change-Id: I6a033e4006e6edfa7c18acc6ae91c9e4e1df0e6a
Signed-off-by: Kaloyan Raev <kaloyan@storj.io>
Reviewed-on: https://review.gerrithub.io/429582
Reviewed-by: JT Olio <hello@jtolio.com>
Tested-by: JT Olio <hello@jtolio.com>
2018-10-24 17:05:08 -06:00
Michal Niewrzal
b2caac4323
Option for generating sample TLS certs for Minio GW (#525)
* Option for generating sample TLS certs for Minio GW

* os.Link errors handled

* rename option and change cert template
2018-10-24 22:17:28 +02:00
Dennis Coyle
3cd2be2b46 Update .clabot (#529) 2018-10-24 19:06:46 +02:00
Egon Elbre
8efb4f0e89
Fix repairing run (#523)
* Fix repairing run
* Fix concurrency bugs
* Add sync2.Limiter concurrency primitive
2018-10-24 15:35:59 +03:00