Commit Graph

6392 Commits

Author SHA1 Message Date
Egon Elbre
7145d2a255 Skip flaky TestPing (#552) 2018-10-30 04:33:38 -06:00
JT Olio
08ec36dae7 Update README.md 2018-10-30 04:33:38 -06:00
Richard Littauer
01a98a58b1 Some README edits (#548)
* Readme edits

Mostly language changes, but some code shifts as uplink was renamed.

* Edit intro contributing paragraph

Co-Authored-By: RichardLitt <richard.littauer@gmail.com>

* Add dev note

Co-Authored-By: RichardLitt <richard.littauer@gmail.com>
2018-10-29 18:36:17 -06:00
Egon Elbre
2328bf1b06
Add tparse to tests (#542) 2018-10-29 16:21:23 +02:00
Egon Elbre
47d056d627
testcontext package (#545) 2018-10-29 16:16:36 +02:00
Egon Elbre
68796d7964
Ensure we can create kademlia with custom storage (#533) 2018-10-26 19:54:00 +03:00
Dennis Coyle
3b7b2afb1f
Added ping support to node client (#491)
* added ping support to node client

* Added tests to Ping

* Added connection creation responsibility to Connection Pool
2018-10-26 12:38:22 -04:00
Michal Niewrzal
79e1ed5d1f
Fix uplink mount for latest api (#539)
Fix uplink mount for changes in store.List
2018-10-26 16:22:20 +02:00
Jennifer Li Johnson
8d779d3d3e Disconnects nodeclient, routing table dbs when done with kademlia (#507)
* disconnect from nodeclient

* cleanup connections in tests

* kademlia disconnects from nodeclient

* updating disconnect method for mocks

* creates separate disconnect and removeAll methods for tests

* adds init to connection pool

* fix folder cleanup and disconnect

* creates and cleans up test db files and disconnects kad

* removes db/.keep

* includes disconnect within cleanup methods

* creates public init method on connection pool to handle mutex copy issues

* remove all after disconnect

* pair creation and destruction

* checks disconnect error

* remove ctx

* fixes mock kad
2018-10-26 10:07:02 -04:00
Bryan White
df1f7a6214 Wait-list gating (#534) 2018-10-26 09:52:37 -04:00
JT Olio
fdfb5f1da9
uplink: fix mount and cli/gateway logical merge conflict (#537)
Change-Id: Ib1661e1c9d3f06783854b2293579ab715d0f1c20
2018-10-25 16:00:15 -06:00
JT Olio
cd51787827 split gateway from uplink
Change-Id: I7504a630cb85539750e32780c1c2347742a9c9c4
Reviewed-on: https://review.gerrithub.io/430644
Reviewed-by: Kaloyan Raev <kaloyan@storj.io>
Tested-by: Kaloyan Raev <kaloyan@storj.io>
2018-10-25 15:01:44 -06:00
Kaloyan Raev
99640225fd
Refactor Path type (#522)
The old paths.Path type is now replaced with the new storj.Path.

storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().

As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:

    storj.SplitPath
    storj.JoinPaths
    encryption.EncryptPath
    encryption.DecryptPath
    encryption.DerivePathKey
    encryption.DeriveContentKey

All code in master is migrated to the new storj.Path type.

The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go

This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
2018-10-25 23:28:16 +03:00
Cameron
f7828e73ea
remove ctx from repairer struct (#535) 2018-10-25 14:59:36 -04:00
paul cannon
e2c0dd437a
offer PostgreSQL storage for pointerdb (#440)
..although it ought to work for other storage.KeyValueStore needs as
well. it's just optimized to work pretty well for a largish hierarchy of
paths.

This includes the addition of "long benchmarks" for KeyValueStore
testing. These will only be run when -test-bench-long is added to the
test flags. In these benchmarks, a large corpus of paths matching a
natural ("real-life") hierarchy is read from paths.data.gz (which you
can get from https://github.com/storj/path-test-corpus) and imported
into a particular KeyValueStore. Recursive and non-recursive queries are
run on it to detect performance problems that arise only at scale.

This also includes alternate implementation of the postgreskv client,
which works in a less-bizarre way for non-recursive queries, but suffers
from poor performance in tests such as the long benchmarks. Once this
alternate impl is committed to the tree, we can remove it again; I just
want it to be available for future reference.
2018-10-25 12:11:28 -05:00
Egon Elbre
6e148e6249
Implement buckets metainfo as a wrapper over storage/buckets (#482) 2018-10-25 19:38:53 +03:00
Bill Thorp
cb454638d9
moved poll, removed ctx params (#531)
* moved poll, removed ctx params
2018-10-25 12:11:50 -04:00
Michal Niewrzal
b3c1ea1852
Add 'mount' command for uplink cli (read-only access) (#345)
* Add 'mount' command for uplink cli

* limit build to unix

* small changes

* Removed in-memory files map, fix in read function

* unmount on interrupt

* add delete support

* some Read improvements

* update to latest api version

* removed Println and better error handling

* fix linter errors

* fix linter errors

* disable async read + review comments

* minor fixes

* remove mutex

* code cleanup

* add prefix support

* simplifying if statement
2018-10-25 12:44:34 +02:00
Egon Elbre
61632095e9
Flatten internal/pkg and remove unused internal/pkg/node (#526) 2018-10-25 11:24:39 +03:00
Shawn Wilkinson
c8376a3eb7 Fix README whitespace (#532) 2018-10-25 10:02:52 +03:00
Richard Littauer
49b6972ad9 Fix logo in README (#527) 2018-10-25 10:01:23 +03:00
Kaloyan Raev
6413307edc Remove EncryptionScheme from pointerdb protobuf
This is an old definition from the very early stage of development. It
is not used anymore.

Change-Id: I6a033e4006e6edfa7c18acc6ae91c9e4e1df0e6a
Signed-off-by: Kaloyan Raev <kaloyan@storj.io>
Reviewed-on: https://review.gerrithub.io/429582
Reviewed-by: JT Olio <hello@jtolio.com>
Tested-by: JT Olio <hello@jtolio.com>
2018-10-24 17:05:08 -06:00
Michal Niewrzal
b2caac4323
Option for generating sample TLS certs for Minio GW (#525)
* Option for generating sample TLS certs for Minio GW

* os.Link errors handled

* rename option and change cert template
2018-10-24 22:17:28 +02:00
Dennis Coyle
3cd2be2b46 Update .clabot (#529) 2018-10-24 19:06:46 +02:00
Egon Elbre
8efb4f0e89
Fix repairing run (#523)
* Fix repairing run
* Fix concurrency bugs
* Add sync2.Limiter concurrency primitive
2018-10-24 15:35:59 +03:00
Bill Thorp
842ebc9546
PriorityQueue refactor (#500)
* PriorityQueue refactor

* changed closest to return big.Int

* test smaller queue, return nil on empty
2018-10-24 08:24:47 -04:00
Egon Elbre
2bb2c50d79
Travis Windows (#519)
* Make Travis run Windows tests
* Simplify git checkout
2018-10-24 14:51:49 +03:00
Michal Niewrzal
fb5ccbc407
Pieces must be name spaced by Satellite (#503)
* Pieces must be name spaced by Satellite

* fixed formatting
2018-10-23 18:03:35 +02:00
Kaloyan Raev
37a1542480
Improve error handling in encryption pkg (#520)
* Improve error handling in encryption pkg

* More Go-ish errors
2018-10-23 17:05:15 +03:00
Egon Elbre
80408541d5
Travis quiet parallel tests (#514) 2018-10-19 21:55:06 +03:00
Dylan Lott
28ee32f322
Updates captainplanet test scripts to use /tmp/capt for test files (#517)
* creates tmp dir and cleans it up after script runs

* updates captplanet test script to use tmpdir

* updates tmpdir to use windows-compatible mktemp
2018-10-19 11:14:29 -06:00
Maximillian von Briesen
94128d63a1
fix captplanet and satellite overwrite flags (#516)
* fix captplanet and satellite overwrite flags
2018-10-19 11:06:02 -04:00
aligeti
a61de403d7
aws CLI cancel handling development (#511)
Handle CTRL+C while using AWS CLI #511 (Storj V3/V3-487)
2018-10-19 10:36:43 -04:00
Cameron
939515e6ee
Increase test coverage for overlay client Choose method (#508)
* add server option to NewMockServer

* add test to check nodes returned by Choose against list of excluded IDs
2018-10-19 10:05:31 -04:00
Kaloyan Raev
ac4805a841
Adapt storj and encryption packages (#513) 2018-10-19 16:38:13 +03:00
Dylan Lott
fc2fe0c294
pkg/utils CollectErrors tests (#496)
* Sets up test harness for CollectErrors

* pkg/utils tests passing

* updates to utils tests
2018-10-18 10:55:00 -06:00
Egon Elbre
03bd93bba7
Make kademlia use less file-descriptors (#498) 2018-10-18 19:20:23 +03:00
Egon Elbre
21026b35f5
Fix error message (#502) 2018-10-18 17:48:57 +03:00
Kaloyan Raev
b9e473a5eb
Encapsulate key encryption in the Key and EncryptedPrivateKey types (#499) 2018-10-18 17:21:08 +03:00
Kaloyan Raev
b34adc310b
Extract encryption package from eestream (#497) 2018-10-18 14:10:29 +03:00
Dylan Lott
6b88f0a36b
Moves CollectErrors into pkg/utils (#493)
Merging but creating a PR for tests shortly
2018-10-17 14:23:44 -06:00
Matt Robinson
d1e7534f9e
use GOPROXY if it's set in the environment (#495) 2018-10-17 15:06:26 -04:00
Matt Robinson
48eb8e5c8e
sign windows builds (#494) 2018-10-17 14:21:18 -04:00
Dylan Lott
19d8b7a2df
return multiple errors when captplanet fails (#478)
* implements CombineErrors on runCfg error chan

* implement error channel timeout and CombineErrors

* implements CombineErrors with time.After timeout

* changes d to duration in args

* changes comment to better reflect what collectErrors does

* appeases the linter gods

* Gets rid of timeout bug in collectErrors
2018-10-17 09:56:39 -06:00
Michal Niewrzal
f9248c21d4
Satellite verification on storage node (#469)
* Satellite verification on storage node

* fix formatting

* fix formatting

* rename SignatureAuth to SignedMessage

* fixes after review

* fix linter errors

* improve errors handling

* remove SignedMessageProvider

* fix liter errors

* params changed to authorization, signed message in audit, minor fixes

* fix formatting
2018-10-17 13:40:11 +02:00
nfarah86
92a8a825c4 Encrypt metadata (#462) 2018-10-17 14:34:50 +03:00
Egon Elbre
8fcec8b8cf
Remove gopkg (#486) 2018-10-16 23:13:21 +03:00
James Hagans
115d257bb1
Audit cron (#466) 2018-10-16 16:02:18 -04:00
Stefan Benten
1e569e3d4d
Add S3 Scheme as valid URL Scheme (#483)
* Add s3 Scheme as valid URL Scheme

* Merge from Master

* Remove Test Case and fix Error

* Add invalid Test Case

* add valid s3 Test Case for good measure

* Improve Speed and Resource Handling
2018-10-16 21:50:37 +02:00
Matt Robinson
bcd18a44bc
give our appearence on windows some love (#429) 2018-10-16 14:48:17 -04:00