Commit Graph

91 Commits

Author SHA1 Message Date
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Kaloyan Raev
ee5f32b0f7
ConcatRanger closes segment readers as soon as data is read (#349)
* ConcatRanger closes a segment readers as soon as data is read

* Fix linter errors

* Fix data race

* Nicer call to CombineErrors
2018-09-14 10:10:43 -04:00
aligeti
3571886fe7 check uplink setup (#339)
* check uplink setup

* code review comments changes
2018-09-13 13:34:01 -04:00
Kaloyan Raev
8f4b29abd0
Lazy segment ranger (#343) 2018-09-12 09:06:44 -04:00
Michal Niewrzal
e90c2140b8 Show file size for uplink ls command (#344)
* Show file size for uplink ls command

* align size to right

* Fix stream store list method to handle meta.Size flag
2018-09-12 09:03:31 -04:00
Egon Elbre
e7e2d4d7c9
use golangci-lint and add missing error checks (#340) 2018-09-11 16:13:25 +03:00
Alexander Leitner
8f07c4d746
Keep track of used space (#338)
* Keep track of space at all times

* Create index on expires

* Updated mock psclient
2018-09-11 08:40:45 -04:00
Jennifer Li Johnson
d0f87f0de1
Overlay Bulk Lookup (#279)
* implements bulk node lookup from overlay cache
2018-09-11 00:52:14 -04:00
Cameron
555e7f1897 CLI commands with bucket store API (#304)
* add mb command

* forgot colon

* add command descriptions

* use utils.ParseURL in commands

* return error message instead of minio.BucketAlreadyExists in mb

* ls command with bucket store functionality

* rb command with bucket store functionality

* rm command with bucket store functionality

* newline

* use print rather than errs for messages, add no buckets messsage

* cp command with bucket store functionality

* remove deprecated getStorjObjects function

* defer utils.LogClose(f) on instead of defer f.Close()

* Check for no buckets after for loop

* add checks for unspecified bucket in bucket store methods

* fix incorrect return types

* add no path error messages in object store methods

* split copy into helpers

* srcObj scheme check in download

* print buckets instead of appending to slice

* check if destObj.Host != srcObj.Host

* better method of handling destination name if not specified

* uplink rename

* final cleanups

* trailing slash fixes

* linting

* more linting

* helpful error messages

* Adjust startAfter after merging #328

* Improve output messages

* Improved error check for empty bucket and path

* No page limit on client side. Rely on server side limit.

* Better time formatting

* Fix paths in recursive list results
2018-09-09 11:31:26 -06:00
Kaloyan Raev
f62a3c383c
Filter prefix from list results (#328) 2018-09-09 09:34:23 +03:00
Egon Elbre
4671ac6c5b Abort trying to upload segments when reader fails (#326)
* abort trying to upload segments when reader fails

* also return the error when it fails
2018-09-08 09:41:40 -06:00
Michal Niewrzal
dfc52e4394 [WIP] Piece Rangers should connect lazily to storage nodes (second approach) (#321)
* Piece Rangers should connect lazily to storage nodes

* code reorganization

* fixed unit tests

* remove if
2018-09-08 16:52:19 +03:00
Egon Elbre
0a2839b2d7
Add IsPrefix support to piecestore (#313) 2018-09-07 17:20:15 +03:00
JT Olio
a4d1070d68
storage/ec: fix some Get socket leaks (#308)
* storage/ec: fix some Get socket leaks

* fix linters
2018-09-05 10:08:39 -07:00
Alexander Leitner
899e1e68f1
Add functions for signing and verifying during bandwidth exchange (#246)
* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* wip

* wip

* actual signatures in tests

(cherry picked from commit 1464853b737f1d712d64fbf90147f535525c8fd9)

* bugfixing

* Generate private key in example

* Generate signatures for pieceranger tests

* Update examples to use TLS

* Use private key from identity inside of example

* Use crypto.PrivateKey interface

* Change err name in defers

* Pass tests

* Pass identity Key to PSClient

* Get tests passing on travis

* Resolve linter complaints
2018-08-27 14:35:27 -04:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
James Hagans
fdfa6e85c8 Streamstore impl of Stor interface (#191)
* first stab at PUT

* only PUT

* working on PUT

* Put with LimitReader

* start of Get

* reorder of files and proto meta

* working on Meta

* working on Meta

* add aware limit reader

* add size from segment put

* rm if for eof

* update to proto meta

* update gen proto file

* working on get

* working on get

* working on get

* working on list

* working on delete

* working on list

* working on meta method

* fix merge error and working on feedback from PR

* update to proto file

* rm size tuple

* mv eof limit reader to new file

* add toMeta

* rm varible names

* add updates from PR feedback

* updates from PR feedback

* updates from PR feedback

* add toMeta size based on total size

* update toMeta size calculation

* rm passthrough

* add default to config for segment size

* fix get method ranger bug

* add object support for nested stream proto

* rm nested stream meta data

* rm test for another PR
2018-08-23 21:56:38 -06:00
JT Olio
9aee5efd99
pointerdb: separate client and server packages (#261)
* pointerdb: separate client and server packages

the reason for this is so that things the server needs (bolt, auth)
don't get imported if all you need is the client. will result in
smaller binaries and less flag definitions

* review comments
2018-08-22 09:07:00 -06:00
Kaloyan Raev
130acf57a5
Make PSClient a Closer, so it can be closed with utils.LogClose() (#247) 2018-08-20 18:11:54 +03:00
Alexander Leitner
b0db33f919
Bandwidth accounting (#134)
* captplanet standalone farmer setup

* Bandwidth Allocation

* utils.Close method changed to utils.LogClose

* Get build temporarily working

* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function

* Update example client to reflect changes in client API

* Update ecclient to use latest PSClient, Make NewPSClient return error also

* Updated pieceranger tests to check for errors; sign method should take byte array

* Handle defers in store.go better

* Fix defer functions in psdb.go

* fun times

* Protobuf bandwidthallocation data is now a byte array

* Remove psservice package and merge it into pstore server

* Write wrapper for database calls

* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment

* Make PSDB tests implementation independent rather than method independent

* get rid of payer, renter in ecclient

* add context monitoring in store and retrieve
2018-08-17 13:40:15 -04:00
Natalie Villasana
ff65663867
adds Bucket Store (#213)
* adds foundation for bucketStore

* adds prefixedObjStore to buckets package, adjusts gateway-storj accordingly

* fixes multi value assignment problems in gateway-storj

* fixes more multi value assignment errors in gateway-storj

* starts changing miniogw tests to accommodate buckets

* creates bucket store mock

* wip - fixing test cases in object tests

* adds get, put, and list object tests, comments out two test cases

* adds happy scenario tests for bucket methods

* fixes bug in list, removes redundant parts from gateway tests

* fixes nit

* Clean up tests from #188

* Fix bug with timestamp conversion in segment store

* fixes segments.Meta test

* Fix regression in listing objects in a bucket

* adds check to see if bucket is empty before deleting

* updates DeleteBucket test to account for empty/full bucket

* adds TODOs for DeleteBucket and MakeBucket for some cases, adjusts tests, filters out minio errors in logging.go

* adds checks for if buckets already exist or not in DeleteBucket and MakeBucket functions; adjusts tests

* adds BucketNotFound error check in bucket store, removes todo

* adds make_bucket to Travis test, updates boltdb client constructor to always create a bucket (table)
2018-08-16 10:32:28 -04:00
Natalie Villasana
6ab323bfec
adds stripe_size field to pointerdb proto file (#218)
* adds erasure_share_size to RS in pointerdb proto

* adds makeErasureScheme helper method to segments store

* cleans up makeErasureScheme function
2018-08-14 11:15:22 -04:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
Kaloyan Raev
ffaebb57df
Improved error message in ECClient (#214)
* Improved error message in ECClient

* Fixed broken test
2018-08-10 12:14:23 +03:00
Natalie Villasana
9ad4949825
adds segment store tests (#198)
* starts adding segmentStore tests

* adds mocked interfaces for segmentStore tests

* adds tests for put, get, delete, and list

* regenerates pointerdb mock and updates calls to accommodate new changes
2018-08-07 11:52:11 -04:00
Kaloyan Raev
aac7c6fbd1
ECClient pads data to fit RS block size (#199) 2018-08-06 17:24:30 +03:00
Maximillian von Briesen
d934733c4d Move api key from pdb client method calls to pdb client constructor (#194)
* move api key from pdb client method calls to pdb client constructor

* update pdb example client
2018-08-02 13:09:50 -06:00
Kaloyan Raev
85eee94814
Fixes V3-173 - ECClient check if given nodes are unique (#186) 2018-08-02 18:12:19 +03:00
Natalie Villasana
a4bb03d0da
adds inline segment support for segmentstore (#162)
* begins adding inline segment support for segmentstore

* adds PeekThresholdReader struct plus Read and isInline methods

* moves PeekThresholdReader to peek.go, adds more simplified Read function

* adds PeekThresholdReader tests

* reverts Read function to earlier version, updates tests to use ReadFull instead

* Get function now handles inline type pointers

* adds correct type Size and ExpirationDate to inline segment

* fixes return value in Put func error condition

* moves thresholdBuf and Read tests into a table test

* adds border case test, fixes redundant parts

* passes sizedReader size to makeRemotePointer
2018-08-01 16:25:06 -04:00
Kaloyan Raev
11e12bff5f Fixes V3-176 - Segment size is stored in the pointer (#185) 2018-07-31 18:49:23 -04:00
Kaloyan Raev
7e136db9cf
Cleanup metadata across layers (#180)
* Cleanup metadata across layers

* Fix pointer db tests
2018-07-30 21:57:50 +03:00
JT Olio
d986c337aa streamstore: add passthrough implementation (#176)
this doesn't implement streamstore, this just allows us to try and
get the june demo working again in the meantime
2018-07-27 12:13:08 -06:00
Kaloyan Raev
daf391e473
Refactor List in PointerDB (#163)
* Refactor List in Pointer DB

* Fix pointerdb-client example

* Fix issue in Path type related to empty paths

* Test for the PointerDB service with some fixes

* Fixed debug message in example: trancated --> more

* GoDoc comments for unexported methods

* TODO comment to check if Put is overwriting

* Log warning if protobuf timestamp cannot be converted

* TODO comment to make ListPageLimit configurable

* Rename 'segment' package to 'segments' to reflect folder name
2018-07-27 09:02:59 +03:00
JT Olio
5f6607935b
captplanet (#159)
* captplanet

I kind of went overboard this weekend.

The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!

To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.

Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.

A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
 * Kademlia
 * OverlayCache
 * PointerDB
 * StatDB
 * PieceStore
 * etc.

A Provider is a collection of Responsibilities that
share an Identity, such as:
 * The heavy client
 * The farmer
 * The gateway

An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.

Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.

To make a Responsibility, one should create a "config"
struct, such as:

```
type Config struct {
  RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
  SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```

To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.

cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.

Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
 * cmd/captplanet/farmer/main.go (a new farmer binary)
 * cmd/captplanet/hc/main.go (a new heavy client binary)
 * cmd/captplanet/gw/main.go (a new minio gateway binary)

Usage:

```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```

Configuration is placed by default in `~/.storj/capt/`

Other changes:

 * introduces new config structs for currently existing
   Responsibilities that conform to the new Responsibility
   interface. Please see the `pkg/*/config.go` files for
   examples.

 * integrates the PointerDB API key with other global
   configuration via flags, instead of through environment
   variables through viper like it's been doing. (ultimately
   this should also change to use the PointerDB config
   struct but this is an okay shortterm solution).

 * changes the Overlay cache to use a URL for database
   configuration instead of separate redis and bolt config
   settings.

 * stubs out some peer identity skeleton code (but not the
   meat).

 * Fixes the SegmentStore to use the overlay client and
   pointerdb clients instead of gRPC client code directly

 * Leaves a very clear spot where we need to tie the object to
   stream to segment store together. There's sort of a "golden
   spike" opportunity to connect all the train tracks together
   at the bottom of pkg/miniogw/config.go, labeled with a
   bunch of TODOs.

Future stuff:

 * I now prefer this design over the original
   pkg/process.Service thing I had been pushing before (sorry!)

 * The experience of trying to have multiple farmers
   configurable concurrently led me to prefer config structs
   over global flags (I finally came around) or using viper
   directly. I think global flags are okay sometimes but in
   general going forward we should try and get all relevant
   config into config structs.

 * If you all like this direction, I think we can go delete my
   old Service interfaces and a bunch of flags and clean up a
   bunch of stuff.

 * If you don't like this direction, it's no sweat at all, and
   despite how much code there is here I'm not very tied to any
   of this! Considering a lot of this was written between midnight
   and 6 am, it might not be any good!

* bind tests
2018-07-24 10:08:28 -06:00
Kaloyan Raev
445044b856 Adjust Segment Store to the updated interface (#160)
* Adjust Segment Store to the updated interface

* Move /pkg/storage/segment to /pkg/storage/segments

* Fix overlay client tests

* Revert changes in NewOverlayClient return value

* Rename `rem` to `seg`

* Implement Meta()
2018-07-23 14:05:02 -06:00
Kaloyan Raev
8ce889ed1b
ObjectStore (#128)
* WIP ObjectStore

* Remove methods for extended attributes

* List returns metadata too

* No real need to prepend "object" in path

* Serialize metadata

* List retuns []ListItem instead of []Path
2018-07-16 23:44:28 +03:00
Cameron
6463b87ebe
Errcheck (#133)
* add errcheck

* fixed linter errors

* fixes

* errcheck fixes in pkg/paths

* Fix errchecks in PieceID.Derive

* Fix ecclient tests

* Move closeConn a little bit above in the exectution flow

* fix new lint errors

* Fatalf -> Printf

* address eclipsed errors

* rename err to closeErr

* rename err to closeErr for differentiation
2018-07-16 15:22:34 -04:00
Natalie Villasana
d82486b85b
adds SegmentStore Put functionality (#138)
* lays out SegmentStore functions to implement

* Merge branch 'master' into segment-store

* adds overlay calls to put

* allows SegmentStore Put to upload a file to ecclient, then save pointer to pointerdb

* Merge branch 'master' into segment-store

* removes new overlay client instance in Put

* fixes syntax

* fixes syntax again

* fixes imports

* fixes typo

* removes pointerdb client from segmentStore struct for now

* changes SegmentStore to segmentStore

* changing types in parameters to fit other function calls

* takes RedundancyStrategy out of Put params

* changes NewClient param back to take an interface (not pointer to interface)

* fixes types

* moves pointer into PutRequest in SegmentStore Put

* passes interfact, not pointer to interface to NewSegmentStore

* fixes some types

* Get returns an instance of Meta

* fixes PutRequest fields

* adds remotePieces slice to pointerdb PutRequest

* ecClient Put now takes *proto.Nodes instead of proto.Nodes

* fixes syntax

* changes ec client dial interface to use *proto.Node

* changes other instances of proto.Node to *proto.Node in ecclient pkg

* adds *proto.Node to Get and Delete functions in ecclient pkg

* changes proto.Node to pointer in ec client_test

* changes proto.Node to pointer in ec client_test

* adds ecclient and pointerdb client to the segmentstore constructor

* adds ecclient and pointerDBClient to segmentStore constructor
2018-07-12 18:37:50 -04:00
Bryan White
5b913c45b9
Transport security (#63)
* wip initial transport security

* wip: transport security (add tests / refactor)

* wip tests

* refactoring - still wip

* refactor, improve tests

* wip tls testing

* fix typo

* wip testing

* wip testing

* wip

* tls_test passing

* code-style improvemente / refactor; service and tls tests passing!

* code-style auto-format

* add TestNewServer_LoadTLS

* refactor; test improvements

* refactor

* add client cert

* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* add separate `Process` tests for bolt and redis-backed overlay

* more testing

* fix gitignore

* fix linter error

* goimports goimports GOIMPORTS GoImPortS!!!!

* wip

* fix port madness

* forgot to add

* add `mux` as handler and shorten context timeouts

* gofreakingimports

* fix comments

* refactor test & add logger/monkit registry

* debugging travis

* add comment

* Set redisAddress to empty string for bolt-test

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* Merge branch 'tls' into tls-upstream

* tls:
  add client cert
  refactor
  refactor; test improvements
  add TestNewServer_LoadTLS
  code-style auto-format
  code-style improvemente / refactor; service and tls tests passing!
  tls_test passing
  wip
  wip testing
  wip testing
  fix typo
  wip tls testing
  refactor, improve tests
  refactoring - still wip
  wip tests
  wip: transport security (add tests / refactor)
  wip initial transport security

* fixing linter things

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* Merge branch 'master' into tls-upstream

* master:
  Add error to the return values of Ranger.Range method (#90)
  udp-forwarding: demo week work! (#84)

* Merge branch 'kad-tests' into tls-upstream

* kad-tests:
  piecestore kademlia newID
  linter fixes
  overlay upgraded
  ports
  cleanup
  wip
  updated coyle/kademlia
  wip
  wip
  wip
  wip
  wip
  remove bkad dependencie from tests
  wip
  wip
  files created
  port changes

* wip

* finish merging service tests

* add test for different client/server certs

* wip

* Merge branch 'master' into tls-upstream

* master:
  Add context to Ranger.Range method (#99)
  Coyle/kad client (#91)

* wip

* wip; refactoring/cleanup

* wip

* Merge branch 'master' into tls

* master:
  Bolt backed overlay cache (#94)
  internal/test: switch errors to error classes (#96)

* wip - test passing

* cleanup

* remove port.go

* cleanup

* Merge branch 'master' into tls

* master:
  hardcode version (#111)
  Coyle/docker fix (#109)
  pkg/kademlia tests and restructuring (#97)
  Use continue instead of return in table tests (#106)
  prepend storjlabs to docker tag (#108)
  Automatically build, tag and push docker images on merge to master (#103)

* more belated merging

* more belated merging

* more belated merging

* add copyrights

* cleanup

* goimports

* refactoring

* wip

* wip

* implement `TLSFileOptions#loadTLS`, refactoring:

`peertls.TestNewClient_LoadTLS` is the failing holdout; Still trying to figure out why I'm getting ECDSA verification is failing.

* not sure if actually working:

Tests are now passing (no more "ECDSA verification failed"); however,
`len(*tls.Certificates.Certificate) == 1` which I don't think should be
the case if the root and leaf are being created correctly.

* Experimenting/bugfixing?:

I think leaf certs should be properly signed by the parent now but not
entirely certain. It's also unclear to me why in
`VerifyPeerCertificate`, `len(rawCerts) == 1` when the certs should
contain both the root and leaf afaik.

* Properly write/read certificate chain (root/leaf):

I think I'm now properly reading and writing the root and leaf
certificate chain such that they're both being received by
`VerifyPeerCertificate`.

The next step is to parse the certificates with `x509.ParseCertificate`
(or similar) and verify that the public keys and signatures match.

* Add tls certificate chain signature veification (spike):

+ `VerifyPeerCertificate` verifies signatures of certificates using the
key of it's parent if there is one; otherwise, it verifies the
certificate is self-signed
+ TODO: refactor
+ TODO: test

* refactoring `VerifyPeerCertificate`

* cleanup

* refactor

* Merge branch 'master' into tls

* master:
  Remove some structural folders we don't seem to be using. (#125)
  license code with agplv3 (#126)
  Update .clabot (#124)
  added team memebers (#123)
  clabot file added (#121)
  ECClient (#110)
  docker image issue fixed (#118)
  Piecestore Farmer CLI  (#92)
  Define Path type (#101)
  adds netstate pagination (#95)
  Transport Client (#89)
  Implement psclient interface (#107)
  pkg/process: start replacing pkg/process with cobra helpers (#98)
  protos/netstate: remove stuff we're not using (#100)
  adding coveralls / code coverage  (#112)

* responding to review feedback / cleanup / add copywrite headers

* suggestions

* realitive

* Merge pull request #1 from coyle/coyle/tls

suggestions

* remove unnecessary `_`s

* Merge branch 'tls' of github.com:bryanchriswhite/storj into tls

* 'tls' of github.com:bryanchriswhite/storj:
  realitive
  suggestions

* Responding to review feedback:

+ refactor `VerifyPeerCertificate`

* remove tls expiration

* remove "hosts" and "clien option" from tls options

* goimports

* linter fixes
2018-07-09 19:43:13 +02:00
Kaloyan Raev
a6bafa993a
Obscure piece id across piece store nodes (#120)
* Obscure piece id across piece store nodes

* Add line separator in imports

* Avoid potential panic if hash < 32 bytes
2018-07-06 11:51:13 +03:00
Kaloyan Raev
d8f1ec1db6
ECClient (#110)
* WIP ECClient

* Get returns RangeCloser

* Introduce RedundancyStrategy

* Constructor takes max buffer memory

* Remove unnecessary NopCloser wrapper

* Added telemetry

* Tests

* Adapt to PSClient from master

* Decode should report error if empty rrs map is passed

* collectErrors helper

* Move to /pkg/storage

* Move to /pkg/storage/ec

* Rename ecclient.go to client.go

* Better logging

* Rename ec.ECClient to ec.Client

* Fix some test execution

* Adopt Transport Client from master
2018-07-03 11:35:01 +03:00