Commit Graph

98 Commits

Author SHA1 Message Date
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Michal Niewrzal
ad327bedb1
Use context to propagate API Key (#383)
* Satellite signs proofs

* wip

* remove direct apikey usage from pdbclient

* adjusting unit tests

* fix linter errors

* unit tests

* linter errors

* remove usless interface

* remove unused code

* improve unit tests

* signature generation

* code review changes

* code review comments

* back to satellite-id signature generation

* remove go-grpc-middlewar dependency

* small step back

* linter fixes

* fix tests

* packages reorganization

* Move TestAPIKeyInjector to grpcauth package
2018-10-09 16:39:14 +02:00
Egon Elbre
2019803c5e
Use RepairThreshold naming consistently (#385) 2018-09-26 17:23:33 +03:00
Maximillian von Briesen
821d0b6f1d
Stream encryption (#302)
* begin adding encryption for remote pieces

* begin adding decryption

* add encryption key as arg to Put and Get

* move encryption/decryption to object store

* Add encryption key to object store constructor

* Add the erasure scheme to object store constructor

* Ensure decrypter is initialized with the stripe size used by encrypter

* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"

This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.

* Revert "Add the erasure scheme to object store constructor"

This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.

* move encryption to stream store

* move decryption stuff to stream store

* revert changes in object store

* add encryptedBlockSize and close rangers on error during Get

* calculate padding sizes correctly

* encryptedBlockSize -> encryptionBlockSize

* pass encryption key and block size into stream store

* remove encryption key and block size from object store constructor

* move encrypter/decrypter initialization

* remove unnecessary cast

* Fix padding issue

* Fix linter

* add todos

* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata

* use different encryption key for each segment

* encrypt data in one step if it is small enough

* refactor and move encryption stuff

* fix errors related to nil slices passed to copy

* fix encrypter vs. decrypter bug

* put encryption stuff in eestream

* get captplanet test to pass

* fix linting errors

* add types for encryption keys/nonces and clean up

* fix tests

* more review changes

* add Cipher type for encryption stuff

* fix rs_test

* Simplify type casting of key and nonce

* Init starting nonce to the segment index

* don't copy derived key

* remove default encryption key; force user to explicitly set it

* move getSegmentPath to streams package

* dont require user to specify encryption key for captplanet

* rename GenericKey and GenericNonce to Key and Nonce

* review changes

* fix linting error

* Download uses the encryption type from metadata

* Store enc block size in metadata and use it for download
2018-09-26 09:32:23 -04:00
Brandon Iglesias
f1a82e588c updating the reed solomon numbers (#350)
* Update config.go

* updating captplanet

* Fix captplanet setup command for >= 100 nodes
2018-09-25 18:38:37 +03:00
Kaloyan Raev
4a176915ef MinioGW should return ObjectNotFound if non-existing file is deleted (#379) 2018-09-24 22:14:21 +03:00
Egon Elbre
8d73b20a63
Basic Multipart support (#351) 2018-09-21 22:44:45 +03:00
Egon Elbre
ebb839da7d
Add ListV2 method (#365) 2018-09-18 18:26:53 +03:00
Kaloyan Raev
ee5f32b0f7
ConcatRanger closes segment readers as soon as data is read (#349)
* ConcatRanger closes a segment readers as soon as data is read

* Fix linter errors

* Fix data race

* Nicer call to CombineErrors
2018-09-14 10:10:43 -04:00
Kaloyan Raev
20863e6bdf Fix recursive listing and deletion in S3 gateway (#333)
* Fix recursive listing and deletion in S3 gateway

* Prepend the prefix only on recursive listing

* Fix tests
2018-09-09 15:51:33 -06:00
JT Olio
fd94300c53
move stacktraces to debug level, disable debug by default (#329)
* move stacktraces to debug level, disable debug by default

* configure zap better

* var to const
2018-09-09 05:27:09 -06:00
Kaloyan Raev
f62a3c383c
Filter prefix from list results (#328) 2018-09-09 09:34:23 +03:00
JT Olio
e0226790e6
miniogw: simplify output (#330) 2018-09-08 12:53:52 -06:00
Kaloyan Raev
2a1a52acc2 Non-recursive listing in Minio Gateway (#327)
* Non-recursive listing in Minio Gateway

* Cleaner code

* Fix tests
2018-09-08 11:10:58 -06:00
Egon Elbre
0a2839b2d7
Add IsPrefix support to piecestore (#313) 2018-09-07 17:20:15 +03:00
Cameron
6251b8e090 extract bucketstore code from NewGateway into helper (#300)
* extract bucketstore code from NewGateway into helper

* fix lines breaks
2018-09-05 19:03:20 +03:00
Cameron
ea9f5b4e63
CopyObject (#242)
* add CopyObject method

* use utils.LogClose

* extract common code from GetObject to getObject helper

* remove rr.Range from getObject helper, create helper putObject

* return rr, err in getObject helper

* extract code from PutObject into putObject helper

* remove commented out text

* remove other commented out code

* WIP trying to get storj cp command to work with copyObject

* fix typo in rb and now it works

* use rr.Size() instead of srcInfo.Size

* Revert "WIP trying to get storj cp command to work with copyObject"

This reverts commit e256b9f9a0fda728d41eb5b9d7a98b5446825842.

* add CopyObject test

* rebase and fix merge conflicts

* check error in gateway-storj test

* fix typo
2018-08-29 11:00:05 -04:00
Alexander Leitner
899e1e68f1
Add functions for signing and verifying during bandwidth exchange (#246)
* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* wip

* wip

* actual signatures in tests

(cherry picked from commit 1464853b737f1d712d64fbf90147f535525c8fd9)

* bugfixing

* Generate private key in example

* Generate signatures for pieceranger tests

* Update examples to use TLS

* Use private key from identity inside of example

* Use crypto.PrivateKey interface

* Change err name in defers

* Pass tests

* Pass identity Key to PSClient

* Get tests passing on travis

* Resolve linter complaints
2018-08-27 14:35:27 -04:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
Maximillian von Briesen
553def620a TLS Support (#252)
* begin adding tls

* remove incomplete line in gw/main.go

* identity fixes+:

+ fix `peertls.NewCert` public key issue
+ fix `peertls.verfiyChain` issue
+ fix identity dial option
+ rename `GenerateCA` to `NewCA` and `generateCAWorker` to `newCAWorker` for better consistency/convention

* use pdbclient instead of pointerdb in miniogw

* fix tests

* go fmt

* make review changes

* modify how context.Background() is used

* more context stuff
2018-08-23 22:01:03 -06:00
James Hagans
fdfa6e85c8 Streamstore impl of Stor interface (#191)
* first stab at PUT

* only PUT

* working on PUT

* Put with LimitReader

* start of Get

* reorder of files and proto meta

* working on Meta

* working on Meta

* add aware limit reader

* add size from segment put

* rm if for eof

* update to proto meta

* update gen proto file

* working on get

* working on get

* working on get

* working on list

* working on delete

* working on list

* working on meta method

* fix merge error and working on feedback from PR

* update to proto file

* rm size tuple

* mv eof limit reader to new file

* add toMeta

* rm varible names

* add updates from PR feedback

* updates from PR feedback

* updates from PR feedback

* add toMeta size based on total size

* update toMeta size calculation

* rm passthrough

* add default to config for segment size

* fix get method ranger bug

* add object support for nested stream proto

* rm nested stream meta data

* rm test for another PR
2018-08-23 21:56:38 -06:00
Dylan Lott
ce3f9c6699
Fix nil bucket error (#269) 2018-08-22 18:19:46 -06:00
JT Olio
9aee5efd99
pointerdb: separate client and server packages (#261)
* pointerdb: separate client and server packages

the reason for this is so that things the server needs (bolt, auth)
don't get imported if all you need is the client. will result in
smaller binaries and less flag definitions

* review comments
2018-08-22 09:07:00 -06:00
Kaloyan Raev
62521bb3f8 Fixes downloads in Minio Browser (#264) 2018-08-22 13:15:13 +02:00
JT Olio
fa390c9ec9
Mock overlay move (#245)
* move mock overlay from client to server

this doesn't really change much, but it does allow you to
run a standalone gateway against captain planet. it still does
not allow you to run a standalone gateway against a standalone
heavy client

* pointerdb: small error fixes

* some cleanups

* fix tests
2018-08-20 14:24:11 -04:00
Kaloyan Raev
d1c508e24a Avoid logging any minio error in miniogw logging wrapper (#240) 2018-08-20 13:55:39 -04:00
Kaloyan Raev
49577a47f8 Configure erasure-share-size to 1024 bytes (#241) 2018-08-20 13:54:30 -04:00
Kaloyan Raev
ae430b8c10 Use utils.LogClose for deferred closing in miniogw (#239) 2018-08-17 13:17:58 -04:00
Natalie Villasana
ff65663867
adds Bucket Store (#213)
* adds foundation for bucketStore

* adds prefixedObjStore to buckets package, adjusts gateway-storj accordingly

* fixes multi value assignment problems in gateway-storj

* fixes more multi value assignment errors in gateway-storj

* starts changing miniogw tests to accommodate buckets

* creates bucket store mock

* wip - fixing test cases in object tests

* adds get, put, and list object tests, comments out two test cases

* adds happy scenario tests for bucket methods

* fixes bug in list, removes redundant parts from gateway tests

* fixes nit

* Clean up tests from #188

* Fix bug with timestamp conversion in segment store

* fixes segments.Meta test

* Fix regression in listing objects in a bucket

* adds check to see if bucket is empty before deleting

* updates DeleteBucket test to account for empty/full bucket

* adds TODOs for DeleteBucket and MakeBucket for some cases, adjusts tests, filters out minio errors in logging.go

* adds checks for if buckets already exist or not in DeleteBucket and MakeBucket functions; adjusts tests

* adds BucketNotFound error check in bucket store, removes todo

* adds make_bucket to Travis test, updates boltdb client constructor to always create a bucket (table)
2018-08-16 10:32:28 -04:00
nfarah86
d396eb6be9
Bug/download (#216)
* fixed downloading empty file

* fixed formatting

* fixed error handling

* fixed cache_test.go to accomodate for error

* fixed error verbose messaging for wrong downlaods

* clean up
2018-08-14 08:22:29 -07:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
Jennifer Li Johnson
198f7fd506
uncomments oc=globalmockoverlay (#222) 2018-08-10 19:19:19 -04:00
Jennifer Li Johnson
5f8142cdcc
Integrating on disk routing table with kademlia (#166)
* adds comment

* runs deps

* adds print statements for debugging add node bkad

* more print statements

* removes bkad from routing and integrates on disk routing table

tests failing :(

wip

* removes testbootstrap

* kademlia_test not working

* adds kad tests back in

* Adds skips for tests broken due to wip kademlia
2018-08-09 15:20:39 -04:00
JT Olio
e66a0d289a
miniogw: make a Config.NewGateway helper, bypass overlay for now (#197) 2018-08-02 21:37:20 -06:00
JT Olio
30e3b503a4
captplanet: configure api key (#195) 2018-08-02 13:30:57 -06:00
Maximillian von Briesen
d934733c4d Move api key from pdb client method calls to pdb client constructor (#194)
* move api key from pdb client method calls to pdb client constructor

* update pdb example client
2018-08-02 13:09:50 -06:00
Natalie Villasana
a4bb03d0da
adds inline segment support for segmentstore (#162)
* begins adding inline segment support for segmentstore

* adds PeekThresholdReader struct plus Read and isInline methods

* moves PeekThresholdReader to peek.go, adds more simplified Read function

* adds PeekThresholdReader tests

* reverts Read function to earlier version, updates tests to use ReadFull instead

* Get function now handles inline type pointers

* adds correct type Size and ExpirationDate to inline segment

* fixes return value in Put func error condition

* moves thresholdBuf and Read tests into a table test

* adds border case test, fixes redundant parts

* passes sizedReader size to makeRemotePointer
2018-08-01 16:25:06 -04:00
JT Olio
389e65d742
miniogw: log all errors (#182)
* miniogw: log all errors

* tests added

* doc comment to satisfy linter

* fix test failure
2018-07-31 08:10:37 -06:00
Kaloyan Raev
7e136db9cf
Cleanup metadata across layers (#180)
* Cleanup metadata across layers

* Fix pointer db tests
2018-07-30 21:57:50 +03:00
JT Olio
d986c337aa streamstore: add passthrough implementation (#176)
this doesn't implement streamstore, this just allows us to try and
get the june demo working again in the meantime
2018-07-27 12:13:08 -06:00
aligeti
ff52a1e7e1
Minio integration with Object store (#156)
* initial WIP integration with Object store

* List WIP

* minio listobject function changes complete

* Code review changes and work in progress for the mock objectstore unit testing cases

* Warning fix redeclaration of err

* Warning fix redeclaration of err

*  code review comments & unit testing inprogress

* fix compilation bug

* Fixed code review comments & added GetObject Mock test case

* rearraged the mock test file and gateway storj test file in to the proper directory

* added the missing file

* code clean up

* fix lint error on the mock generated code

* modified per code review comments

* added the PutObject mock test case

* added the GetObjectInfo  mock test case

* added listobject mock test case

* fixed package from storj to miniogw

* resolved the gateway-storj.go initialization merge conflict
2018-07-27 08:34:40 -04:00
JT Olio
45a3c2e974
pkg/provider: with pkg/provider merged, make a single heavy client binary, gateway binary, and deprecate old services (#165)
* pkg/provider: with pkg/provider merged, make a single heavy client binary and deprecate old services

* add setup to gw binary too

* captplanet: output what addresses everything is listening on

* revert peertls/io_util changes

* define config flag across all commands

* use trimsuffix
2018-07-26 08:21:35 -06:00
JT Olio
5f6607935b
captplanet (#159)
* captplanet

I kind of went overboard this weekend.

The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!

To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.

Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.

A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
 * Kademlia
 * OverlayCache
 * PointerDB
 * StatDB
 * PieceStore
 * etc.

A Provider is a collection of Responsibilities that
share an Identity, such as:
 * The heavy client
 * The farmer
 * The gateway

An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.

Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.

To make a Responsibility, one should create a "config"
struct, such as:

```
type Config struct {
  RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
  SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```

To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.

cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.

Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
 * cmd/captplanet/farmer/main.go (a new farmer binary)
 * cmd/captplanet/hc/main.go (a new heavy client binary)
 * cmd/captplanet/gw/main.go (a new minio gateway binary)

Usage:

```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```

Configuration is placed by default in `~/.storj/capt/`

Other changes:

 * introduces new config structs for currently existing
   Responsibilities that conform to the new Responsibility
   interface. Please see the `pkg/*/config.go` files for
   examples.

 * integrates the PointerDB API key with other global
   configuration via flags, instead of through environment
   variables through viper like it's been doing. (ultimately
   this should also change to use the PointerDB config
   struct but this is an okay shortterm solution).

 * changes the Overlay cache to use a URL for database
   configuration instead of separate redis and bolt config
   settings.

 * stubs out some peer identity skeleton code (but not the
   meat).

 * Fixes the SegmentStore to use the overlay client and
   pointerdb clients instead of gRPC client code directly

 * Leaves a very clear spot where we need to tie the object to
   stream to segment store together. There's sort of a "golden
   spike" opportunity to connect all the train tracks together
   at the bottom of pkg/miniogw/config.go, labeled with a
   bunch of TODOs.

Future stuff:

 * I now prefer this design over the original
   pkg/process.Service thing I had been pushing before (sorry!)

 * The experience of trying to have multiple farmers
   configurable concurrently led me to prefer config structs
   over global flags (I finally came around) or using viper
   directly. I think global flags are okay sometimes but in
   general going forward we should try and get all relevant
   config into config structs.

 * If you all like this direction, I think we can go delete my
   old Service interfaces and a bunch of flags and clean up a
   bunch of stuff.

 * If you don't like this direction, it's no sweat at all, and
   despite how much code there is here I'm not very tied to any
   of this! Considering a lot of this was written between midnight
   and 6 am, it might not be any good!

* bind tests
2018-07-24 10:08:28 -06:00
Cameron
6463b87ebe
Errcheck (#133)
* add errcheck

* fixed linter errors

* fixes

* errcheck fixes in pkg/paths

* Fix errchecks in PieceID.Derive

* Fix ecclient tests

* Move closeConn a little bit above in the exectution flow

* fix new lint errors

* Fatalf -> Printf

* address eclipsed errors

* rename err to closeErr

* rename err to closeErr for differentiation
2018-07-16 15:22:34 -04:00
aligeti
8d8350fea7 Implement miniogw in terms of ObjectStore (#117)
* port changes

* Task monitor and setup merge from the staging

* Restructure + additional interface

* Add NewOverlayClient

* integrated DHT client interface

* added test for interface

* PR comments addressed

* lint issue

* added generated protobuf

* adding new interface

* added the interface framework

* deleted file

* fixes compilation errors and integrates new dhtcclient interface

* merged netstat latest changes and dht new interface chagnes

* fixed the address's port

* adding comments

* PR comments addressed

* netclient interface dial method added

* rename and integrated transportclient with minio gateway

* rename and code clean up

* made changes based on the Dennis's changes on the kad-client

* Code review comment changes based on kaloyan review comments

* reverted the changes to be similar to master

* removed unused file

* renamed to transportclient

* added the review changes

* store the address of the client

* updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters

* updated the test case to test the bad address passed condition

* updated the code per code review comments

* Bolt backed overlay cache (#94)

* wip

* add separate `Process` tests for bolt and redis-backed overlay

* more testing

* fix gitignore

* fix linter error

* goimports goimports GOIMPORTS GoImPortS!!!!

* fix port madness

* forgot to add

* add `mux` as handler and shorten context timeouts

* gofreakingimports

* fix comments

* refactor test & add logger/monkit registry

* debugging travis

* add comment

* Set redisAddress to empty string for bolt-test

* travis experiment

* refactoring tests

* Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache

* Automatically build, tag and push docker images on merge to master (#103)

* port changes

* build overlay on successful merge to master

* fixes to Makefile

* permissions

* dep ensure

* gopath

* let's try vgo

* remove dep

* maybe alpine is the issue

* tagging go version on build

* stupid vgo

* vgo

* adding tags to push

* quotes

* local linting fixes & stupid travis

* prepend storjlabs to docker tag (#108)

* port changes

* fixing tag name

* Use continue instead of return in table tests (#106)

I did a dumb mistake for some of the table tests, which made some of the
test cases not being executed.

* pkg/kademlia tests and restructuring (#97)

* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* add changes from kad demo

* PR comments addresses

* go func

* force travis build

* fixed merge conflicts

* fixed merge conflicts

* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests

* linter issues

* linting issues

* fixed merge conflicts

* linter is stupid

* Coyle/docker fix (#109)

* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge branch 'master' of https://github.com/storj/storj

* fixing tag name

* no idea

* testing

* changes

* testing on travis

* testing

* changes to travis build

* new approach

* Merge branch 'master' into coyle/docker-fix

* hardcode version (#111)

* hardcode version

* adding coveralls / code coverage  (#112)

* adding coveralls

* adding code coverage badge

* fixing badges

* verbose

* swap tests and coverage

* extra line

* maybe

* maybe

* moar

* gover maybe

* testing

* cleanup

* protos/netstate: remove stuff we're not using (#100)

* protos/netstate: remove stuff we're not using

* protos/netstate: add metadata field for segmentstore

* fix netstate client test

* pkg/process: start replacing pkg/process with cobra helpers (#98)

* Implement psclient interface (#107)

* Implement psclient interface

* Add string method to pieceID type

* try to fix linter errors

* Whoops missed an error

* More linter errors

* Typo

* Lol double typo

*  Get everything working, begin adding tests for psclient rpc

* goimports

* Forgot to change the piecestore cli when changed the piecestore code

* Fix CLI

* remove ID length, added validator to pieceID

* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer

* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser

* Forgot to remove offset

* Added message upon successful store

* Do that thing dennis and kaloyan wanted

* goimports

* Make closeConn a part of the interface for psclient

* Use interface

* Removed uneccessary new lines

* goimport

* Whoops

* Actually we don't want to use the interface in Piece Ranger

* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples

* Make comments look nicer

* modified transport client based on the the design discussion

* modified transport client based on the the design discussion

* added the as discussed connection cache interface functionality

* added the as discussed connection cache interface functionality

* transport client changes

* transport client per code review changes

* per the code review comments

* transport client incorporates review comments

* fixes lint warnings

* lint warning fixes....client interface has to be Client

* initial draft of Objectstore

* transport client review changes

* client.go changes

* transport.go changes

* added test case

* added test cases

* streams iface

* comment fix

* object store changes

* comment fix

* initialized the objectstore in gw

* Added PutObject with test support for encryption file

* added object store test cases

* tested & integrated the objectstore with miniogw

* handled the ranger and paths

* indentation change

* Kalyon's code review comments resolution implemented after the 30min code review meeting

* Compilation error fix

* fixes the tavis build warnings

* corrects the ListObject  return type to be slice of slice

* corrects the ListObject  return type to be slice of slice

*  added the serialization using protobuf

* added the unmarshalling of data in getobject()

* Jt's Review comments

* Kaloyan's review comments, moved the unmarshalling logic and other minor code indentation fixes

* more code reivew

* more code reivew

* Changes the expiration time to zeroTime and added error check in putObject function

* Changes the expiration time to zeroTime and added error check in putObject function

* minor warning fix- had to add a comment and fix the wording

* added a TODO comment per kaloyan

* code clean up removed unused variables
2018-07-12 11:29:02 -06:00
Kaloyan Raev
d8f1ec1db6
ECClient (#110)
* WIP ECClient

* Get returns RangeCloser

* Introduce RedundancyStrategy

* Constructor takes max buffer memory

* Remove unnecessary NopCloser wrapper

* Added telemetry

* Tests

* Adapt to PSClient from master

* Decode should report error if empty rrs map is passed

* collectErrors helper

* Move to /pkg/storage

* Move to /pkg/storage/ec

* Rename ecclient.go to client.go

* Better logging

* Rename ec.ECClient to ec.Client

* Fix some test execution

* Adopt Transport Client from master
2018-07-03 11:35:01 +03:00
aligeti
fa65f449ed
light client upload file support -V3 gateway (#57)
* integrated eestream's serve-pieces functionality

* added ref to http request function

* created dummy bucket list

* Initialized the buckets with files with hardcoded sample data

* supports upload Object(s)

* uploads to corresponding folders

* code cleanup for review

* updated based on code review comments

* updates based on missed code review comments

* updated with review comments

* implemented review comments

* merged latest and tested

* added filepath.Join()

* updates based on the comments

* fixes the eestreamer parameter due to merge
2018-06-01 08:51:13 -04:00
JT Olds
fc7cfde9f7 pkg/miniogw: a minio-based s3 gateway for storj. just stubs right now (#34) 2018-05-08 19:49:54 -04:00