Commit Graph

1170 Commits

Author SHA1 Message Date
aligeti
26f68fa774 Transport Client (#89)
* port changes

* Task monitor and setup merge from the staging

* Restructure + additional interface

* Add NewOverlayClient

* integrated DHT client interface

* added test for interface

* PR comments addressed

* lint issue

* added generated protobuf

* adding new interface

* added the interface framework

* deleted file

* fixes compilation errors and integrates new dhtcclient interface

* merged netstat latest changes and dht new interface chagnes

* fixed the address's port

* adding comments

* PR comments addressed

* netclient interface dial method added

* rename and integrated transportclient with minio gateway

* rename and code clean up

* made changes based on the Dennis's changes on the kad-client

* Code review comment changes based on kaloyan review comments

* reverted the changes to be similar to master

* removed unused file

* renamed to transportclient

* added the review changes

* store the address of the client

* updates per the code review comments, changes-> added error retry connection attempt logic, added error conditions including nil parameters

* updated the test case to test the bad address passed condition

* updated the code per code review comments

* Bolt backed overlay cache (#94)

* wip

* add separate `Process` tests for bolt and redis-backed overlay

* more testing

* fix gitignore

* fix linter error

* goimports goimports GOIMPORTS GoImPortS!!!!

* fix port madness

* forgot to add

* add `mux` as handler and shorten context timeouts

* gofreakingimports

* fix comments

* refactor test & add logger/monkit registry

* debugging travis

* add comment

* Set redisAddress to empty string for bolt-test

* travis experiment

* refactoring tests

* Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache

* Automatically build, tag and push docker images on merge to master (#103)

* port changes

* build overlay on successful merge to master

* fixes to Makefile

* permissions

* dep ensure

* gopath

* let's try vgo

* remove dep

* maybe alpine is the issue

* tagging go version on build

* stupid vgo

* vgo

* adding tags to push

* quotes

* local linting fixes & stupid travis

* prepend storjlabs to docker tag (#108)

* port changes

* fixing tag name

* Use continue instead of return in table tests (#106)

I did a dumb mistake for some of the table tests, which made some of the
test cases not being executed.

* pkg/kademlia tests and restructuring (#97)

* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* add changes from kad demo

* PR comments addresses

* go func

* force travis build

* fixed merge conflicts

* fixed merge conflicts

* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests

* linter issues

* linting issues

* fixed merge conflicts

* linter is stupid

* Coyle/docker fix (#109)

* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge branch 'master' of https://github.com/storj/storj

* fixing tag name

* no idea

* testing

* changes

* testing on travis

* testing

* changes to travis build

* new approach

* Merge branch 'master' into coyle/docker-fix

* hardcode version (#111)

* hardcode version

* adding coveralls / code coverage  (#112)

* adding coveralls

* adding code coverage badge

* fixing badges

* verbose

* swap tests and coverage

* extra line

* maybe

* maybe

* moar

* gover maybe

* testing

* cleanup

* protos/netstate: remove stuff we're not using (#100)

* protos/netstate: remove stuff we're not using

* protos/netstate: add metadata field for segmentstore

* fix netstate client test

* pkg/process: start replacing pkg/process with cobra helpers (#98)

* Implement psclient interface (#107)

* Implement psclient interface

* Add string method to pieceID type

* try to fix linter errors

* Whoops missed an error

* More linter errors

* Typo

* Lol double typo

*  Get everything working, begin adding tests for psclient rpc

* goimports

* Forgot to change the piecestore cli when changed the piecestore code

* Fix CLI

* remove ID length, added validator to pieceID

* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer

* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser

* Forgot to remove offset

* Added message upon successful store

* Do that thing dennis and kaloyan wanted

* goimports

* Make closeConn a part of the interface for psclient

* Use interface

* Removed uneccessary new lines

* goimport

* Whoops

* Actually we don't want to use the interface in Piece Ranger

* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples

* Make comments look nicer

* modified transport client based on the the design discussion

* modified transport client based on the the design discussion

* added the as discussed connection cache interface functionality

* added the as discussed connection cache interface functionality

* transport client changes

* transport client per code review changes

* per the code review comments

* transport client incorporates review comments

* fixes lint warnings

* lint warning fixes....client interface has to be Client

* client.go changes

* transport.go changes

* added test case

* added test cases

* comment fix

* comment fix
2018-06-29 21:28:06 +03:00
Alexander Leitner
900f67e3d0 Implement psclient interface (#107)
* Implement psclient interface

* Add string method to pieceID type

* try to fix linter errors

* Whoops missed an error

* More linter errors

* Typo

* Lol double typo

*  Get everything working, begin adding tests for psclient rpc

* goimports

* Forgot to change the piecestore cli when changed the piecestore code

* Fix CLI

* remove ID length, added validator to pieceID

* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer

* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser

* Forgot to remove offset

* Added message upon successful store

* Do that thing dennis and kaloyan wanted

* goimports

* Make closeConn a part of the interface for psclient

* Use interface

* Removed uneccessary new lines

* goimport

* Whoops

* Actually we don't want to use the interface in Piece Ranger

* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples

* Make comments look nicer
2018-06-27 21:42:54 +03:00
JT Olio
613d1c25e2 pkg/process: start replacing pkg/process with cobra helpers (#98) 2018-06-27 11:02:49 +03:00
JT Olio
462c0758d1 protos/netstate: remove stuff we're not using (#100)
* protos/netstate: remove stuff we're not using

* protos/netstate: add metadata field for segmentstore

* fix netstate client test
2018-06-26 11:18:05 -04:00
Dennis Coyle
a5ff5016c9
pkg/kademlia tests and restructuring (#97)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* add changes from kad demo

* PR comments addresses

* go func

* force travis build

* fixed merge conflicts

* fixed merge conflicts

* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests

* linter issues

* linting issues

* fixed merge conflicts

* linter is stupid
2018-06-22 09:33:57 -04:00
Kaloyan Raev
35d5761ee6
Use continue instead of return in table tests (#106)
I did a dumb mistake for some of the table tests, which made some of the
test cases not being executed.
2018-06-22 12:23:19 +03:00
Bryan White
c6f8144221 Bolt backed overlay cache (#94)
* wip

* add separate `Process` tests for bolt and redis-backed overlay

* more testing

* fix gitignore

* fix linter error

* goimports goimports GOIMPORTS GoImPortS!!!!

* fix port madness

* forgot to add

* add `mux` as handler and shorten context timeouts

* gofreakingimports

* fix comments

* refactor test & add logger/monkit registry

* debugging travis

* add comment

* Set redisAddress to empty string for bolt-test

* travis experiment

* refactoring tests

* Merge remote-tracking branch 'upstream/master' into bolt-backed-overlay-cache
2018-06-20 10:28:46 -04:00
JT Olio
5ee6b76d41 internal/test: switch errors to error classes (#96)
* internal/test: switch errors to error classes

if you construct an error directly at package init time, you
won't get useful stack traces or anything. zeebo/errs expects that
you always construct an error (.New) when the error actually
happens. instead, you call Class at init time, then use (Class).Has
to test for error type membership.

* fix linter

* fix test
2018-06-19 20:03:46 +02:00
Kaloyan Raev
045dae5f0d
Add context to Ranger.Range method (#99) 2018-06-19 18:59:09 +03:00
Dennis Coyle
9e40141cbc
Coyle/kad client (#91)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Restructure + additional interface

* Add NewOverlayClient

* added test for interface

* PR comments addressed

* lint issue

* added generated protobuf

* fixed merge conflicts

* adding comments

* PR comments addressed
2018-06-19 10:00:15 -04:00
Kaloyan Raev
7c8d6f8081
Add error to the return values of Ranger.Range method (#90)
* Add error to the return values of Ranger.Range method

* Addressing code review feedback

* Set Content-Type header only if ctype is not empty
2018-06-18 19:46:49 +03:00
Bryan White
1ebd66d880 Cleanup & bolt-backed cache (#87)
* wip post-demos cleanup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* Reorganize:

storage
├── common
│   └── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common
│   ├── common.go `package storage`
│   └── testing.go `package storage` <--
└── ...

internal
├── app
│   └── cli
├── pkg
│   └── readcloser
└── test
    └── util.go `package test` <--

* remove comment

* add and use goimports

* add test types & fix some lint errors

* better typing

* fixing linter issues/comments

* goimports

* goimports

* more linter fixes; replace panic with NoError assertions in tests

* fix typo/more linter errors

* moar better linter fixes

* even moar better linter fixes

* linter

* add localPort back

* fixing exports, imports, and add comments
2018-06-13 14:22:32 -04:00
JT Olio
258d5fec44 Piecestore: demo week work (#82)
* piecestore: connect to kad

* piecestore: Linter errors

* piecestore: added pstore config command line utility

* piecestore: removed main.go, implement methods and structs

* piecestore: Import Cam's config code into piecestore-farmer

* piecestore: moving farmer to urfavecli

* piecestore: added create command

* piecestore: Removed old config, added server start code to cli

* piecestore: Get server code working

* piecestore: Changed default dir for storing piece store data; added ttl to config

* piecestore: Generate id; add bootstrap ip for kad

* piecestore: Separate kad port and server port

* piecestore: goimports

* piecestore: Removed print

* piecestore: use pkg/process

* piecestore: Better config file

* piecestore: base58 encode for id

* piecestore: base58 encode and clean up cli

* piecestore: Typo

* piecestore: removed unnecessary variable

* piecestore: Fixed more typos

* piecestore: Place data in a directory based on nodeid

* piecestore: base58 encode instead

* piecestore: Add dependency to go.mod

* piecestore: Fix typo in rpc server start; clear data on failed piece upload
2018-06-12 11:03:45 -04:00
JT Olio
3b899724f5 Remove storj-v2-cli (#85)
* rename storj cli to storj-v2-cli

* Remove storj-v2-cli
2018-06-11 17:40:29 -04:00
Dennis Coyle
e65e115222 port changes (#80) 2018-06-07 10:15:57 -04:00
Dennis Coyle
c0446e21ae change bootstrap hardcoded value (#78)
approved for demo
2018-06-07 09:42:44 -04:00
Dylan Lott
325a70d514 Cache (#67)
* add reference to dht to overlay client struct

* wip

* wip

* Implement FindNode

* get nodes

* WIP

* Merge in Dennis kademlia code, get it working with our code

* ping and moar

* WIP trying to get cache working with kademlia

* WIP more wiring up

* WIP

* Update service cli commands

* WIP

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* Added initial network overlay explorer page

* Updating and building with dockerfile

* Working on adding bootstrap node code

* WIP merging in dennis' code

* WIP

* connects cache to pkg/kademlia implementation

* WIP redis cache

* testing

* Add bootstrap network function for CLI usage

* cleanup

* call bootstrap on init network

* Add BootstrapNetwork function to interface

* Merge in dennis kad code

* WIP updates to redis/overlay client interface

* WIP trying to get the DHT connected to the cache

* go mod & test

* deps

* Bootstrap node now setting up correctly

- Need to pass it through CLI commands better

* WIP adding refresh and walk functions, added cli flags

- added cli flags for custom bootstrap port and ip

* PR comments addressed

* adding FindStorageNodes to overlay cache

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort

* updates to findstoragenodes response and request

* WIP merge in progress, having issues with a panic

* wip

* adjustments

* update port for dht bootstrap test

* Docker

* wip

* dockerfile

* fixes

* makefile changes

* Update port in NewKademlia call

* Update local kademlia DHT config

* kubernetes yaml

* cleanup

* making tests pass

* k8s yaml

* lint issues

* Edit cli flags to allow for configurable bootstrap IP and Port args

* cleanup

* cache walking the network now

* Rough prototype of Walk function laid out

* Move walk function into bootstrap function

* Update dht.go

* changes to yaml

* goimports
2018-06-05 17:06:37 -04:00
JT Olio
6be2baf9f9
pkg/piecestore: use readersource (#75)
* pkg/piecestore: use readersource

* pkg/piecestore: fix linting
2018-06-05 08:00:48 -06:00
Kaloyan Raev
3a9ec8b680
Fix the position of offset and length in gRPC Ranger (#74) 2018-06-05 08:27:58 -04:00
Dennis Coyle
e22810da15
Kademlia Package (#58)
* wip

* wip

* get nodes

* ping and moar

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* testing

* cleanup

* go mod & test

* deps

* PR comments addressed

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort
2018-06-05 07:48:19 -04:00
JT Olio
836a7d2b1d
change metric sending to be every minute during dev (#73) 2018-06-04 19:50:41 -06:00
Alexander Leitner
db17557d0d
piece store accept data when size isn't specified (#72)
* added spawn scripts

* wip

* Make server not require length

* Added test for 0 length

* Added test for accepting length of 0

* linter errors

* Make store not take anything more than an id and maybe a ttl if you want
2018-06-04 17:30:42 -04:00
nfarah86
f9af3b6ee4
Mutex/auth grpc (#50)
* initial commit for PUT request authorization

* inital auth for put request

* auth. request working for all req

* deleted library

* removed .db files

* work in progress for modifying test suite to accomodate credentials

* modified tests

* gofmt, fixed code based on suggestions; test passed

* gofmt again

* merged nat's update on pointers, passed tests, cleanup from git rebase

* fixed fmt

* fixed fmt on tests

* work in progress

* reduced code

* fixed naming conventions

* added line in code

* fixed server bug, merged new code to server, added env

* fixed linter; getting cright issues on piecestore

* added comments for what passes on the creds
2018-06-04 09:45:07 -07:00
Alexander Leitner
040edbb7f3
DetermineID in piecestore (#71)
* added spawn scripts

* Determine random id for storing

* Moved determine id to rpc example

* Added tests

* Better test

* goimports

* Updated tests

* Fix typos
2018-06-04 12:35:31 -04:00
JT Olio
0e8391c30f
pkg/process: add hw metrics (#70) 2018-06-04 10:07:57 -06:00
Kaloyan Raev
916e0b0ee0 gRPC Ranger (#44)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Add example of http server using piece store

* Changed piecestore code to make it more optial for error handelling

* Merged with piecestore

* Added missing package

* Forgot io import

* gofmt

* gofmt

* Forgot io

* Make path by hash exported

* updated to simplify again whoops

* Updated server to work real good

* Forgot ampersand

* Updated to match FilePiece

* Merged in cam's delete code

* Remove unused io

* Added RPC code

* Give the download request a reader

* Removed http server stuff; changed receive stream to say io.reader

* Added expiration date to shardInfo

* gRPC Ranger

* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions

* Adapt to latest changes in piece store rpc api

* added ttl info request

* Initialize grpcRanger type with named fields

* Move scripts to http server pr; added close method for Retrieve api

* added rpc server tests for getting piece meta data and retrieval routes

* Adapt to PieceStreamReader now being a ReadCloser

* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf

* Imported cams test

* Bump gometalinter deadline

* Adapt to package name changes

* Remove Garbage

* Adapt to latest changes in piece store rpc api

* NewCustomRoute constructor to allow mocking the gRPC client

* Name struct values in constructor.
2018-06-02 22:20:17 -06:00
Alexander Leitner
ff8e191a9a
Added rpc server (#40)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Add example of http server using piece store

* Changed piecestore code to make it more optial for error handelling

* Merged with piecestore

* Added missing package

* Forgot io import

* gofmt

* gofmt

* Forgot io

* Make path by hash exported

* updated to simplify again whoops

* Updated server to work real good

* Forgot ampersand

* Updated to match FilePiece

* Merged in cam's delete code

* Remove unused io

* Added RPC code

* Give the download request a reader

* Removed http server stuff; changed receive stream to say io.reader

* Added expiration date to shardInfo

* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions

* added ttl info request

* Move scripts to http server pr; added close method for Retrieve api

* added rpc server tests for getting piece meta data and retrieval routes

* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf

* Imported cams test

* Bump gometalinter deadline

* WIP adding tests

* added tests for store and delete routes

* Add changes as requested by Kaloyan, also cleaned up some code

* Get the code actually working whoops

* More cleanup

* Separating database calls from api.go

* need to rename expiration

* Added some changes requested by JT

* Fix read size

* Fixed total amount to read

* added tests

* Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages

* Moved rpc client and server to piece store

* Moved piecestore protobuf to the /protos folder

* Cleaned up messages

* Clean up linter errors

* Added missing sqlite import

* Add ability to do iterative reads and writes to pstore

* Incrementally read data

* Fix linter and import errors

* Solve linter Error

* Change return types

* begin test refactor

* refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go

* Cleaning up tests

* Added retrieve tests

* refactored delete tests

* Deleted old tests

* Updated cmd/piecestore to reflect changes to piecestore

* Refactored server tests and server/client store code

* gofmt

* WIP implementing TTL struct

* Read 4k at a time when Retrieving

* implemented ttl struct

* Accidentally removed fpiece dependency?

* Double resolve merge conflict

* Moved client to the examples since it is an example

* Change hash to id in protobuf. TODO: Change client and server to reflect these changes

* Update client and protobuf

* changed hash to id

* Handle eof properly in retrieve

* Id -> ID

* forgot to change import for client after moving

* Moved client and server main to examples

* Made changes requested by JT

* checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close()

* Print not fatal

* Handle overflow when reading from server

* added const IDLength

* removed types from comments

* Add reader/writer for download data from and uploading data to server

* goimports and comments

* fixed nits, casts, added OK const, DBCleanup now exits program on error

* Add stream reader and writer for server

* Fix errors

* i beforee except after c lol

* customizable data dir

* Forgot to rename variable

* customizable data dir

* Handle closing of server stream properly

* linter

* pass on inserting the same data twice

* linter

* linter

* Do the easy things JT asked for

* Handle overflow from reads properly; handle custom db path

* Handle overflow for server stream read; TODO Combine server and client stream reads

* Allow for TTL of 0 to stay forever

* Make Client cleaner for user
2018-06-02 14:14:59 -04:00
JT Olio
f2407727d4 minio/netstate/piecestore: integrate with pkg/process for telemetry reporting (#65)
* pkg/piecestore: fix

* minio/netstate/piecestore: integrate with pkg/process for telemetry reporting
2018-06-01 10:07:58 -04:00
aligeti
fa65f449ed
light client upload file support -V3 gateway (#57)
* integrated eestream's serve-pieces functionality

* added ref to http request function

* created dummy bucket list

* Initialized the buckets with files with hardcoded sample data

* supports upload Object(s)

* uploads to corresponding folders

* code cleanup for review

* updated based on code review comments

* updates based on missed code review comments

* updated with review comments

* implemented review comments

* merged latest and tested

* added filepath.Join()

* updates based on the comments

* fixes the eestreamer parameter due to merge
2018-06-01 08:51:13 -04:00
Alexander Leitner
5e97cf7a2e Moved filepiece into storj (#66)
* Moved filepiece into storj

* Fix linter errors

* Seek comment for linter

* gofmt/golinter accidentally removed import

* Fix small typos

* Use the weird iota. P cool dude ✌️

* Do things the cool way

* Changes requested by kaloyan

* didn't need test main
2018-05-31 22:42:45 +03:00
JT Olio
4c4cf3c34e pkg/process: update collector address (#64) 2018-05-30 19:15:04 -04:00
Kaloyan Raev
ad9b7e01a6
Path encryption library (#61)
* Path encryption library

* Use base64 instead of hex encoding

* Prepend version number to encrypted path segments

* Remove redundant var alias

* Simplified returns

* wrap errors
2018-05-30 18:33:27 +03:00
Kaloyan Raev
ed72a5e542
Optimize eestream upload for slow piece storage nodes (#48)
* Buffered eestream EncodeReader

* Extracted fillBuffer() helper function

* Slow channels will be closed if there are still at least k fast channels

* Doc comment for maxBufferMemory

* Use timer more efficiently

* Timer initialization should be inside the for-loop

* Parallel copy of encoded data to reader buffer channels

* Transfer input read errors to output encoded readers

* minimum and optimum thresholds

* Use time.AfterFunc

* Simplify error handling in constructor
2018-05-30 18:27:09 +03:00
JT Olio
22b1fe4e21 pkg/process: add pkg/telemetry plumbing (#47)
* pkg/process: add pkg/telemetry plumbing

* pkg/process: add debug endpoints

* fix linting
2018-05-30 10:03:44 -04:00
Kaloyan Raev
06bd257092 Remove randomly failing eestream test case (#62) 2018-05-29 20:58:41 -06:00
Natalie Villasana
6723064bfb
Enables netstate service to save pointers (#49)
* adds pointer to netstate proto file

* generated updated netstate proto

* changes boltdb netstate to save pointers as values

* updates netstate Put to save Pointers, updates client example to put a pointer, adds grpc status errors, updates tests, changes boltdb 'File' struct to 'PointerEntry'

* updates netstate client example and client test to save pointers, updates netstate List and Delete

* begins adding netstate-http tests

* removes netstate http service

* re-adds netstate auth

* updates boltdb netstate test

* changes encrypted_unencrypted_size from int64 to bytes in netstate proto

* updates READMEs
2018-05-29 22:47:40 -04:00
Dennis Coyle
d6ddb65bf8 basic structure (#39)
* wip - basic structure

* cleanup

* zap logger

* handling errors | using urfav/cli

* remove deadcode

* move NewLogger to utils

* single process method

* copyright

* more generic Process

* simplified

* added NewServer Back
2018-05-16 12:47:59 -06:00
Alexander Leitner
4b03c08cf6 Added piecestore (#36)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Changed piecestore code to make it more optial for error handelling

* Added missing package

* Forgot io import

* gofmt

* Make path by hash exported

* updated to simplify again whoops

* Forgot ampersand

* Updated to match FilePiece

* Change store to use io.CopyN

* Clear golinter errors

* Updated go.mod

* Updated go.mod
2018-05-16 06:40:44 -06:00
aligeti
0376dc4bd2 AES GCM implementation and unit test code (#19)
* AES GCM implementation and unit test code

* modified and tested per the code review comments

* modified and tested per the code review comments

* updated to return err

* removed the debug printf commented code

* support of aes-gcm

* updated the renaming convention per GOLANG coding standards

* Initial Go & C biniding with libstorj

* Initial GO & C bindings with libstorj

* fixing the callback

* moved a millimeter :-) on c to go to gone....

*  added error condition, per review comment

* removed the .idea directory and also movie.avi file

* deleting files not to be in this pull request

* Revert "deleting files not to be in this pull request"

This reverts commit 026b834fe00f6b20a7566e71973fe224c12f533f.

* deleting files not to be in this pull request

* resolving conflicts

* syncing the file to master

* Use aes gcm in eestream rs tests

* Use aes gcm in serve example

* Fixed comment
2018-05-15 18:11:03 +03:00
Natalie Villasana
1a06c2b0c7 adding grpc interface to network state (#33)
* adds proto files for netstate crud

* moves netstate grpc client lib into pkg/netstate where grpc netstate service is defined

* starts adding grpc client and server tests

* moves creation of grpc server into cmd/netstate/main.go, removes pkg/netstate/service.go, adds more client testing

* changed all 'Path' and 'Value' fields from strings to bytes, updated tests

* changes Get and Delete in proto file to receive 'requests' instead of 'file paths', adds tests for Get, List, and Delete

* changes netstate-routes to get 'fileValue' bytes not 'fileInfo'

* adds example rpc client in 'examples' and adds more specific debug logs

* adds readmes for netstate rpc services and updates netstate-routes
2018-05-14 18:31:26 -06:00
Kaloyan Raev
6f1c131028 Constructor for HTTP Ranger with known size (#43) 2018-05-14 14:34:08 -06:00
Kaloyan Raev
90ca28f7bc eestream avoid waiting for slow pieces (#30)
* WIP eestream avoid waiting for slow pieces

* Use non-blocking select to harvest all available inbufs

* Better way to check for elapsed time in tests

* Improve comment

* Determine EOF based on expected size

* Remove unused readerError type

* Configurable readers channel size

* Close properly decodedReader in tests

* Use context for properly closing the decodedReader

* Handle infectious errors using the new descendent classes

* Refactor decodedReader.Read() into helper functions

* Reenable TestRSErrors

* Test with Rangers

* Remove obsolete comment
2018-05-14 14:30:57 -06:00
Dennis Coyle
4be0503f4a DHT interfaces (#28)
* first pass at interface for dht

* wip

* comments addressed from @jtolds

* adjust GetBuckets

* missed comment

* addressed PR comments from @jtolds
2018-05-09 08:04:51 -06:00
JT Olds
fc7cfde9f7 pkg/miniogw: a minio-based s3 gateway for storj. just stubs right now (#34) 2018-05-08 19:49:54 -04:00
JT Olio
78989a2d5a fix hanging travis test 2018-05-08 17:16:16 -06:00
JT Olds
2ef8e581b2 fix travis, and get a clean lint (#32)
* fix travis

* lint fixes

* travis: enable tests

* fix some tests, disable a broken eestream test for now
2018-05-08 18:02:01 -04:00
Natalie Villasana
c73a4f9d68
Merge pull request #20 from coyle/clean-lint
Clean lint!
2018-05-07 14:14:40 -04:00
Dennis Coyle
5e9b3f7d6a merge conflicts 2018-05-07 14:04:53 -04:00
JT Olds
5be1da982f pkg/telemetry: telemetry package using zeebo/admission (#26)
* pkg/telemetry: use zeebo/admission

* pkg/telemetry: updated to use latest zeebo/admission changes
2018-05-07 14:03:40 -04:00
Kaloyan Raev
1c56a4cf5c Decoder can tolerate readers with unexpected EOF and random data (#27)
* Decoder can tolerate readers with unexpected EOF

* Return EOF if required number of inbufs are at EOF

* Use existing randData() to generate random data for tests

* Test case for io.ErrUnexpectedEOF

* Add TransformReaderSize constructor
2018-05-04 14:52:49 -06:00
Dennis Coyle
a51087a343 merge conflicts 2018-05-02 15:10:40 -04:00
JT Olds
4f40280c71 some small import cleanups, go generate overlay protos (#25)
checking in go generated stuff is recommended by
https://blog.golang.org/generate
2018-05-01 13:24:32 -04:00
Dennis Coyle
5d8d0cdda0 renamed ranger.Closer -> ranger.RangeCloser 2018-04-30 10:49:04 -04:00
Kaloyan Raev
8c96f4e5a5 Avoid extra Range requests for content type detection 2018-04-30 06:08:41 -06:00
Kaloyan Raev
4635621a4d Move helper structs inside functions 2018-04-30 06:06:25 -06:00
Kaloyan Raev
acce2bfc08 Reduce network latency by initializing Rangers in parallel 2018-04-30 06:06:25 -06:00
Dennis Coyle
bb8d4f8d22 cleanup 2018-04-25 10:55:26 -04:00
Kaloyan Raev
69239e9e17 Do not fail decoding on first read error (#17)
* Do not fail decoding on first read error

Try decoding with the rest successfully read inputs.

* some small code improvements

* Allocate map memory upfront
2018-04-23 20:17:50 -06:00
JT Olds
00854b9736
pkg/ranger: FileRanger (#18) 2018-04-23 20:16:34 -06:00
Dennis Coyle
ecbd5f08c3 Adding gRPC server/clients for overlay (#14)
* overlay proto

* grpc server and client

* fix import to storj

* tests

* change imports

* imports cleanup/comments

* PR comments addressed from @jtolds
2018-04-23 09:54:22 -06:00
Kaloyan Raev
f96cc9c955 HTTP Ranger (#11)
* HTTPRanger

* Migrate eestream library to use ReadCloser instead of Reader

* Use MultiReadCloser instead of MultiReader

* Adapt example cli cmds to latest serverbox changes

* Close LazyReadCloser only if only generated

* Close ReadClosers sequentially

* Close response body on unexpected status code

* Avoid double close of the pipe reader

* Better formatting for imports

* Exit instead of panicking
2018-04-17 07:39:14 -06:00
Dennis Coyle
2c08e56b20 fix import to storj 2018-04-16 10:09:01 -06:00
Dennis Coyle
1975b98eba grpc server and client 2018-04-16 10:09:01 -06:00
Kaloyan Raev
7fde8b908a
Light client implementation with get-info and list-buckets commands (#2)
* Light client implementation with get-info and list-buckets commands

* Fix client package name

* Fix go.mod to work with vgo

* Use single `fmt.Printf` for `get-info` output

* Use unnamed import to client package

* Simplify usage of sha256 and sha512 sums

* Remove obsolete test code

* Use helper structs for unmarshalling bridge info

* Remove LGPL license files and adjust copyright headers

* Use github.com/zeebo/errs

* Use httptest for test http server

* Use viper for env var management

* Nested struct for swagger

* Add github.com/zeebo/errs to go.mod

* More bucket tests

* word wrap long line

* Use zeebo/errs for crypto errors
2018-04-16 16:42:06 +03:00
JT Olio
c90143a58d eestream: make secretbox example harder to fall victim to nonce reuse 2018-04-16 06:52:23 -06:00
Kaloyan Raev
6c2c4b92e4
Merge pull request #1 from jtolds/master
merge eestream
2018-04-11 22:32:06 +03:00
JT Olio
d84757d49a pkg/ranger: fix linting errors 2018-04-11 08:05:41 -06:00
JT Olio
80fa10a06e pkg/eestream: initial 2018-04-11 07:46:34 -06:00
JT Olio
21d8a20037 ranger package 2018-04-11 06:55:40 -06:00
Dennis Coyle
b236d7dffb keep 2018-04-06 12:36:55 -04:00