Commit Graph

5303 Commits

Author SHA1 Message Date
JT Olio
f2407727d4 minio/netstate/piecestore: integrate with pkg/process for telemetry reporting (#65)
* pkg/piecestore: fix

* minio/netstate/piecestore: integrate with pkg/process for telemetry reporting
2018-06-01 10:07:58 -04:00
aligeti
fa65f449ed
light client upload file support -V3 gateway (#57)
* integrated eestream's serve-pieces functionality

* added ref to http request function

* created dummy bucket list

* Initialized the buckets with files with hardcoded sample data

* supports upload Object(s)

* uploads to corresponding folders

* code cleanup for review

* updated based on code review comments

* updates based on missed code review comments

* updated with review comments

* implemented review comments

* merged latest and tested

* added filepath.Join()

* updates based on the comments

* fixes the eestreamer parameter due to merge
2018-06-01 08:51:13 -04:00
Alexander Leitner
5e97cf7a2e Moved filepiece into storj (#66)
* Moved filepiece into storj

* Fix linter errors

* Seek comment for linter

* gofmt/golinter accidentally removed import

* Fix small typos

* Use the weird iota. P cool dude ✌️

* Do things the cool way

* Changes requested by kaloyan

* didn't need test main
2018-05-31 22:42:45 +03:00
JT Olio
4c4cf3c34e pkg/process: update collector address (#64) 2018-05-30 19:15:04 -04:00
Kaloyan Raev
ad9b7e01a6
Path encryption library (#61)
* Path encryption library

* Use base64 instead of hex encoding

* Prepend version number to encrypted path segments

* Remove redundant var alias

* Simplified returns

* wrap errors
2018-05-30 18:33:27 +03:00
Kaloyan Raev
ed72a5e542
Optimize eestream upload for slow piece storage nodes (#48)
* Buffered eestream EncodeReader

* Extracted fillBuffer() helper function

* Slow channels will be closed if there are still at least k fast channels

* Doc comment for maxBufferMemory

* Use timer more efficiently

* Timer initialization should be inside the for-loop

* Parallel copy of encoded data to reader buffer channels

* Transfer input read errors to output encoded readers

* minimum and optimum thresholds

* Use time.AfterFunc

* Simplify error handling in constructor
2018-05-30 18:27:09 +03:00
JT Olio
22b1fe4e21 pkg/process: add pkg/telemetry plumbing (#47)
* pkg/process: add pkg/telemetry plumbing

* pkg/process: add debug endpoints

* fix linting
2018-05-30 10:03:44 -04:00
Kaloyan Raev
06bd257092 Remove randomly failing eestream test case (#62) 2018-05-29 20:58:41 -06:00
Natalie Villasana
6723064bfb
Enables netstate service to save pointers (#49)
* adds pointer to netstate proto file

* generated updated netstate proto

* changes boltdb netstate to save pointers as values

* updates netstate Put to save Pointers, updates client example to put a pointer, adds grpc status errors, updates tests, changes boltdb 'File' struct to 'PointerEntry'

* updates netstate client example and client test to save pointers, updates netstate List and Delete

* begins adding netstate-http tests

* removes netstate http service

* re-adds netstate auth

* updates boltdb netstate test

* changes encrypted_unencrypted_size from int64 to bytes in netstate proto

* updates READMEs
2018-05-29 22:47:40 -04:00
Bryan White
9aeff0ab57 dep init (#56)
ignoring jenkins
2018-05-29 20:28:46 -06:00
Dennis Coyle
d6ddb65bf8 basic structure (#39)
* wip - basic structure

* cleanup

* zap logger

* handling errors | using urfav/cli

* remove deadcode

* move NewLogger to utils

* single process method

* copyright

* more generic Process

* simplified

* added NewServer Back
2018-05-16 12:47:59 -06:00
Alexander Leitner
4b03c08cf6 Added piecestore (#36)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Changed piecestore code to make it more optial for error handelling

* Added missing package

* Forgot io import

* gofmt

* Make path by hash exported

* updated to simplify again whoops

* Forgot ampersand

* Updated to match FilePiece

* Change store to use io.CopyN

* Clear golinter errors

* Updated go.mod

* Updated go.mod
2018-05-16 06:40:44 -06:00
nfarah86
6b49aed88a Mutex/process api key (#29)
* initial commit

* created httpreq struct

* checked against env key

* wrapped up comparing xapikey to flag input

* added readme

* added example and updated code with viper

* added ConstantTimeCompare function

* fixed spacing

* updated with comments, and updated readme

* updated main.go with setting env variable

* fixed linter errors
2018-05-15 15:30:32 -04:00
aligeti
0376dc4bd2 AES GCM implementation and unit test code (#19)
* AES GCM implementation and unit test code

* modified and tested per the code review comments

* modified and tested per the code review comments

* updated to return err

* removed the debug printf commented code

* support of aes-gcm

* updated the renaming convention per GOLANG coding standards

* Initial Go & C biniding with libstorj

* Initial GO & C bindings with libstorj

* fixing the callback

* moved a millimeter :-) on c to go to gone....

*  added error condition, per review comment

* removed the .idea directory and also movie.avi file

* deleting files not to be in this pull request

* Revert "deleting files not to be in this pull request"

This reverts commit 026b834fe00f6b20a7566e71973fe224c12f533f.

* deleting files not to be in this pull request

* resolving conflicts

* syncing the file to master

* Use aes gcm in eestream rs tests

* Use aes gcm in serve example

* Fixed comment
2018-05-15 18:11:03 +03:00
Natalie Villasana
1a06c2b0c7 adding grpc interface to network state (#33)
* adds proto files for netstate crud

* moves netstate grpc client lib into pkg/netstate where grpc netstate service is defined

* starts adding grpc client and server tests

* moves creation of grpc server into cmd/netstate/main.go, removes pkg/netstate/service.go, adds more client testing

* changed all 'Path' and 'Value' fields from strings to bytes, updated tests

* changes Get and Delete in proto file to receive 'requests' instead of 'file paths', adds tests for Get, List, and Delete

* changes netstate-routes to get 'fileValue' bytes not 'fileInfo'

* adds example rpc client in 'examples' and adds more specific debug logs

* adds readmes for netstate rpc services and updates netstate-routes
2018-05-14 18:31:26 -06:00
Kaloyan Raev
6f1c131028 Constructor for HTTP Ranger with known size (#43) 2018-05-14 14:34:08 -06:00
Kaloyan Raev
90ca28f7bc eestream avoid waiting for slow pieces (#30)
* WIP eestream avoid waiting for slow pieces

* Use non-blocking select to harvest all available inbufs

* Better way to check for elapsed time in tests

* Improve comment

* Determine EOF based on expected size

* Remove unused readerError type

* Configurable readers channel size

* Close properly decodedReader in tests

* Use context for properly closing the decodedReader

* Handle infectious errors using the new descendent classes

* Refactor decodedReader.Read() into helper functions

* Reenable TestRSErrors

* Test with Rangers

* Remove obsolete comment
2018-05-14 14:30:57 -06:00
Kaloyan Raev
a10ffc2ae6 Ignore VSCode settings and debug files (#41) 2018-05-10 12:23:26 -04:00
Dennis Coyle
4be0503f4a DHT interfaces (#28)
* first pass at interface for dht

* wip

* comments addressed from @jtolds

* adjust GetBuckets

* missed comment

* addressed PR comments from @jtolds
2018-05-09 08:04:51 -06:00
JT Olds
dd2fbb5069 don't require shallow clones of old commits (#38)
* don't require shallow clones of old commits

doesn't look like github allows it. hints here:
https://stackoverflow.com/questions/31278902/how-to-shallow-clone-a-specific-commit-with-depth-1

* don't try to update submodules
2018-05-09 16:21:53 +03:00
JT Olds
fc7cfde9f7 pkg/miniogw: a minio-based s3 gateway for storj. just stubs right now (#34) 2018-05-08 19:49:54 -04:00
JT Olio
78989a2d5a fix hanging travis test 2018-05-08 17:16:16 -06:00
JT Olds
2ef8e581b2 fix travis, and get a clean lint (#32)
* fix travis

* lint fixes

* travis: enable tests

* fix some tests, disable a broken eestream test for now
2018-05-08 18:02:01 -04:00
Natalie Villasana
c73a4f9d68
Merge pull request #20 from coyle/clean-lint
Clean lint!
2018-05-07 14:14:40 -04:00
Dennis Coyle
5e9b3f7d6a merge conflicts 2018-05-07 14:04:53 -04:00
JT Olds
5be1da982f pkg/telemetry: telemetry package using zeebo/admission (#26)
* pkg/telemetry: use zeebo/admission

* pkg/telemetry: updated to use latest zeebo/admission changes
2018-05-07 14:03:40 -04:00
Dennis Coyle
6cd6be2297 Merge remote-tracking branch 'upstream/master' 2018-05-07 13:53:14 -04:00
Dennis Coyle
02d86ae016 Merge branch 'master' of https://github.com/coyle/storj 2018-05-07 13:52:58 -04:00
Natalie Ventura Villasana
7907733629 adds panics to netstate test 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
afb06df26d changes the import statements to storj.io 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
4e8336f2ab adds zap logger, some flags, some nit fixes 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
d93dacb53e fixes some error handling, updates storj import links 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
8d8fd76748 adds suggested organization, zeebo, err handling 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
5dec8fafaf adds bolt crud interface w put, get, list, delete 2018-05-07 11:03:59 -06:00
Kaloyan Raev
1c56a4cf5c Decoder can tolerate readers with unexpected EOF and random data (#27)
* Decoder can tolerate readers with unexpected EOF

* Return EOF if required number of inbufs are at EOF

* Use existing randData() to generate random data for tests

* Test case for io.ErrUnexpectedEOF

* Add TransformReaderSize constructor
2018-05-04 14:52:49 -06:00
Dennis Coyle
a51087a343 merge conflicts 2018-05-02 15:10:40 -04:00
JT Olds
4f40280c71 some small import cleanups, go generate overlay protos (#25)
checking in go generated stuff is recommended by
https://blog.golang.org/generate
2018-05-01 13:24:32 -04:00
Dennis Coyle
1580f58cbb go.mod fixes 2018-04-30 15:28:02 -04:00
Dennis Coyle
52741af9ce make travis work? 2018-04-30 15:27:30 -04:00
Dennis Coyle
f31fede825
Merge pull request #23 from MeijeSibbel/patch-1
update readme
2018-04-30 10:53:41 -04:00
Dennis Coyle
5d8d0cdda0 renamed ranger.Closer -> ranger.RangeCloser 2018-04-30 10:49:04 -04:00
Kaloyan Raev
8c96f4e5a5 Avoid extra Range requests for content type detection 2018-04-30 06:08:41 -06:00
Kaloyan Raev
4635621a4d Move helper structs inside functions 2018-04-30 06:06:25 -06:00
Kaloyan Raev
acce2bfc08 Reduce network latency by initializing Rangers in parallel 2018-04-30 06:06:25 -06:00
Meije Sibbel
86a2371961
update readme
1. Fixed a few spelling issues.
2. Added capital letter to the Storj name instead of a lowercase one. 
3. Reformulated some sentences. 
3. Some other small fixes.
2018-04-27 14:53:46 +02:00
Dennis Coyle
8f84a9be91 echos for path 2018-04-26 09:04:15 -04:00
Dennis Coyle
6fd6de1013 new version 2018-04-25 15:11:30 -04:00
Dennis Coyle
99efd1b75d little g 2018-04-25 15:10:14 -04:00
Dennis Coyle
d25dbce576 wheres root 2018-04-25 15:08:26 -04:00
Dennis Coyle
60b917d83d go version name again 2018-04-25 14:57:30 -04:00