Commit Graph

19 Commits

Author SHA1 Message Date
Natalie Villasana
9ad4949825
adds segment store tests (#198)
* starts adding segmentStore tests

* adds mocked interfaces for segmentStore tests

* adds tests for put, get, delete, and list

* regenerates pointerdb mock and updates calls to accommodate new changes
2018-08-07 11:52:11 -04:00
Kaloyan Raev
aac7c6fbd1
ECClient pads data to fit RS block size (#199) 2018-08-06 17:24:30 +03:00
Natalie Villasana
d82486b85b
adds SegmentStore Put functionality (#138)
* lays out SegmentStore functions to implement

* Merge branch 'master' into segment-store

* adds overlay calls to put

* allows SegmentStore Put to upload a file to ecclient, then save pointer to pointerdb

* Merge branch 'master' into segment-store

* removes new overlay client instance in Put

* fixes syntax

* fixes syntax again

* fixes imports

* fixes typo

* removes pointerdb client from segmentStore struct for now

* changes SegmentStore to segmentStore

* changing types in parameters to fit other function calls

* takes RedundancyStrategy out of Put params

* changes NewClient param back to take an interface (not pointer to interface)

* fixes types

* moves pointer into PutRequest in SegmentStore Put

* passes interfact, not pointer to interface to NewSegmentStore

* fixes some types

* Get returns an instance of Meta

* fixes PutRequest fields

* adds remotePieces slice to pointerdb PutRequest

* ecClient Put now takes *proto.Nodes instead of proto.Nodes

* fixes syntax

* changes ec client dial interface to use *proto.Node

* changes other instances of proto.Node to *proto.Node in ecclient pkg

* adds *proto.Node to Get and Delete functions in ecclient pkg

* changes proto.Node to pointer in ec client_test

* changes proto.Node to pointer in ec client_test

* adds ecclient and pointerdb client to the segmentstore constructor

* adds ecclient and pointerDBClient to segmentStore constructor
2018-07-12 18:37:50 -04:00
Kaloyan Raev
d8f1ec1db6
ECClient (#110)
* WIP ECClient

* Get returns RangeCloser

* Introduce RedundancyStrategy

* Constructor takes max buffer memory

* Remove unnecessary NopCloser wrapper

* Added telemetry

* Tests

* Adapt to PSClient from master

* Decode should report error if empty rrs map is passed

* collectErrors helper

* Move to /pkg/storage

* Move to /pkg/storage/ec

* Rename ecclient.go to client.go

* Better logging

* Rename ec.ECClient to ec.Client

* Fix some test execution

* Adopt Transport Client from master
2018-07-03 11:35:01 +03:00
Kaloyan Raev
35d5761ee6
Use continue instead of return in table tests (#106)
I did a dumb mistake for some of the table tests, which made some of the
test cases not being executed.
2018-06-22 12:23:19 +03:00
Kaloyan Raev
045dae5f0d
Add context to Ranger.Range method (#99) 2018-06-19 18:59:09 +03:00
Kaloyan Raev
7c8d6f8081
Add error to the return values of Ranger.Range method (#90)
* Add error to the return values of Ranger.Range method

* Addressing code review feedback

* Set Content-Type header only if ctype is not empty
2018-06-18 19:46:49 +03:00
Kaloyan Raev
ed72a5e542
Optimize eestream upload for slow piece storage nodes (#48)
* Buffered eestream EncodeReader

* Extracted fillBuffer() helper function

* Slow channels will be closed if there are still at least k fast channels

* Doc comment for maxBufferMemory

* Use timer more efficiently

* Timer initialization should be inside the for-loop

* Parallel copy of encoded data to reader buffer channels

* Transfer input read errors to output encoded readers

* minimum and optimum thresholds

* Use time.AfterFunc

* Simplify error handling in constructor
2018-05-30 18:27:09 +03:00
Kaloyan Raev
06bd257092 Remove randomly failing eestream test case (#62) 2018-05-29 20:58:41 -06:00
aligeti
0376dc4bd2 AES GCM implementation and unit test code (#19)
* AES GCM implementation and unit test code

* modified and tested per the code review comments

* modified and tested per the code review comments

* updated to return err

* removed the debug printf commented code

* support of aes-gcm

* updated the renaming convention per GOLANG coding standards

* Initial Go & C biniding with libstorj

* Initial GO & C bindings with libstorj

* fixing the callback

* moved a millimeter :-) on c to go to gone....

*  added error condition, per review comment

* removed the .idea directory and also movie.avi file

* deleting files not to be in this pull request

* Revert "deleting files not to be in this pull request"

This reverts commit 026b834fe00f6b20a7566e71973fe224c12f533f.

* deleting files not to be in this pull request

* resolving conflicts

* syncing the file to master

* Use aes gcm in eestream rs tests

* Use aes gcm in serve example

* Fixed comment
2018-05-15 18:11:03 +03:00
Kaloyan Raev
90ca28f7bc eestream avoid waiting for slow pieces (#30)
* WIP eestream avoid waiting for slow pieces

* Use non-blocking select to harvest all available inbufs

* Better way to check for elapsed time in tests

* Improve comment

* Determine EOF based on expected size

* Remove unused readerError type

* Configurable readers channel size

* Close properly decodedReader in tests

* Use context for properly closing the decodedReader

* Handle infectious errors using the new descendent classes

* Refactor decodedReader.Read() into helper functions

* Reenable TestRSErrors

* Test with Rangers

* Remove obsolete comment
2018-05-14 14:30:57 -06:00
JT Olds
2ef8e581b2 fix travis, and get a clean lint (#32)
* fix travis

* lint fixes

* travis: enable tests

* fix some tests, disable a broken eestream test for now
2018-05-08 18:02:01 -04:00
Kaloyan Raev
1c56a4cf5c Decoder can tolerate readers with unexpected EOF and random data (#27)
* Decoder can tolerate readers with unexpected EOF

* Return EOF if required number of inbufs are at EOF

* Use existing randData() to generate random data for tests

* Test case for io.ErrUnexpectedEOF

* Add TransformReaderSize constructor
2018-05-04 14:52:49 -06:00
Kaloyan Raev
4635621a4d Move helper structs inside functions 2018-04-30 06:06:25 -06:00
Kaloyan Raev
acce2bfc08 Reduce network latency by initializing Rangers in parallel 2018-04-30 06:06:25 -06:00
Kaloyan Raev
69239e9e17 Do not fail decoding on first read error (#17)
* Do not fail decoding on first read error

Try decoding with the rest successfully read inputs.

* some small code improvements

* Allocate map memory upfront
2018-04-23 20:17:50 -06:00
Kaloyan Raev
f96cc9c955 HTTP Ranger (#11)
* HTTPRanger

* Migrate eestream library to use ReadCloser instead of Reader

* Use MultiReadCloser instead of MultiReader

* Adapt example cli cmds to latest serverbox changes

* Close LazyReadCloser only if only generated

* Close ReadClosers sequentially

* Close response body on unexpected status code

* Avoid double close of the pipe reader

* Better formatting for imports

* Exit instead of panicking
2018-04-17 07:39:14 -06:00
JT Olio
c90143a58d eestream: make secretbox example harder to fall victim to nonce reuse 2018-04-16 06:52:23 -06:00
JT Olio
80fa10a06e pkg/eestream: initial 2018-04-11 07:46:34 -06:00