Commit Graph

424 Commits

Author SHA1 Message Date
Egon Elbre
ec0850fa9f
examples: piecestore-cli use cobra directly (#409) 2018-10-02 17:48:09 +03:00
Egon Elbre
dac03acbc4
examples: piecestore-client, use cobra directly (#410) 2018-10-02 17:47:39 +03:00
James Hagans
19ec28f700
Streams Store mock test (#373)
* add basic test for Stream Store
2018-10-01 21:48:11 -04:00
Jennifer Li Johnson
54996e1edb
pkg structure and repair queue enqueue/dequeue (#397)
* pkg structure and repair queue implementation

* adds zeebo

* gets redis working with queue

* modifies interface

* changes re feedback

* pr changes w encoding and enqueue dequeue modifications

* test force error

* concurrent enqueue/dequeue

* refactor sequential to use only 1 slice

* added token for time conflicts
2018-10-01 19:25:41 -04:00
Egon Elbre
ab64aa0b97
add missing copyright headers (#403) 2018-10-01 14:29:27 +03:00
Egon Elbre
0076b17c58
cmd/s3-benchmark: add exit error information (#402) 2018-10-01 14:28:45 +03:00
Michal Niewrzal
39b3a09456
Fix uplink cp progress bar (#394)
* Fix uplink cp progress bar

* disable progress for logging above warn

* remove log level check
2018-09-30 22:25:58 +02:00
Egon Elbre
67784e0667
Lint examples folder (#391) 2018-09-28 21:08:44 +03:00
Egon Elbre
96e55d3cf3
Fix go.mod (#395) 2018-09-28 18:44:02 +03:00
Egon Elbre
e83502f33d
s3-benchmark: add aws-cli support (#386) 2018-09-28 15:40:08 +03:00
Egon Elbre
c9cabaf73a
storage/filestore (#384) 2018-09-28 09:59:27 +03:00
Dylan Lott
0920f293b8
patches missing node issue in mock overlay (#387) 2018-09-27 15:41:56 -05:00
Kaloyan Raev
f94c584768
Rename: DecodedBlockSize>StripeSize, EncodedBlockSize>ErasureShareSize (#392) 2018-09-27 14:52:18 +03:00
Kaloyan Raev
bc0f697929
Pointer stores only nodes with successfully uploaded pieces (#390) 2018-09-27 13:45:19 +03:00
Egon Elbre
2019803c5e
Use RepairThreshold naming consistently (#385) 2018-09-26 17:23:33 +03:00
Egon Elbre
550858c5a7
pkg/telemetry: use smaller mock in test (#370) 2018-09-26 17:00:54 +03:00
Maximillian von Briesen
821d0b6f1d
Stream encryption (#302)
* begin adding encryption for remote pieces

* begin adding decryption

* add encryption key as arg to Put and Get

* move encryption/decryption to object store

* Add encryption key to object store constructor

* Add the erasure scheme to object store constructor

* Ensure decrypter is initialized with the stripe size used by encrypter

* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"

This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.

* Revert "Add the erasure scheme to object store constructor"

This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.

* move encryption to stream store

* move decryption stuff to stream store

* revert changes in object store

* add encryptedBlockSize and close rangers on error during Get

* calculate padding sizes correctly

* encryptedBlockSize -> encryptionBlockSize

* pass encryption key and block size into stream store

* remove encryption key and block size from object store constructor

* move encrypter/decrypter initialization

* remove unnecessary cast

* Fix padding issue

* Fix linter

* add todos

* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata

* use different encryption key for each segment

* encrypt data in one step if it is small enough

* refactor and move encryption stuff

* fix errors related to nil slices passed to copy

* fix encrypter vs. decrypter bug

* put encryption stuff in eestream

* get captplanet test to pass

* fix linting errors

* add types for encryption keys/nonces and clean up

* fix tests

* more review changes

* add Cipher type for encryption stuff

* fix rs_test

* Simplify type casting of key and nonce

* Init starting nonce to the segment index

* don't copy derived key

* remove default encryption key; force user to explicitly set it

* move getSegmentPath to streams package

* dont require user to specify encryption key for captplanet

* rename GenericKey and GenericNonce to Key and Nonce

* review changes

* fix linting error

* Download uses the encryption type from metadata

* Store enc block size in metadata and use it for download
2018-09-26 09:32:23 -04:00
Egon Elbre
c65b114941
Add captplanet tests for IPv6 (#371) 2018-09-25 21:23:21 +03:00
aligeti
97cbacb299
storage node quick check and startup validation (#368)
* storage node quick check and startup validation

*  rearranged the startup validation and quick check logic

* travis lint warning fixes

* travis lint warning fixes

* travis lint warning fixes

* code changes per review comments

* code clean dev debug info

* travis lint wranings

* code changes per code review comments

* code changes per code review comments

* code update per review

* sqlite SUM is having issue when getting the SUM of an empty column; filepath was checking a directory that doesn't exist when starting server; Example updated to print allocated and used space

* storage node quick check and startup validation

*  rearranged the startup validation and quick check logic

* travis lint warning fixes

* travis lint warning fixes

* travis lint warning fixes

* code changes per review comments

* code clean dev debug info

* travis lint wranings

* code changes per code review comments

* code changes per code review comments

* code update per review

* no file or directory error

* Updated mock PSClient
2018-09-25 12:49:55 -04:00
Brandon Iglesias
f1a82e588c updating the reed solomon numbers (#350)
* Update config.go

* updating captplanet

* Fix captplanet setup command for >= 100 nodes
2018-09-25 18:38:37 +03:00
Matt Robinson
5c226238f3
Update kubernetes info for staging cluster (#382) 2018-09-25 10:55:31 -04:00
Kaloyan Raev
bbcac47d2f Fix data race in eestream (#377) 2018-09-25 14:39:14 +03:00
Kaloyan Raev
4a176915ef MinioGW should return ObjectNotFound if non-existing file is deleted (#379) 2018-09-24 22:14:21 +03:00
Egon Elbre
8d73b20a63
Basic Multipart support (#351) 2018-09-21 22:44:45 +03:00
Egon Elbre
4042ebdfea
ensure that protobuffer package and go_package are separate (#369) 2018-09-21 21:48:54 +03:00
Jennifer Li Johnson
85b5ff338f
Creates data repair queue interface and protobuf RepairSegment messgage (#367) 2018-09-20 15:08:43 -04:00
Matt Robinson
d2dcd8a136 Revert break for local files (#366) 2018-09-20 14:44:28 -04:00
Alexander Leitner
5a4f5e55ff
Add progress bar (#358)
* Add progress bar

* Turn progress bar on and off

* fix dependencies
2018-09-20 14:41:42 -04:00
Jennifer Li Johnson
74a39432de
routing table driven tests (#357)
* update testfindnear

* test connection success

* test split bucket

* TestDetermineDifferingBitIndex

* TestDetermineLeafDepth

* TestGetKBucketRange

* TestGetNodeIDsWithinKBucket

* TestNodeIsWithinNearestK

* add node

* updates add node test with actual kbucket and node keys

* cleanup
2018-09-19 14:37:28 -04:00
Matt Robinson
e1f44ec782 Make the next shirt, better (#362) 2018-09-18 09:26:59 -06:00
Egon Elbre
ebb839da7d
Add ListV2 method (#365) 2018-09-18 18:26:53 +03:00
Egon Elbre
50cd8211c2
Use specific aws-cli version (#364) 2018-09-18 17:30:09 +03:00
Matt Robinson
89fafd01d3
Handle release builds and email status (#361)
* Handle release builds and email status to a list.

* Tested jenkinsfile
2018-09-18 09:46:04 -04:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Egon Elbre
f6c48dcb60
Fix windows linter errors (#355) 2018-09-14 20:56:18 +03:00
Matt Robinson
0141c862f3 Stop travis from making binaries for now (#356)
* Disable this for now

* Minimize number of changes
2018-09-14 13:18:19 -04:00
Kaloyan Raev
ee5f32b0f7
ConcatRanger closes segment readers as soon as data is read (#349)
* ConcatRanger closes a segment readers as soon as data is read

* Fix linter errors

* Fix data race

* Nicer call to CombineErrors
2018-09-14 10:10:43 -04:00
Stefan Benten
05ce4e5269
Fix path handling (#354) 2018-09-14 15:14:04 +02:00
aligeti
3571886fe7 check uplink setup (#339)
* check uplink setup

* code review comments changes
2018-09-13 13:34:01 -04:00
Matt Robinson
0567f9cb57 Clean up Travis and Jenkins (#353)
* Make travis do what Jenkins will do. Make Jenkins not do what Travis does.

* Enable docker in travis
2018-09-13 12:55:14 -04:00
Alexander Leitner
3399376eb2
Limit to only 1 ttl database write when storing data on storage nodes (#348)
* Limit to only 1 database write

* Check file system rather than database

* Move check to storefile. We need to figure out how to fix this mess

* piecestore should not overwrite data, it should fail when trying to write to a file that already exists

* Format errors, delete unused function in psdb for checking if TTL exists

* Combine errors better
2018-09-13 10:30:45 -04:00
Egon Elbre
a7d34bea4b
Add missing copyright headers (#352) 2018-09-13 16:30:28 +03:00
Alexander Leitner
4397b90349
Only create index if not exists (#347) 2018-09-12 12:23:21 -04:00
Matt Robinson
0b947355da A bunch of small fixes to make OPsy things better (#346) 2018-09-12 11:02:53 -04:00
Kaloyan Raev
8f4b29abd0
Lazy segment ranger (#343) 2018-09-12 09:06:44 -04:00
Michal Niewrzal
e90c2140b8 Show file size for uplink ls command (#344)
* Show file size for uplink ls command

* align size to right

* Fix stream store list method to handle meta.Size flag
2018-09-12 09:03:31 -04:00
Egon Elbre
10d9420f24
command-line benchmark and visualization (#342) 2018-09-11 18:35:46 +03:00
Egon Elbre
4486e265bb
Cleanup after tests (#341) 2018-09-11 16:57:12 +03:00
Egon Elbre
e7e2d4d7c9
use golangci-lint and add missing error checks (#340) 2018-09-11 16:13:25 +03:00
Alexander Leitner
8f07c4d746
Keep track of used space (#338)
* Keep track of space at all times

* Create index on expires

* Updated mock psclient
2018-09-11 08:40:45 -04:00