Commit Graph

363 Commits

Author SHA1 Message Date
Egon Elbre
764234109f
remove urfave/cli from dependencies (#415) 2018-10-03 12:15:17 +03:00
Egon Elbre
96e55d3cf3
Fix go.mod (#395) 2018-09-28 18:44:02 +03:00
Maximillian von Briesen
821d0b6f1d
Stream encryption (#302)
* begin adding encryption for remote pieces

* begin adding decryption

* add encryption key as arg to Put and Get

* move encryption/decryption to object store

* Add encryption key to object store constructor

* Add the erasure scheme to object store constructor

* Ensure decrypter is initialized with the stripe size used by encrypter

* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"

This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.

* Revert "Add the erasure scheme to object store constructor"

This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.

* move encryption to stream store

* move decryption stuff to stream store

* revert changes in object store

* add encryptedBlockSize and close rangers on error during Get

* calculate padding sizes correctly

* encryptedBlockSize -> encryptionBlockSize

* pass encryption key and block size into stream store

* remove encryption key and block size from object store constructor

* move encrypter/decrypter initialization

* remove unnecessary cast

* Fix padding issue

* Fix linter

* add todos

* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata

* use different encryption key for each segment

* encrypt data in one step if it is small enough

* refactor and move encryption stuff

* fix errors related to nil slices passed to copy

* fix encrypter vs. decrypter bug

* put encryption stuff in eestream

* get captplanet test to pass

* fix linting errors

* add types for encryption keys/nonces and clean up

* fix tests

* more review changes

* add Cipher type for encryption stuff

* fix rs_test

* Simplify type casting of key and nonce

* Init starting nonce to the segment index

* don't copy derived key

* remove default encryption key; force user to explicitly set it

* move getSegmentPath to streams package

* dont require user to specify encryption key for captplanet

* rename GenericKey and GenericNonce to Key and Nonce

* review changes

* fix linting error

* Download uses the encryption type from metadata

* Store enc block size in metadata and use it for download
2018-09-26 09:32:23 -04:00
aligeti
97cbacb299
storage node quick check and startup validation (#368)
* storage node quick check and startup validation

*  rearranged the startup validation and quick check logic

* travis lint warning fixes

* travis lint warning fixes

* travis lint warning fixes

* code changes per review comments

* code clean dev debug info

* travis lint wranings

* code changes per code review comments

* code changes per code review comments

* code update per review

* sqlite SUM is having issue when getting the SUM of an empty column; filepath was checking a directory that doesn't exist when starting server; Example updated to print allocated and used space

* storage node quick check and startup validation

*  rearranged the startup validation and quick check logic

* travis lint warning fixes

* travis lint warning fixes

* travis lint warning fixes

* code changes per review comments

* code clean dev debug info

* travis lint wranings

* code changes per code review comments

* code changes per code review comments

* code update per review

* no file or directory error

* Updated mock PSClient
2018-09-25 12:49:55 -04:00
Egon Elbre
10d9420f24
command-line benchmark and visualization (#342) 2018-09-11 18:35:46 +03:00
Egon Elbre
7a906285e3 Small cleanup at top-level (#336)
* cleanup some top-level packages

* remove unused index.html

* update indirect tags
2018-09-10 07:23:08 -07:00
Alexander Leitner
c68cd1232d Prevent sql injection for farmer (#322)
* Validate incoming ID

* Pass tests, cleanup code

* Smarter regex

* Update server.go
2018-09-08 11:06:44 -06:00
JT Olio
8da996d61a infectious: pull in optimizations (#320) 2018-09-07 10:32:54 +03:00
Egon Elbre
83df0ee1b0
Implement ListV2 with storage rework (#303)
1. Added KeyValueStore.Iterate for implementing the different List, ListV2 etc. implementations. This allows for more efficient use of memory depending on the situation.
2. Implemented an inmemory teststore for running tests. This should allow to replace MockKeyValueStore in most places.
3. Rewrote tests
4. Pulled out logger from bolt implementation so it can be used for all other storage implementations.
5. Fixed multiple things in bolt and redis implementations.
2018-09-05 19:10:35 +03:00
Cameron
ea9f5b4e63
CopyObject (#242)
* add CopyObject method

* use utils.LogClose

* extract common code from GetObject to getObject helper

* remove rr.Range from getObject helper, create helper putObject

* return rr, err in getObject helper

* extract code from PutObject into putObject helper

* remove commented out text

* remove other commented out code

* WIP trying to get storj cp command to work with copyObject

* fix typo in rb and now it works

* use rr.Size() instead of srcInfo.Size

* Revert "WIP trying to get storj cp command to work with copyObject"

This reverts commit e256b9f9a0fda728d41eb5b9d7a98b5446825842.

* add CopyObject test

* rebase and fix merge conflicts

* check error in gateway-storj test

* fix typo
2018-08-29 11:00:05 -04:00
JT Olio
58c810519f
cross-compilation: fix darwin (#293)
* cross-compilation: fix darwin

* minio without fsnotify

* update Gopkg.lock

* make release target
2018-08-27 15:06:55 -06:00
JT Olio
3154086e58 go1.11: update monotime (#287) 2018-08-24 20:23:09 -04:00
Egon Elbre
f80185a807
Preliminary support for windows (#266)
* Don't use url.Parse for bolt paths: filepaths may not be valid URL-s.
* go.mod: update dependencies
* README.md: add Windows instructions
* pkg/overlay: check for the correct path and text in error
* pkg/overlay: fix tests for windows
* pkg/piecestore: make windows tests pass
* pkg/telemetry: skip test, as it doesn't shutdown nicely
* storage/redis: ensure that redis is clean before running tests
2018-08-23 10:05:56 +03:00