* add storeConfig struct and getSegmentStore helper for creating a segment store
* implement segment store in repairer, remove unnecessary repairer Repair method
* change repair method parameter from int to int32 to match type being passed in
* implement repairer service in captplanet
* rework Config, set Config defaults in captplanet/setup
* add filter field into OverlayOptions message
* chooseFiltered method, add excluded parameter in populate method
* change excluded type to []dht.NodeID in ChooseFiltered, change comment
* change name filter to excluded_nodes in proto
* implement helper function contains
* delete ChooseFiltered and add its functionality into Choose method to keep original author's history, add excluded argument into Choose calls
* regenerate mock_client.go
* regenerate protobuf
* adding the repair() func
* update test case to use new IDFromString function
* modified the repair() and updated streams mock
* modified the repair() and updated streams mock
* Options struct
* adding the repair() func
* modified the repair() and updated streams mock
* modified the repair() and updated streams mock
* integrating the segment repair()
* development repair with hack working
* repair segment changes
* integrated with mini hacks and rigged up test case with dev debug info
* integrated with ec and overlay
* added repair test case
* made the getNewUniqueNodes() to recursively go thru choose() to find get the required number of unique nodes
* cleaned up code
The old paths.Path type is now replaced with the new storj.Path.
storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().
As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:
storj.SplitPath
storj.JoinPaths
encryption.EncryptPath
encryption.DecryptPath
encryption.DerivePathKey
encryption.DeriveContentKey
All code in master is migrated to the new storj.Path type.
The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go
This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
* handle nil nodes in ec Put
* read and discard readers for nil nodes
* test 2 nil nodes, unique wont return false with nil nodes
* Discard reader data for nil nodes
* edit control flow
* add filter field into OverlayOptions message
* chooseFiltered method, add excluded parameter in populate method
* change excluded type to []dht.NodeID in ChooseFiltered, change comment
* change name filter to excluded_nodes in proto
* implement helper function contains
* delete ChooseFiltered and add its functionality into Choose method to keep original author's history, add excluded argument into Choose calls
* regenerate mock_client.go
* regenerate protobuf
* update test case to use new IDFromString function
* remove old kademlia test code
* begin adding path encryption
* do not encrypt/decrypt first element of path (bucket)
* add path encryption for delete and list
* use encrypted paths in streamstore.Meta
* fix listing with encrypted paths
* move encrypt/decryptAfterBucket to streamstore
* fix listing with no prefix
* remove duplicate logic for listing with no prefix
* Initial Layout
* Commit to test File Handling OS independed
* Hide struct properties to prevent manual interaction
* Fix Linting Errors
* 1st Working Windows Version
* Add missing Error Handling
* Fix Linting Errors
* Remove dependencies
* Further Improvements
* Remove commented code
* Improve comments and error messages
* No pointers to FPath
* Improve comment
* Do not filepath.ToSlash URL path
* Extract helper functions for parsing local path and Storj path
* Minor Improvements based on PR Comments
* Fix Linting Error and make Regex private
* Improve Layout
* Rework FPath and add tests
* Add more tests cases for windows
* Use for-loop instead of goto
* Use FPath in all uplink commands
* Add guard checks
* Add Test Cases and add comments
* merged the lasted master changes
* debug working of handling ctrl+c
* Handling of clean up of partially uploaded segments and pieces
* code cleanup per code comment
* updates based on code review comments
* Clean up last segment handling
* Fix increment for AES-GCM nonce
* Fix stream size calculation
* Adapt stream store tests
* Fix Delete method
* Rename info callback to segmentInfo
* Clearer calculation for offset in Nonce.AESGCMNonce()
* Adapt to the new little-endian nonce increment
* begin adding encryption for remote pieces
* begin adding decryption
* add encryption key as arg to Put and Get
* move encryption/decryption to object store
* Add encryption key to object store constructor
* Add the erasure scheme to object store constructor
* Ensure decrypter is initialized with the stripe size used by encrypter
* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"
This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.
* Revert "Add the erasure scheme to object store constructor"
This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.
* move encryption to stream store
* move decryption stuff to stream store
* revert changes in object store
* add encryptedBlockSize and close rangers on error during Get
* calculate padding sizes correctly
* encryptedBlockSize -> encryptionBlockSize
* pass encryption key and block size into stream store
* remove encryption key and block size from object store constructor
* move encrypter/decrypter initialization
* remove unnecessary cast
* Fix padding issue
* Fix linter
* add todos
* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata
* use different encryption key for each segment
* encrypt data in one step if it is small enough
* refactor and move encryption stuff
* fix errors related to nil slices passed to copy
* fix encrypter vs. decrypter bug
* put encryption stuff in eestream
* get captplanet test to pass
* fix linting errors
* add types for encryption keys/nonces and clean up
* fix tests
* more review changes
* add Cipher type for encryption stuff
* fix rs_test
* Simplify type casting of key and nonce
* Init starting nonce to the segment index
* don't copy derived key
* remove default encryption key; force user to explicitly set it
* move getSegmentPath to streams package
* dont require user to specify encryption key for captplanet
* rename GenericKey and GenericNonce to Key and Nonce
* review changes
* fix linting error
* Download uses the encryption type from metadata
* Store enc block size in metadata and use it for download
* storage node quick check and startup validation
* rearranged the startup validation and quick check logic
* travis lint warning fixes
* travis lint warning fixes
* travis lint warning fixes
* code changes per review comments
* code clean dev debug info
* travis lint wranings
* code changes per code review comments
* code changes per code review comments
* code update per review
* sqlite SUM is having issue when getting the SUM of an empty column; filepath was checking a directory that doesn't exist when starting server; Example updated to print allocated and used space
* storage node quick check and startup validation
* rearranged the startup validation and quick check logic
* travis lint warning fixes
* travis lint warning fixes
* travis lint warning fixes
* code changes per review comments
* code clean dev debug info
* travis lint wranings
* code changes per code review comments
* code changes per code review comments
* code update per review
* no file or directory error
* Updated mock PSClient
* add mb command
* forgot colon
* add command descriptions
* use utils.ParseURL in commands
* return error message instead of minio.BucketAlreadyExists in mb
* ls command with bucket store functionality
* rb command with bucket store functionality
* rm command with bucket store functionality
* newline
* use print rather than errs for messages, add no buckets messsage
* cp command with bucket store functionality
* remove deprecated getStorjObjects function
* defer utils.LogClose(f) on instead of defer f.Close()
* Check for no buckets after for loop
* add checks for unspecified bucket in bucket store methods
* fix incorrect return types
* add no path error messages in object store methods
* split copy into helpers
* srcObj scheme check in download
* print buckets instead of appending to slice
* check if destObj.Host != srcObj.Host
* better method of handling destination name if not specified
* uplink rename
* final cleanups
* trailing slash fixes
* linting
* more linting
* helpful error messages
* Adjust startAfter after merging #328
* Improve output messages
* Improved error check for empty bucket and path
* No page limit on client side. Rely on server side limit.
* Better time formatting
* Fix paths in recursive list results
