* WIP starting to wire up the kademlia CLI tool
* WIP wiring up kad cli tools
* WIP starting to wire up the kademlia CLI tool
* WIP wiring up kad cli tools
* Got everything wired up
* WIP starting to wire up the kademlia CLI tool
* WIP wiring up kad cli tools
* merge in upstream
* WIP wiring up kad cli tools
* Got everything wired up
* WIP trying to get CLI to connect
* Inspector connects to overlay now
* Some refactoring
* Linter fixes
* Linter fixes
* Switch to pkg/process instead of using rootCmd.Execute
* add storeConfig struct and getSegmentStore helper for creating a segment store
* implement segment store in repairer, remove unnecessary repairer Repair method
* change repair method parameter from int to int32 to match type being passed in
* implement repairer service in captplanet
* rework Config, set Config defaults in captplanet/setup
The old paths.Path type is now replaced with the new storj.Path.
storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().
As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:
storj.SplitPath
storj.JoinPaths
encryption.EncryptPath
encryption.DecryptPath
encryption.DerivePathKey
encryption.DeriveContentKey
All code in master is migrated to the new storj.Path type.
The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go
This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
* implements CombineErrors on runCfg error chan
* implement error channel timeout and CombineErrors
* implements CombineErrors with time.After timeout
* changes d to duration in args
* changes comment to better reflect what collectErrors does
* appeases the linter gods
* Gets rid of timeout bug in collectErrors
* Initial Layout
* Commit to test File Handling OS independed
* Hide struct properties to prevent manual interaction
* Fix Linting Errors
* 1st Working Windows Version
* Add missing Error Handling
* Fix Linting Errors
* Remove dependencies
* Further Improvements
* Remove commented code
* Improve comments and error messages
* No pointers to FPath
* Improve comment
* Do not filepath.ToSlash URL path
* Extract helper functions for parsing local path and Storj path
* Minor Improvements based on PR Comments
* Fix Linting Error and make Regex private
* Improve Layout
* Rework FPath and add tests
* Add more tests cases for windows
* Use for-loop instead of goto
* Use FPath in all uplink commands
* Add guard checks
* Add Test Cases and add comments
* Let's do it right this time
* Oh travis...
* Handle redis URL
* Travis... why u gotta be like this?
* Handle when address does not use redis scheme
* Start repairer
* Match provider.Responsibility interface
* Simplify if statement
* Config doesn't need to be a pointer
* Initialize doesn't need to be exported
* Don't run checker or repairer on startup
* Fix travis complaints
* Creates cron-job for checker, adds it to captplanet and satellite
* removes datarepair from satellite & captplanet run
* Delete config.go
* removes unused datarepair imports
* adds comments to fix linter
* Loads cache from context for PointerDB access
* WIP adds overlay lookups to pointerdb requests
* Pointer lookup code is added for Get
* adds feature flag for pointerdb return
* refactors pointerdb code
* removes some unnecessary debug logs
* Fixes indent in config
* adds early return for non-remote pointers
* formats code, removes some comments
* Fixes tests broken by pointer proto changes
* adds error check and merges variable declaration
* removes commented out proto import
* adds error check to pdbclient
* merged the lasted master changes
* debug working of handling ctrl+c
* Handling of clean up of partially uploaded segments and pieces
* code cleanup per code comment
* updates based on code review comments
* setup repairer loop
* added read from queue
* Refactor to make things easier to import
* add more control flow to repairer
* add comment
* basic interval structure for running check/repair
* change function name GetNext to Dequeue
* better increment/decrement syntax
* export Repairer struct
* delete 'unreachable code'
* add mon.Task() to Repairer.Repair
* remove 24 hour interval
* set maxRepair on Config as well as Repairer
* add comment for Repairer struct, check err
* comment out runCfg.Repair in cmd/satellite/main.go because it is NI yet
* begin adding encryption for remote pieces
* begin adding decryption
* add encryption key as arg to Put and Get
* move encryption/decryption to object store
* Add encryption key to object store constructor
* Add the erasure scheme to object store constructor
* Ensure decrypter is initialized with the stripe size used by encrypter
* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"
This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.
* Revert "Add the erasure scheme to object store constructor"
This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.
* move encryption to stream store
* move decryption stuff to stream store
* revert changes in object store
* add encryptedBlockSize and close rangers on error during Get
* calculate padding sizes correctly
* encryptedBlockSize -> encryptionBlockSize
* pass encryption key and block size into stream store
* remove encryption key and block size from object store constructor
* move encrypter/decrypter initialization
* remove unnecessary cast
* Fix padding issue
* Fix linter
* add todos
* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata
* use different encryption key for each segment
* encrypt data in one step if it is small enough
* refactor and move encryption stuff
* fix errors related to nil slices passed to copy
* fix encrypter vs. decrypter bug
* put encryption stuff in eestream
* get captplanet test to pass
* fix linting errors
* add types for encryption keys/nonces and clean up
* fix tests
* more review changes
* add Cipher type for encryption stuff
* fix rs_test
* Simplify type casting of key and nonce
* Init starting nonce to the segment index
* don't copy derived key
* remove default encryption key; force user to explicitly set it
* move getSegmentPath to streams package
* dont require user to specify encryption key for captplanet
* rename GenericKey and GenericNonce to Key and Nonce
* review changes
* fix linting error
* Download uses the encryption type from metadata
* Store enc block size in metadata and use it for download
* add mb command
* forgot colon
* add command descriptions
* use utils.ParseURL in commands
* return error message instead of minio.BucketAlreadyExists in mb
* ls command with bucket store functionality
* rb command with bucket store functionality
* rm command with bucket store functionality
* newline
* use print rather than errs for messages, add no buckets messsage
* cp command with bucket store functionality
* remove deprecated getStorjObjects function
* defer utils.LogClose(f) on instead of defer f.Close()
* Check for no buckets after for loop
* add checks for unspecified bucket in bucket store methods
* fix incorrect return types
* add no path error messages in object store methods
* split copy into helpers
* srcObj scheme check in download
* print buckets instead of appending to slice
* check if destObj.Host != srcObj.Host
* better method of handling destination name if not specified
* uplink rename
* final cleanups
* trailing slash fixes
* linting
* more linting
* helpful error messages
* Adjust startAfter after merging #328
* Improve output messages
* Improved error check for empty bucket and path
* No page limit on client side. Rely on server side limit.
* Better time formatting
* Fix paths in recursive list results
* Normalize the dockerfiles and entrypoints
* fix formattting and docker-compose file
* add missing env var to satellite dockerfile
* Remove build args no longer in use
* More changes to make it all work
* Finally fix the binary builds for a couple os/arch combinations
* Make the file executable before adding it to the zip
* This cache hack wasn't suppose to make it into the PR
* Guard against calling the binary target by itself
* Order matters on OS X
* Vendored my image
* No 386 builds until storage is fixed.
* First pass at building binaries
* Finished building images
* Reworked some bits to save binaries in a better place.
* First pass at uploading binaries
* Builds for freebsd fail currently
* Add ignore file for docker builds to not invalidate the cache as often
* Docker image for 1.11-alpine exists now
* Name windows binaries correctly
* The makefile is used in tests.
* Take a swing at updating the tests for 1.11
* Switch to something with glibc so the race detector works.
* Remove unused .PHONY targets
* add CopyObject method
* use utils.LogClose
* extract common code from GetObject to getObject helper
* remove rr.Range from getObject helper, create helper putObject
* return rr, err in getObject helper
* extract code from PutObject into putObject helper
* remove commented out text
* remove other commented out code
* WIP trying to get storj cp command to work with copyObject
* fix typo in rb and now it works
* use rr.Size() instead of srcInfo.Size
* Revert "WIP trying to get storj cp command to work with copyObject"
This reverts commit e256b9f9a0fda728d41eb5b9d7a98b5446825842.
* add CopyObject test
* rebase and fix merge conflicts
* check error in gateway-storj test
* fix typo
* wip ca/ident cmds
* minor improvements and commenting
* combine id and ca commands and add $CONFDIR
* add `NewIdenity` test
* refactor `NewCA` benchmarks
* linter fixes
* First pass at an all-in-one compose project
* Sort out dependencies so uplink can build in docker
* Added the final bits to make this work
* Add readme for all-in-one
* Fix sed to work on OS X and Linux
* Make the readme for all-in-one better
* Only bring up the satellite and uplink after fixing the mock overlay
* Use a dummy value for mock overlay for now
* Finishing touches
* Fix url form for aws cli tool
* Move files out of all-in-one directory to help with directory confusion.
* Added a make target to make all-in-one even better
* move mock overlay from client to server
this doesn't really change much, but it does allow you to
run a standalone gateway against captain planet. it still does
not allow you to run a standalone gateway against a standalone
heavy client
* pointerdb: small error fixes
* some cleanups
* fix tests
* captplanet standalone farmer setup
* Bandwidth Allocation
* utils.Close method changed to utils.LogClose
* Get build temporarily working
* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function
* Update example client to reflect changes in client API
* Update ecclient to use latest PSClient, Make NewPSClient return error also
* Updated pieceranger tests to check for errors; sign method should take byte array
* Handle defers in store.go better
* Fix defer functions in psdb.go
* fun times
* Protobuf bandwidthallocation data is now a byte array
* Remove psservice package and merge it into pstore server
* Write wrapper for database calls
* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment
* Make PSDB tests implementation independent rather than method independent
* get rid of payer, renter in ecclient
* add context monitoring in store and retrieve
* WIP first commit
* change args check
* rename to cli, rename folder to storj, const defaultConfDir, use process.Exec()
* typos
* WIP create helper for getting minio objectlayer, changed setup command to try to match changes in setting up identity
* add rb command
* changes for setting up identity
* remove concurrency field from setupCfg
* rename variable returned by url.Parse to u across all commands for consistency
* add marker functionality to ls command
* add download functionality to cp command
* fix typo in rb init, use short variable names
