* WIP possible discovery service impl
* Adds discovery service to CaptPlanet
* Updates the config and server for discovery service
* updates testplanet to use discovery package
* update satellite imports
* Removes unnecessary cache test
* linter fixes
* adds discovery startup to captplanet
* invoke refresh
* updates to discovery refresh cycle
* Make implementation more consistent with previous implementation
* add wait before trying to upload
* sleep a bit more
* remove kademlia bootstrap
* updates
* remove comments
* cmd/statreceiver: lua-scriptable stat receiver
Change-Id: I3ce0fe3f1ef4b1f4f27eed90bac0e91cfecf22d7
* some updates
Change-Id: I7c3485adcda1278fce01ae077b4761b3ddb9fb7a
* more comments
Change-Id: I0bb22993cd934c3d40fc1da80d07e49e686b80dd
* linter fixes
Change-Id: Ied014304ecb9aadcf00a6b66ad28f856a428d150
* catch errors
Change-Id: I6e1920f1fd941e66199b30bc427285c19769fc70
* review feedback
Change-Id: I9d4051851eab18970c5f5ddcf4ff265508e541d3
* errorgroup improvements
Change-Id: I4699dda3022f0485fbb50c9dafe692d3921734ff
* too tricky
the previous thing was better for memory with lots of errors at a time
but https://play.golang.org/p/RweTMRjoSCt is too much of a foot gun
Change-Id: I23f0b3d77dd4288fcc20b3756a7110359576bf44
* Merge bwagreement db into satellite master db
* adjust to recent tally changes
* linter problems
* linter problems
* returning db structs in more optimal way
* added pointer for assignment
* error message changed
* better param message
* adds channel for getting node out of lookup
* WIP adding the channels to lookups
* WIP adding channel to node lookups
* Wires up FindNodes method with channels
* WIP adds a test suite for lookup - tests are still failing
* WIP wires up use of testplanet for kademlia lookup tests
* WIP merging in node id changes
* Merges in pkg/storj node type changes
* Tests passing
* Lookup node working via Inspector now
* updates
* WIP working on getting tests passing
* WIP getting tests passing
* FindNode works
* Linter fix
* Adds copyrights to lookup_test
* removes a fmt.Printf I missed
* Removes commented out lines
* Pulls statdb stats into overlay cache whenever cache.Put() is called
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to filter based on node stats
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to exclude duplicate IP addresses
* preparing for use of `customtype` gogo extension with `NodeID` type
* review changes
* preparing for use of `customtype` gogo extension with `NodeID` type
* review changes
* wip
* tests passing
* wip fixing tests
* more wip test fixing
* remove NodeIDList from proto files
* linter fixes
* linter fixes
* linter/review fixes
* more freaking linter fixes
* omg just kill me - linterrrrrrrr
* travis linter, i will muder you and your family in your sleep
* goimports everything - burn in hell travis
* goimports update
* go mod tidy
* initial commit of inspector gadget wireup
* change name of comman dline tool, setup grpc server
* Get inspector cli working with grpc client
* Wired up CountNodes command
* WIP getting buckets response working
* Added GetBucket command
* WIP working on get buckets command
* WIP working on bucket list
* Still WIP
* WIP getting bucket counts to work
* Some clean up of unnecessary changes
* List Buckets and Get Bucket are working
* Removing logs, getting ready for review
* initial commit of inspector gadget wireup
* change name of comman dline tool, setup grpc server
* Get inspector cli working with grpc client
* Wired up CountNodes command
* WIP getting buckets response working
* Added GetBucket command
* WIP working on get buckets command
* WIP working on bucket list
* Still WIP
* WIP getting bucket counts to work
* Some clean up of unnecessary changes
* List Buckets and Get Bucket are working
* Removing logs, getting ready for review
* Fix error return
* Trying to get tests passing
* Adds method on dht mock for tests
* Add dbx files back
* Fix package import error in dbx file
* Adds copyrights to pass linter
* tidy go mod
* Updates from code review
* Updates inspector to take flag arguments for address
* Format list-buckets output more prettier
* Wiring up PING in kad inspector tools
* initial commit of inspector gadget wireup
* change name of comman dline tool, setup grpc server
* Get inspector cli working with grpc client
* Wired up CountNodes command
* WIP getting buckets response working
* Added GetBucket command
* WIP working on get buckets command
* WIP working on bucket list
* Still WIP
* WIP getting bucket counts to work
* Some clean up of unnecessary changes
* List Buckets and Get Bucket are working
* Removing logs, getting ready for review
* initial commit of inspector gadget wireup
* change name of comman dline tool, setup grpc server
* Get inspector cli working with grpc client
* Wired up CountNodes command
* WIP getting buckets response working
* Added GetBucket command
* WIP working on get buckets command
* WIP working on bucket list
* Still WIP
* WIP getting bucket counts to work
* Some clean up of unnecessary changes
* List Buckets and Get Bucket are working
* Removing logs, getting ready for review
* Fix error return
* Trying to get tests passing
* Adds method on dht mock for tests
* Add dbx files back
* Fix package import error in dbx file
* Adds copyrights to pass linter
* tidy go mod
* Updates from code review
* Updates inspector to take flag arguments for address
* Format list-buckets output more prettier
* Write access for uplink mount command
* fix linter problems
* Fix closing wrong variable and debug logs
* goimports
* fix problem with missing getattr for large files
* better map initialization
* fixes after comments
* add support for making directory
* add support for removing directories
* optimize OpenDir and Rmdir methods
* use storj.JoinPaths
* fix linter issues
* remove useless error check
* Signature verification
* Clean up agreement sender to have less errors
* overlay address in captnplanet
* Refactor bandwidth.proto to not use streams
* Make sure the send worked
* Handle connection to satellite
* Save renter public key inside of renter bandwidth allocations
* Default diag to sqlite. Make configurable
* Separate bw server and dbm; regenerate dbx files
* Make sure test uses protobufs
* Demonstrate creating bandwidth allocations
* WIP starting to wire up the kademlia CLI tool
* WIP wiring up kad cli tools
* WIP starting to wire up the kademlia CLI tool
* WIP wiring up kad cli tools
* Got everything wired up
* WIP starting to wire up the kademlia CLI tool
* WIP wiring up kad cli tools
* merge in upstream
* WIP wiring up kad cli tools
* Got everything wired up
* WIP trying to get CLI to connect
* Inspector connects to overlay now
* Some refactoring
* Linter fixes
* Linter fixes
* Switch to pkg/process instead of using rootCmd.Execute
* add storeConfig struct and getSegmentStore helper for creating a segment store
* implement segment store in repairer, remove unnecessary repairer Repair method
* change repair method parameter from int to int32 to match type being passed in
* implement repairer service in captplanet
* rework Config, set Config defaults in captplanet/setup
The old paths.Path type is now replaced with the new storj.Path.
storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().
As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:
storj.SplitPath
storj.JoinPaths
encryption.EncryptPath
encryption.DecryptPath
encryption.DerivePathKey
encryption.DeriveContentKey
All code in master is migrated to the new storj.Path type.
The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go
This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
* implements CombineErrors on runCfg error chan
* implement error channel timeout and CombineErrors
* implements CombineErrors with time.After timeout
* changes d to duration in args
* changes comment to better reflect what collectErrors does
* appeases the linter gods
* Gets rid of timeout bug in collectErrors
* Initial Layout
* Commit to test File Handling OS independed
* Hide struct properties to prevent manual interaction
* Fix Linting Errors
* 1st Working Windows Version
* Add missing Error Handling
* Fix Linting Errors
* Remove dependencies
* Further Improvements
* Remove commented code
* Improve comments and error messages
* No pointers to FPath
* Improve comment
* Do not filepath.ToSlash URL path
* Extract helper functions for parsing local path and Storj path
* Minor Improvements based on PR Comments
* Fix Linting Error and make Regex private
* Improve Layout
* Rework FPath and add tests
* Add more tests cases for windows
* Use for-loop instead of goto
* Use FPath in all uplink commands
* Add guard checks
* Add Test Cases and add comments
* Let's do it right this time
* Oh travis...
* Handle redis URL
* Travis... why u gotta be like this?
* Handle when address does not use redis scheme
* Start repairer
* Match provider.Responsibility interface
* Simplify if statement
* Config doesn't need to be a pointer
* Initialize doesn't need to be exported
* Don't run checker or repairer on startup
* Fix travis complaints
* Creates cron-job for checker, adds it to captplanet and satellite
* removes datarepair from satellite & captplanet run
* Delete config.go
* removes unused datarepair imports
* adds comments to fix linter
* Loads cache from context for PointerDB access
* WIP adds overlay lookups to pointerdb requests
* Pointer lookup code is added for Get
* adds feature flag for pointerdb return
* refactors pointerdb code
* removes some unnecessary debug logs
* Fixes indent in config
* adds early return for non-remote pointers
* formats code, removes some comments
* Fixes tests broken by pointer proto changes
* adds error check and merges variable declaration
* removes commented out proto import
* adds error check to pdbclient
* merged the lasted master changes
* debug working of handling ctrl+c
* Handling of clean up of partially uploaded segments and pieces
* code cleanup per code comment
* updates based on code review comments
* setup repairer loop
* added read from queue
* Refactor to make things easier to import
* add more control flow to repairer
* add comment
* basic interval structure for running check/repair
* change function name GetNext to Dequeue
* better increment/decrement syntax
* export Repairer struct
* delete 'unreachable code'
* add mon.Task() to Repairer.Repair
* remove 24 hour interval
* set maxRepair on Config as well as Repairer
* add comment for Repairer struct, check err
* comment out runCfg.Repair in cmd/satellite/main.go because it is NI yet
* begin adding encryption for remote pieces
* begin adding decryption
* add encryption key as arg to Put and Get
* move encryption/decryption to object store
* Add encryption key to object store constructor
* Add the erasure scheme to object store constructor
* Ensure decrypter is initialized with the stripe size used by encrypter
* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"
This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.
* Revert "Add the erasure scheme to object store constructor"
This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.
* move encryption to stream store
* move decryption stuff to stream store
* revert changes in object store
* add encryptedBlockSize and close rangers on error during Get
* calculate padding sizes correctly
* encryptedBlockSize -> encryptionBlockSize
* pass encryption key and block size into stream store
* remove encryption key and block size from object store constructor
* move encrypter/decrypter initialization
* remove unnecessary cast
* Fix padding issue
* Fix linter
* add todos
* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata
* use different encryption key for each segment
* encrypt data in one step if it is small enough
* refactor and move encryption stuff
* fix errors related to nil slices passed to copy
* fix encrypter vs. decrypter bug
* put encryption stuff in eestream
* get captplanet test to pass
* fix linting errors
* add types for encryption keys/nonces and clean up
* fix tests
* more review changes
* add Cipher type for encryption stuff
* fix rs_test
* Simplify type casting of key and nonce
* Init starting nonce to the segment index
* don't copy derived key
* remove default encryption key; force user to explicitly set it
* move getSegmentPath to streams package
* dont require user to specify encryption key for captplanet
* rename GenericKey and GenericNonce to Key and Nonce
* review changes
* fix linting error
* Download uses the encryption type from metadata
* Store enc block size in metadata and use it for download
* add mb command
* forgot colon
* add command descriptions
* use utils.ParseURL in commands
* return error message instead of minio.BucketAlreadyExists in mb
* ls command with bucket store functionality
* rb command with bucket store functionality
* rm command with bucket store functionality
* newline
* use print rather than errs for messages, add no buckets messsage
* cp command with bucket store functionality
* remove deprecated getStorjObjects function
* defer utils.LogClose(f) on instead of defer f.Close()
* Check for no buckets after for loop
* add checks for unspecified bucket in bucket store methods
* fix incorrect return types
* add no path error messages in object store methods
* split copy into helpers
* srcObj scheme check in download
* print buckets instead of appending to slice
* check if destObj.Host != srcObj.Host
* better method of handling destination name if not specified
* uplink rename
* final cleanups
* trailing slash fixes
* linting
* more linting
* helpful error messages
* Adjust startAfter after merging #328
* Improve output messages
* Improved error check for empty bucket and path
* No page limit on client side. Rely on server side limit.
* Better time formatting
* Fix paths in recursive list results
* Normalize the dockerfiles and entrypoints
* fix formattting and docker-compose file
* add missing env var to satellite dockerfile
* Remove build args no longer in use
* More changes to make it all work
* Finally fix the binary builds for a couple os/arch combinations
* Make the file executable before adding it to the zip
* This cache hack wasn't suppose to make it into the PR
* Guard against calling the binary target by itself
* Order matters on OS X
* Vendored my image
* No 386 builds until storage is fixed.
* First pass at building binaries
* Finished building images
* Reworked some bits to save binaries in a better place.
* First pass at uploading binaries
* Builds for freebsd fail currently
* Add ignore file for docker builds to not invalidate the cache as often
* Docker image for 1.11-alpine exists now
* Name windows binaries correctly
* The makefile is used in tests.
* Take a swing at updating the tests for 1.11
* Switch to something with glibc so the race detector works.
* Remove unused .PHONY targets
* add CopyObject method
* use utils.LogClose
* extract common code from GetObject to getObject helper
* remove rr.Range from getObject helper, create helper putObject
* return rr, err in getObject helper
* extract code from PutObject into putObject helper
* remove commented out text
* remove other commented out code
* WIP trying to get storj cp command to work with copyObject
* fix typo in rb and now it works
* use rr.Size() instead of srcInfo.Size
* Revert "WIP trying to get storj cp command to work with copyObject"
This reverts commit e256b9f9a0fda728d41eb5b9d7a98b5446825842.
* add CopyObject test
* rebase and fix merge conflicts
* check error in gateway-storj test
* fix typo
* wip ca/ident cmds
* minor improvements and commenting
* combine id and ca commands and add $CONFDIR
* add `NewIdenity` test
* refactor `NewCA` benchmarks
* linter fixes
* First pass at an all-in-one compose project
* Sort out dependencies so uplink can build in docker
* Added the final bits to make this work
* Add readme for all-in-one
* Fix sed to work on OS X and Linux
* Make the readme for all-in-one better
* Only bring up the satellite and uplink after fixing the mock overlay
* Use a dummy value for mock overlay for now
* Finishing touches
* Fix url form for aws cli tool
* Move files out of all-in-one directory to help with directory confusion.
* Added a make target to make all-in-one even better
* move mock overlay from client to server
this doesn't really change much, but it does allow you to
run a standalone gateway against captain planet. it still does
not allow you to run a standalone gateway against a standalone
heavy client
* pointerdb: small error fixes
* some cleanups
* fix tests
* captplanet standalone farmer setup
* Bandwidth Allocation
* utils.Close method changed to utils.LogClose
* Get build temporarily working
* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function
* Update example client to reflect changes in client API
* Update ecclient to use latest PSClient, Make NewPSClient return error also
* Updated pieceranger tests to check for errors; sign method should take byte array
* Handle defers in store.go better
* Fix defer functions in psdb.go
* fun times
* Protobuf bandwidthallocation data is now a byte array
* Remove psservice package and merge it into pstore server
* Write wrapper for database calls
* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment
* Make PSDB tests implementation independent rather than method independent
* get rid of payer, renter in ecclient
* add context monitoring in store and retrieve
* WIP first commit
* change args check
* rename to cli, rename folder to storj, const defaultConfDir, use process.Exec()
* typos
* WIP create helper for getting minio objectlayer, changed setup command to try to match changes in setting up identity
* add rb command
* changes for setting up identity
* remove concurrency field from setupCfg
* rename variable returned by url.Parse to u across all commands for consistency
* add marker functionality to ls command
* add download functionality to cp command
* fix typo in rb init, use short variable names
