* Let's do it right this time
* Oh travis...
* Handle redis URL
* Travis... why u gotta be like this?
* Handle when address does not use redis scheme
* Start repairer
* Match provider.Responsibility interface
* Simplify if statement
* Config doesn't need to be a pointer
* Initialize doesn't need to be exported
* Don't run checker or repairer on startup
* Fix travis complaints
* Creates cron-job for checker, adds it to captplanet and satellite
* removes datarepair from satellite & captplanet run
* Delete config.go
* removes unused datarepair imports
* adds comments to fix linter
* Loads cache from context for PointerDB access
* WIP adds overlay lookups to pointerdb requests
* Pointer lookup code is added for Get
* adds feature flag for pointerdb return
* refactors pointerdb code
* removes some unnecessary debug logs
* Fixes indent in config
* adds early return for non-remote pointers
* formats code, removes some comments
* Fixes tests broken by pointer proto changes
* adds error check and merges variable declaration
* removes commented out proto import
* adds error check to pdbclient
* merged the lasted master changes
* debug working of handling ctrl+c
* Handling of clean up of partially uploaded segments and pieces
* code cleanup per code comment
* updates based on code review comments
* setup repairer loop
* added read from queue
* Refactor to make things easier to import
* add more control flow to repairer
* add comment
* basic interval structure for running check/repair
* change function name GetNext to Dequeue
* better increment/decrement syntax
* export Repairer struct
* delete 'unreachable code'
* add mon.Task() to Repairer.Repair
* remove 24 hour interval
* set maxRepair on Config as well as Repairer
* add comment for Repairer struct, check err
* comment out runCfg.Repair in cmd/satellite/main.go because it is NI yet
* begin adding encryption for remote pieces
* begin adding decryption
* add encryption key as arg to Put and Get
* move encryption/decryption to object store
* Add encryption key to object store constructor
* Add the erasure scheme to object store constructor
* Ensure decrypter is initialized with the stripe size used by encrypter
* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"
This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.
* Revert "Add the erasure scheme to object store constructor"
This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.
* move encryption to stream store
* move decryption stuff to stream store
* revert changes in object store
* add encryptedBlockSize and close rangers on error during Get
* calculate padding sizes correctly
* encryptedBlockSize -> encryptionBlockSize
* pass encryption key and block size into stream store
* remove encryption key and block size from object store constructor
* move encrypter/decrypter initialization
* remove unnecessary cast
* Fix padding issue
* Fix linter
* add todos
* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata
* use different encryption key for each segment
* encrypt data in one step if it is small enough
* refactor and move encryption stuff
* fix errors related to nil slices passed to copy
* fix encrypter vs. decrypter bug
* put encryption stuff in eestream
* get captplanet test to pass
* fix linting errors
* add types for encryption keys/nonces and clean up
* fix tests
* more review changes
* add Cipher type for encryption stuff
* fix rs_test
* Simplify type casting of key and nonce
* Init starting nonce to the segment index
* don't copy derived key
* remove default encryption key; force user to explicitly set it
* move getSegmentPath to streams package
* dont require user to specify encryption key for captplanet
* rename GenericKey and GenericNonce to Key and Nonce
* review changes
* fix linting error
* Download uses the encryption type from metadata
* Store enc block size in metadata and use it for download
* add mb command
* forgot colon
* add command descriptions
* use utils.ParseURL in commands
* return error message instead of minio.BucketAlreadyExists in mb
* ls command with bucket store functionality
* rb command with bucket store functionality
* rm command with bucket store functionality
* newline
* use print rather than errs for messages, add no buckets messsage
* cp command with bucket store functionality
* remove deprecated getStorjObjects function
* defer utils.LogClose(f) on instead of defer f.Close()
* Check for no buckets after for loop
* add checks for unspecified bucket in bucket store methods
* fix incorrect return types
* add no path error messages in object store methods
* split copy into helpers
* srcObj scheme check in download
* print buckets instead of appending to slice
* check if destObj.Host != srcObj.Host
* better method of handling destination name if not specified
* uplink rename
* final cleanups
* trailing slash fixes
* linting
* more linting
* helpful error messages
* Adjust startAfter after merging #328
* Improve output messages
* Improved error check for empty bucket and path
* No page limit on client side. Rely on server side limit.
* Better time formatting
* Fix paths in recursive list results
* Normalize the dockerfiles and entrypoints
* fix formattting and docker-compose file
* add missing env var to satellite dockerfile
* Remove build args no longer in use
* More changes to make it all work
* Finally fix the binary builds for a couple os/arch combinations
* Make the file executable before adding it to the zip
* This cache hack wasn't suppose to make it into the PR
* Guard against calling the binary target by itself
* Order matters on OS X
* Vendored my image
* No 386 builds until storage is fixed.
* First pass at building binaries
* Finished building images
* Reworked some bits to save binaries in a better place.
* First pass at uploading binaries
* Builds for freebsd fail currently
* Add ignore file for docker builds to not invalidate the cache as often
* Docker image for 1.11-alpine exists now
* Name windows binaries correctly
* The makefile is used in tests.
* Take a swing at updating the tests for 1.11
* Switch to something with glibc so the race detector works.
* Remove unused .PHONY targets
* add CopyObject method
* use utils.LogClose
* extract common code from GetObject to getObject helper
* remove rr.Range from getObject helper, create helper putObject
* return rr, err in getObject helper
* extract code from PutObject into putObject helper
* remove commented out text
* remove other commented out code
* WIP trying to get storj cp command to work with copyObject
* fix typo in rb and now it works
* use rr.Size() instead of srcInfo.Size
* Revert "WIP trying to get storj cp command to work with copyObject"
This reverts commit e256b9f9a0fda728d41eb5b9d7a98b5446825842.
* add CopyObject test
* rebase and fix merge conflicts
* check error in gateway-storj test
* fix typo
* wip ca/ident cmds
* minor improvements and commenting
* combine id and ca commands and add $CONFDIR
* add `NewIdenity` test
* refactor `NewCA` benchmarks
* linter fixes
* First pass at an all-in-one compose project
* Sort out dependencies so uplink can build in docker
* Added the final bits to make this work
* Add readme for all-in-one
* Fix sed to work on OS X and Linux
* Make the readme for all-in-one better
* Only bring up the satellite and uplink after fixing the mock overlay
* Use a dummy value for mock overlay for now
* Finishing touches
* Fix url form for aws cli tool
* Move files out of all-in-one directory to help with directory confusion.
* Added a make target to make all-in-one even better
* move mock overlay from client to server
this doesn't really change much, but it does allow you to
run a standalone gateway against captain planet. it still does
not allow you to run a standalone gateway against a standalone
heavy client
* pointerdb: small error fixes
* some cleanups
* fix tests
* captplanet standalone farmer setup
* Bandwidth Allocation
* utils.Close method changed to utils.LogClose
* Get build temporarily working
* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function
* Update example client to reflect changes in client API
* Update ecclient to use latest PSClient, Make NewPSClient return error also
* Updated pieceranger tests to check for errors; sign method should take byte array
* Handle defers in store.go better
* Fix defer functions in psdb.go
* fun times
* Protobuf bandwidthallocation data is now a byte array
* Remove psservice package and merge it into pstore server
* Write wrapper for database calls
* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment
* Make PSDB tests implementation independent rather than method independent
* get rid of payer, renter in ecclient
* add context monitoring in store and retrieve
* WIP first commit
* change args check
* rename to cli, rename folder to storj, const defaultConfDir, use process.Exec()
* typos
* WIP create helper for getting minio objectlayer, changed setup command to try to match changes in setting up identity
* add rb command
* changes for setting up identity
* remove concurrency field from setupCfg
* rename variable returned by url.Parse to u across all commands for consistency
* add marker functionality to ls command
* add download functionality to cp command
* fix typo in rb init, use short variable names
* add statdb proto and example client
* server logic
* update readme
* remove boltdb from service.go
* sqlite3
* add statdb server executable file
* create statdb node table if it does not exist already
* get UpdateBatch working
* update based on jt review
* remove some commented lines
* fix linting issues
* reformat
* apiKey -> APIKey
* update statdb client apiKey->APIKey
* pkg/provider: with pkg/provider merged, make a single heavy client binary and deprecate old services
* add setup to gw binary too
* captplanet: output what addresses everything is listening on
* revert peertls/io_util changes
* define config flag across all commands
* use trimsuffix
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
* Add dockerfile and yaml for setting up piecestore servers
* Fix dockerfile for @aleitner (#115)
* Fix dockerfile for @aleitner
* Move files for @coyle
* Update yaml
* My linter had some errors so I resolved them
* Make jenkins do the needful
* Make piecestore-farmer look like overlay's build process
* Fix service spec to work in staging
* Make Jenkins push images, but not deploy them, yet.
* Modify entrypoint to fit new verbs
* Update piecestore-farmer entrypoint script to handle new app output
* WIP creating admin node service
- WIP changing the process pkg to accept multiple services
- WIP looping over services passed to process
- add netstate/service.go file and abstract it for service processing
* implement goroutine to launch each process
* goroutines working with multiple services
* code review fixes
* more code updates for review
* Add pkg lock and mod files back in
* code review updates
* update process.Main with better concurrent error handling
* Update error handling and pass ctx to StartService
* Update error handling with channel implementation
* Merge in upstream changes
- Simplify error handling channels
* updates
* Updates per reviewable
* fix test
* Setup test exec
* Scaffold test setup
* process main test working
* update admin process test
* Test multiple processes done
* Add error classes for testing, test main logger error
* Updates to tests
* Update how process.Main() handles configs
* Complete merge
* Update Gopkg and add Copyright
* Fix cyclical import issue
- Added .coverprofile to gitignore
- Update admin main.go function call
* remove unnecessary line
* Updates
* DRY up cmd/netstate package
* update service function calls
* updates
* Trying no-ops in examples
* rename netstate to pointerdb
* trying to fix merge
* dep ensure and run tests
* remove flag.Parse
* Update deps
* Skip offending test in pkg/process, to be fixed later
* separated commands and actions from main.go, created delete command
* refactored piecestore-farmer to use cobra CLI
* added piecestore-farmer main.go and long command descriptions
* pkg/process: start replacing pkg/process with cobra helpers
* address review comments
* addressed github comments, rename pstore.DetermineID to GenerateID, remove utils.NewID and use pstore.GenerateID instead
* addressed comments
* pulled proc and got CLI working with process package
* goimports
* stdout formatting
* WIP adding tests for pstoreFarmer CLI, fixed bug in check for preexisting config
* copyright header, change exported variable
* changed error formatting to be consistent
* more tests
* stdout, add (\n)s in zap logs, set home inside SetConfigPath() instead of passing in
* resolved merge conflicts
* goimports and fixed merge error
* removed generateID func to use kademlia.NewID instead, refactored to match kademlia.NewKademlia()
* Implement psclient interface
* Add string method to pieceID type
* try to fix linter errors
* Whoops missed an error
* More linter errors
* Typo
* Lol double typo
* Get everything working, begin adding tests for psclient rpc
* goimports
* Forgot to change the piecestore cli when changed the piecestore code
* Fix CLI
* remove ID length, added validator to pieceID
* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer
* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser
* Forgot to remove offset
* Added message upon successful store
* Do that thing dennis and kaloyan wanted
* goimports
* Make closeConn a part of the interface for psclient
* Use interface
* Removed uneccessary new lines
* goimport
* Whoops
* Actually we don't want to use the interface in Piece Ranger
* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples
* Make comments look nicer
* piecestore: connect to kad
* piecestore: Linter errors
* piecestore: added pstore config command line utility
* piecestore: removed main.go, implement methods and structs
* piecestore: Import Cam's config code into piecestore-farmer
* piecestore: moving farmer to urfavecli
* piecestore: added create command
* piecestore: Removed old config, added server start code to cli
* piecestore: Get server code working
* piecestore: Changed default dir for storing piece store data; added ttl to config
* piecestore: Generate id; add bootstrap ip for kad
* piecestore: Separate kad port and server port
* piecestore: goimports
* piecestore: Removed print
* piecestore: use pkg/process
* piecestore: Better config file
* piecestore: base58 encode for id
* piecestore: base58 encode and clean up cli
* piecestore: Typo
* piecestore: removed unnecessary variable
* piecestore: Fixed more typos
* piecestore: Place data in a directory based on nodeid
* piecestore: base58 encode instead
* piecestore: Add dependency to go.mod
* piecestore: Fix typo in rpc server start; clear data on failed piece upload
* add reference to dht to overlay client struct
* wip
* wip
* Implement FindNode
* get nodes
* WIP
* Merge in Dennis kademlia code, get it working with our code
* ping and moar
* WIP trying to get cache working with kademlia
* WIP more wiring up
* WIP
* Update service cli commands
* WIP
* added GetNodes
* added nodes to Kbucket
* default transport changed to TCP
* GetBuckets interface changed
* filling in more routing
* timestamp methods
* removed store
* Added initial network overlay explorer page
* Updating and building with dockerfile
* Working on adding bootstrap node code
* WIP merging in dennis' code
* WIP
* connects cache to pkg/kademlia implementation
* WIP redis cache
* testing
* Add bootstrap network function for CLI usage
* cleanup
* call bootstrap on init network
* Add BootstrapNetwork function to interface
* Merge in dennis kad code
* WIP updates to redis/overlay client interface
* WIP trying to get the DHT connected to the cache
* go mod & test
* deps
* Bootstrap node now setting up correctly
- Need to pass it through CLI commands better
* WIP adding refresh and walk functions, added cli flags
- added cli flags for custom bootstrap port and ip
* PR comments addressed
* adding FindStorageNodes to overlay cache
* fix GetBucket
* using SplitHostPort
* Use JoinHostPort
* updates to findstoragenodes response and request
* WIP merge in progress, having issues with a panic
* wip
* adjustments
* update port for dht bootstrap test
* Docker
* wip
* dockerfile
* fixes
* makefile changes
* Update port in NewKademlia call
* Update local kademlia DHT config
* kubernetes yaml
* cleanup
* making tests pass
* k8s yaml
* lint issues
* Edit cli flags to allow for configurable bootstrap IP and Port args
* cleanup
* cache walking the network now
* Rough prototype of Walk function laid out
* Move walk function into bootstrap function
* Update dht.go
* changes to yaml
* goimports
* added spawn scripts
* wip
* Make server not require length
* Added test for 0 length
* Added test for accepting length of 0
* linter errors
* Make store not take anything more than an id and maybe a ttl if you want
* initial commit for PUT request authorization
* inital auth for put request
* auth. request working for all req
* deleted library
* removed .db files
* work in progress for modifying test suite to accomodate credentials
* modified tests
* gofmt, fixed code based on suggestions; test passed
* gofmt again
* merged nat's update on pointers, passed tests, cleanup from git rebase
* fixed fmt
* fixed fmt on tests
* work in progress
* reduced code
* fixed naming conventions
* added line in code
* fixed server bug, merged new code to server, added env
* fixed linter; getting cright issues on piecestore
* added comments for what passes on the creds
* added spawn scripts
* Determine random id for storing
* Moved determine id to rpc example
* Added tests
* Better test
* goimports
* Updated tests
* Fix typos
* Added piecestore
* gofmt
* Added requested changes
* Added cli
* Removed ranger because I wanted something that can stand alone
* Add example of http server using piece store
* Changed piecestore code to make it more optial for error handelling
* Merged with piecestore
* Added missing package
* Forgot io import
* gofmt
* gofmt
* Forgot io
* Make path by hash exported
* updated to simplify again whoops
* Updated server to work real good
* Forgot ampersand
* Updated to match FilePiece
* Merged in cam's delete code
* Remove unused io
* Added RPC code
* Give the download request a reader
* Removed http server stuff; changed receive stream to say io.reader
* Added expiration date to shardInfo
* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions
* added ttl info request
* Move scripts to http server pr; added close method for Retrieve api
* added rpc server tests for getting piece meta data and retrieval routes
* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf
* Imported cams test
* Bump gometalinter deadline
* WIP adding tests
* added tests for store and delete routes
* Add changes as requested by Kaloyan, also cleaned up some code
* Get the code actually working whoops
* More cleanup
* Separating database calls from api.go
* need to rename expiration
* Added some changes requested by JT
* Fix read size
* Fixed total amount to read
* added tests
* Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages
* Moved rpc client and server to piece store
* Moved piecestore protobuf to the /protos folder
* Cleaned up messages
* Clean up linter errors
* Added missing sqlite import
* Add ability to do iterative reads and writes to pstore
* Incrementally read data
* Fix linter and import errors
* Solve linter Error
* Change return types
* begin test refactor
* refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go
* Cleaning up tests
* Added retrieve tests
* refactored delete tests
* Deleted old tests
* Updated cmd/piecestore to reflect changes to piecestore
* Refactored server tests and server/client store code
* gofmt
* WIP implementing TTL struct
* Read 4k at a time when Retrieving
* implemented ttl struct
* Accidentally removed fpiece dependency?
* Double resolve merge conflict
* Moved client to the examples since it is an example
* Change hash to id in protobuf. TODO: Change client and server to reflect these changes
* Update client and protobuf
* changed hash to id
* Handle eof properly in retrieve
* Id -> ID
* forgot to change import for client after moving
* Moved client and server main to examples
* Made changes requested by JT
* checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close()
* Print not fatal
* Handle overflow when reading from server
* added const IDLength
* removed types from comments
* Add reader/writer for download data from and uploading data to server
* goimports and comments
* fixed nits, casts, added OK const, DBCleanup now exits program on error
* Add stream reader and writer for server
* Fix errors
* i beforee except after c lol
* customizable data dir
* Forgot to rename variable
* customizable data dir
* Handle closing of server stream properly
* linter
* pass on inserting the same data twice
* linter
* linter
* Do the easy things JT asked for
* Handle overflow from reads properly; handle custom db path
* Handle overflow for server stream read; TODO Combine server and client stream reads
* Allow for TTL of 0 to stay forever
* Make Client cleaner for user
* adds pointer to netstate proto file
* generated updated netstate proto
* changes boltdb netstate to save pointers as values
* updates netstate Put to save Pointers, updates client example to put a pointer, adds grpc status errors, updates tests, changes boltdb 'File' struct to 'PointerEntry'
* updates netstate client example and client test to save pointers, updates netstate List and Delete
* begins adding netstate-http tests
* removes netstate http service
* re-adds netstate auth
* updates boltdb netstate test
* changes encrypted_unencrypted_size from int64 to bytes in netstate proto
* updates READMEs
* Added piecestore
* gofmt
* Added requested changes
* Added cli
* Removed ranger because I wanted something that can stand alone
* Changed piecestore code to make it more optial for error handelling
* Added missing package
* Forgot io import
* gofmt
* Make path by hash exported
* updated to simplify again whoops
* Forgot ampersand
* Updated to match FilePiece
* Change store to use io.CopyN
* Clear golinter errors
* Updated go.mod
* Updated go.mod
* adds proto files for netstate crud
* moves netstate grpc client lib into pkg/netstate where grpc netstate service is defined
* starts adding grpc client and server tests
* moves creation of grpc server into cmd/netstate/main.go, removes pkg/netstate/service.go, adds more client testing
* changed all 'Path' and 'Value' fields from strings to bytes, updated tests
* changes Get and Delete in proto file to receive 'requests' instead of 'file paths', adds tests for Get, List, and Delete
* changes netstate-routes to get 'fileValue' bytes not 'fileInfo'
* adds example rpc client in 'examples' and adds more specific debug logs
* adds readmes for netstate rpc services and updates netstate-routes
* Light client implementation with get-info and list-buckets commands
* Fix client package name
* Fix go.mod to work with vgo
* Use single `fmt.Printf` for `get-info` output
* Use unnamed import to client package
* Simplify usage of sha256 and sha512 sums
* Remove obsolete test code
* Use helper structs for unmarshalling bridge info
* Remove LGPL license files and adjust copyright headers
* Use github.com/zeebo/errs
* Use httptest for test http server
* Use viper for env var management
* Nested struct for swagger
* Add github.com/zeebo/errs to go.mod
* More bucket tests
* word wrap long line
* Use zeebo/errs for crypto errors
