* remove api key from statdb server reqs; add statdb UpdateUptime and UpdateAuditSuccess to server
* update api key authentication in statdb server
* add todos for future statdb updates
* add UpdateUptime and UpdateAuditSuccess to statdb server
* fix apikey stuff in config.go and statdb_test.go
* fix tests
* update sdbclient.NewClient call in audit package
* fix UpdateUptime and UpdateAuditSuccess in sdbclient
* set api key from statdb/config.go
* change package for statdb tests
* linter fixes
* remove todo comments
* fix sdbclient err checking
* move validate auth functionality to auth package
* update description for statdb api key
* remove import
* add sdbclient.UpdateUptime; update args for sdbclient.CreateEntryIfNotExists
* add auditcount to node stats; restructure statdb.CreateEntryIfNotExists
* add noop mock sdbclient
* add the ability to create a node in statdb without "default" stats
* update statdb.CreateEntryIfNotExists
* take fewer args for sdbclient.CreateWithStats/FindValidNodes
* add sdbclient.UpdateAuditSuccess
* update sdbclient.Update so that all fields are updated when called (reduce args)
* update error checking in statdb.Create
* protobuf for sending bandwidth agreements to satellite from storage nodes
* Setup process for sending agreements
* Add payer_id to db with bandwidth agreements for better sorting
* Linter errors
* Read agreements from PSDB
* Try writing message to server
* Cleanup
* Basic functionality
* Better error handelling
* Fix test
* setup config and server structure for receiving bandwidth agreements
* Resolve linter issues
* Optional commit for if we want to handle deletes all at once
* add identity to Server, add logic for receiving bandwidth messsages
* Bandwidth agreement DBX creation and integration with bw agreement endpoint
Co-authored-by: Kishore <kishore@storj.io>
Co-authored-by: Cam <cameron@storj.io>
* protobuf for sending bandwidth agreements to satellite from storage nodes
* Setup process for sending agreements
* Add payer_id to db with bandwidth agreements for better sorting
* Linter errors
* Read agreements from PSDB
* Try writing message to server
* Cleanup
* Basic functionality
* Better error handelling
* Fix test
* setup config and server structure for receiving bandwidth agreements
* Resolve linter issues
* Optional commit for if we want to handle deletes all at once
* add identity to Server, add logic for receiving bandwidth messsages
* Bandwidth agreement DBX creation and integration with bw agreement endpoint
Co-authored-by: Kishore <kishore@storj.io>
Co-authored-by: Cam <cameron@storj.io>
* added postgres create/read/delete test function
Co-authored-by: kishore <kishore@storj.io
Co-authored-by: cam <cameron@storj.io>
* edit comment
* removed sqlite3 driver from dbx
* remove generated sqlite code, add dbx read limitoffset
* remove getServerAndDB function, rename getDBPath to getPSQLInfo
* WIP writing server endpoint test
* code review changes
* protobuf for sending bandwidth agreements to satellite from storage nodes
* Setup process for sending agreements
* Add payer_id to db with bandwidth agreements for better sorting
* Renamed payer to satellite in psdb
The old paths.Path type is now replaced with the new storj.Path.
storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().
As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:
storj.SplitPath
storj.JoinPaths
encryption.EncryptPath
encryption.DecryptPath
encryption.DerivePathKey
encryption.DeriveContentKey
All code in master is migrated to the new storj.Path type.
The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go
This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
* begin adding encryption for remote pieces
* begin adding decryption
* add encryption key as arg to Put and Get
* move encryption/decryption to object store
* Add encryption key to object store constructor
* Add the erasure scheme to object store constructor
* Ensure decrypter is initialized with the stripe size used by encrypter
* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"
This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.
* Revert "Add the erasure scheme to object store constructor"
This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.
* move encryption to stream store
* move decryption stuff to stream store
* revert changes in object store
* add encryptedBlockSize and close rangers on error during Get
* calculate padding sizes correctly
* encryptedBlockSize -> encryptionBlockSize
* pass encryption key and block size into stream store
* remove encryption key and block size from object store constructor
* move encrypter/decrypter initialization
* remove unnecessary cast
* Fix padding issue
* Fix linter
* add todos
* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata
* use different encryption key for each segment
* encrypt data in one step if it is small enough
* refactor and move encryption stuff
* fix errors related to nil slices passed to copy
* fix encrypter vs. decrypter bug
* put encryption stuff in eestream
* get captplanet test to pass
* fix linting errors
* add types for encryption keys/nonces and clean up
* fix tests
* more review changes
* add Cipher type for encryption stuff
* fix rs_test
* Simplify type casting of key and nonce
* Init starting nonce to the segment index
* don't copy derived key
* remove default encryption key; force user to explicitly set it
* move getSegmentPath to streams package
* dont require user to specify encryption key for captplanet
* rename GenericKey and GenericNonce to Key and Nonce
* review changes
* fix linting error
* Download uses the encryption type from metadata
* Store enc block size in metadata and use it for download
* storage node quick check and startup validation
* rearranged the startup validation and quick check logic
* travis lint warning fixes
* travis lint warning fixes
* travis lint warning fixes
* code changes per review comments
* code clean dev debug info
* travis lint wranings
* code changes per code review comments
* code changes per code review comments
* code update per review
* sqlite SUM is having issue when getting the SUM of an empty column; filepath was checking a directory that doesn't exist when starting server; Example updated to print allocated and used space
* storage node quick check and startup validation
* rearranged the startup validation and quick check logic
* travis lint warning fixes
* travis lint warning fixes
* travis lint warning fixes
* code changes per review comments
* code clean dev debug info
* travis lint wranings
* code changes per code review comments
* code changes per code review comments
* code update per review
* no file or directory error
* Updated mock PSClient
* Added initial functions for signing and verifying
* whoops
* Get client up to speed
* Added initial functions for signing and verifying
* whoops
* Get client up to speed
* wip
* wip
* actual signatures in tests
(cherry picked from commit 1464853b737f1d712d64fbf90147f535525c8fd9)
* bugfixing
* Generate private key in example
* Generate signatures for pieceranger tests
* Update examples to use TLS
* Use private key from identity inside of example
* Use crypto.PrivateKey interface
* Change err name in defers
* Pass tests
* Pass identity Key to PSClient
* Get tests passing on travis
* Resolve linter complaints
* begin adding tls
* remove incomplete line in gw/main.go
* identity fixes+:
+ fix `peertls.NewCert` public key issue
+ fix `peertls.verfiyChain` issue
+ fix identity dial option
+ rename `GenerateCA` to `NewCA` and `generateCAWorker` to `newCAWorker` for better consistency/convention
* use pdbclient instead of pointerdb in miniogw
* fix tests
* go fmt
* make review changes
* modify how context.Background() is used
* more context stuff
* pointerdb: separate client and server packages
the reason for this is so that things the server needs (bolt, auth)
don't get imported if all you need is the client. will result in
smaller binaries and less flag definitions
* review comments
* captplanet standalone farmer setup
* Bandwidth Allocation
* utils.Close method changed to utils.LogClose
* Get build temporarily working
* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function
* Update example client to reflect changes in client API
* Update ecclient to use latest PSClient, Make NewPSClient return error also
* Updated pieceranger tests to check for errors; sign method should take byte array
* Handle defers in store.go better
* Fix defer functions in psdb.go
* fun times
* Protobuf bandwidthallocation data is now a byte array
* Remove psservice package and merge it into pstore server
* Write wrapper for database calls
* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment
* Make PSDB tests implementation independent rather than method independent
* get rid of payer, renter in ecclient
* add context monitoring in store and retrieve
* add statdb proto and example client
* server logic
* update readme
* remove boltdb from service.go
* sqlite3
* add statdb server executable file
* create statdb node table if it does not exist already
* get UpdateBatch working
* update based on jt review
* remove some commented lines
* fix linting issues
* reformat
* apiKey -> APIKey
* update statdb client apiKey->APIKey
* Refactor List in Pointer DB
* Fix pointerdb-client example
* Fix issue in Path type related to empty paths
* Test for the PointerDB service with some fixes
* Fixed debug message in example: trancated --> more
* GoDoc comments for unexported methods
* TODO comment to check if Put is overwriting
* Log warning if protobuf timestamp cannot be converted
* TODO comment to make ListPageLimit configurable
* Rename 'segment' package to 'segments' to reflect folder name
* captplanet
I kind of went overboard this weekend.
The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!
To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.
Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.
A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
* Kademlia
* OverlayCache
* PointerDB
* StatDB
* PieceStore
* etc.
A Provider is a collection of Responsibilities that
share an Identity, such as:
* The heavy client
* The farmer
* The gateway
An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.
Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.
To make a Responsibility, one should create a "config"
struct, such as:
```
type Config struct {
RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```
To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.
cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.
Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
* cmd/captplanet/farmer/main.go (a new farmer binary)
* cmd/captplanet/hc/main.go (a new heavy client binary)
* cmd/captplanet/gw/main.go (a new minio gateway binary)
Usage:
```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```
Configuration is placed by default in `~/.storj/capt/`
Other changes:
* introduces new config structs for currently existing
Responsibilities that conform to the new Responsibility
interface. Please see the `pkg/*/config.go` files for
examples.
* integrates the PointerDB API key with other global
configuration via flags, instead of through environment
variables through viper like it's been doing. (ultimately
this should also change to use the PointerDB config
struct but this is an okay shortterm solution).
* changes the Overlay cache to use a URL for database
configuration instead of separate redis and bolt config
settings.
* stubs out some peer identity skeleton code (but not the
meat).
* Fixes the SegmentStore to use the overlay client and
pointerdb clients instead of gRPC client code directly
* Leaves a very clear spot where we need to tie the object to
stream to segment store together. There's sort of a "golden
spike" opportunity to connect all the train tracks together
at the bottom of pkg/miniogw/config.go, labeled with a
bunch of TODOs.
Future stuff:
* I now prefer this design over the original
pkg/process.Service thing I had been pushing before (sorry!)
* The experience of trying to have multiple farmers
configurable concurrently led me to prefer config structs
over global flags (I finally came around) or using viper
directly. I think global flags are okay sometimes but in
general going forward we should try and get all relevant
config into config structs.
* If you all like this direction, I think we can go delete my
old Service interfaces and a bunch of flags and clean up a
bunch of stuff.
* If you don't like this direction, it's no sweat at all, and
despite how much code there is here I'm not very tied to any
of this! Considering a lot of this was written between midnight
and 6 am, it might not be any good!
* bind tests
* working on put request for nsclient
* working on put request for nsclient
* netstate put
* netstate put
* wip testing client
* wip - testing client
and working through some errors
* wip - testing client
and working through some errors
* put request works
* put request works for client
* get request working
* get request working
* get request working-minor edit
* get request working-minor edit
* list request works
* list request works
* working through delete error
* working through delete error
* fixed exp client, still working through delete error
* fixed exp client, still working through delete error
* delete works; fixed formatting issues
* delete works; fixed formatting issues
* deleted comment
* deleted comment
* resolving merge conflicts
* resolving merge conflict
* fixing merge conflict
* implemented and modified kayloyans paths file
* working on testing
* added test for path_test.go
* fixed string, read through netstate test
* deleted env variables
* initial commit for mocking out grpc client- got it working
* mocked grpc client
* mock put passed test
* 2 tests pass for PUT with mock
* put requests test pass, wip- want mini review
* get tests pass mock
* list test working
* initial commit for list test
* all list req. working, starting on delete tests
* delete tests passed
* cleaned up tests
* resolved merge conflicts
* resolved merge conflicts
* fixed linter errors
* fixed error found in travis
* initial commit for fixes from PR comments
* fixed pr comments and linting
* added error handling for api creds, and rebased
* fixes from dennis comments
* fixed pr with dennis suggestioon
* added copyrights to files
* fixed casing per dennis great comment
* fixed travis complaint on sprintf
* Updates to config handling
- Add functions to load in configs
- Standardize location and naming of config files
- Configuration over convention style of config file handling for each
service
* update config handling and correclty handle cli flags being set
* generate configs from default if no config is found
- renamed pointerdbDB to pointerdb for clarity in config file
- set sane default for pkg/overlay boltDB file
- set srvPort to default to 8082 to avoid port collision on default
setting
* linter updates
* move boltdb path vars into function
* update tests to handle config environment changes
* --fix exec test mocks
* update tests to use viper instead of flag library
* fix typo
* add redis-server to services in travis for tests
* update examples with new config env function signature
* fix tests
* WIP creating admin node service
- WIP changing the process pkg to accept multiple services
- WIP looping over services passed to process
- add netstate/service.go file and abstract it for service processing
* implement goroutine to launch each process
* goroutines working with multiple services
* code review fixes
* more code updates for review
* Add pkg lock and mod files back in
* code review updates
* update process.Main with better concurrent error handling
* Update error handling and pass ctx to StartService
* Update error handling with channel implementation
* Merge in upstream changes
- Simplify error handling channels
* updates
* Updates per reviewable
* fix test
* Setup test exec
* Scaffold test setup
* process main test working
* update admin process test
* Test multiple processes done
* Add error classes for testing, test main logger error
* Updates to tests
* Update how process.Main() handles configs
* Complete merge
* Update Gopkg and add Copyright
* Fix cyclical import issue
- Added .coverprofile to gitignore
- Update admin main.go function call
* remove unnecessary line
* Updates
* DRY up cmd/netstate package
* update service function calls
* updates
* Trying no-ops in examples
* rename netstate to pointerdb
* trying to fix merge
* dep ensure and run tests
* remove flag.Parse
* Update deps
* Skip offending test in pkg/process, to be fixed later
