* fix audit stripe selector to work if last segment is smaller than stripe size
* fix audit bug related to indexing an incomplete list of nodes returned by overlay
* add storeConfig struct and getSegmentStore helper for creating a segment store
* implement segment store in repairer, remove unnecessary repairer Repair method
* change repair method parameter from int to int32 to match type being passed in
* implement repairer service in captplanet
* rework Config, set Config defaults in captplanet/setup
* protobuf for sending bandwidth agreements to satellite from storage nodes
* Setup process for sending agreements
* Add payer_id to db with bandwidth agreements for better sorting
* Linter errors
* Read agreements from PSDB
* Try writing message to server
* Cleanup
* Basic functionality
* Better error handelling
* Fix test
* setup config and server structure for receiving bandwidth agreements
* Resolve linter issues
* Optional commit for if we want to handle deletes all at once
* add identity to Server, add logic for receiving bandwidth messsages
* Bandwidth agreement DBX creation and integration with bw agreement endpoint
Co-authored-by: Kishore <kishore@storj.io>
Co-authored-by: Cam <cameron@storj.io>
* protobuf for sending bandwidth agreements to satellite from storage nodes
* Setup process for sending agreements
* Add payer_id to db with bandwidth agreements for better sorting
* Linter errors
* Read agreements from PSDB
* Try writing message to server
* Cleanup
* Basic functionality
* Better error handelling
* Fix test
* setup config and server structure for receiving bandwidth agreements
* Resolve linter issues
* Optional commit for if we want to handle deletes all at once
* add identity to Server, add logic for receiving bandwidth messsages
* Bandwidth agreement DBX creation and integration with bw agreement endpoint
Co-authored-by: Kishore <kishore@storj.io>
Co-authored-by: Cam <cameron@storj.io>
* added postgres create/read/delete test function
Co-authored-by: kishore <kishore@storj.io
Co-authored-by: cam <cameron@storj.io>
* edit comment
* removed sqlite3 driver from dbx
* remove generated sqlite code, add dbx read limitoffset
* remove getServerAndDB function, rename getDBPath to getPSQLInfo
* WIP writing server endpoint test
* code review changes
* protobuf for sending bandwidth agreements to satellite from storage nodes
* Setup process for sending agreements
* Add payer_id to db with bandwidth agreements for better sorting
* Renamed payer to satellite in psdb
* add filter field into OverlayOptions message
* chooseFiltered method, add excluded parameter in populate method
* change excluded type to []dht.NodeID in ChooseFiltered, change comment
* change name filter to excluded_nodes in proto
* implement helper function contains
* delete ChooseFiltered and add its functionality into Choose method to keep original author's history, add excluded argument into Choose calls
* regenerate mock_client.go
* regenerate protobuf
* adding the repair() func
* update test case to use new IDFromString function
* modified the repair() and updated streams mock
* modified the repair() and updated streams mock
* Options struct
* adding the repair() func
* modified the repair() and updated streams mock
* modified the repair() and updated streams mock
* integrating the segment repair()
* development repair with hack working
* repair segment changes
* integrated with mini hacks and rigged up test case with dev debug info
* integrated with ec and overlay
* added repair test case
* made the getNewUniqueNodes() to recursively go thru choose() to find get the required number of unique nodes
* cleaned up code
* disconnect from nodeclient
* cleanup connections in tests
* kademlia disconnects from nodeclient
* updating disconnect method for mocks
* creates separate disconnect and removeAll methods for tests
* adds init to connection pool
* fix folder cleanup and disconnect
* creates and cleans up test db files and disconnects kad
* removes db/.keep
* includes disconnect within cleanup methods
* creates public init method on connection pool to handle mutex copy issues
* remove all after disconnect
* pair creation and destruction
* checks disconnect error
* remove ctx
* fixes mock kad
The old paths.Path type is now replaced with the new storj.Path.
storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().
As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:
storj.SplitPath
storj.JoinPaths
encryption.EncryptPath
encryption.DecryptPath
encryption.DerivePathKey
encryption.DeriveContentKey
All code in master is migrated to the new storj.Path type.
The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go
This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
..although it ought to work for other storage.KeyValueStore needs as
well. it's just optimized to work pretty well for a largish hierarchy of
paths.
This includes the addition of "long benchmarks" for KeyValueStore
testing. These will only be run when -test-bench-long is added to the
test flags. In these benchmarks, a large corpus of paths matching a
natural ("real-life") hierarchy is read from paths.data.gz (which you
can get from https://github.com/storj/path-test-corpus) and imported
into a particular KeyValueStore. Recursive and non-recursive queries are
run on it to detect performance problems that arise only at scale.
This also includes alternate implementation of the postgreskv client,
which works in a less-bizarre way for non-recursive queries, but suffers
from poor performance in tests such as the long benchmarks. Once this
alternate impl is committed to the tree, we can remove it again; I just
want it to be available for future reference.
