We don't use reverse listing in any of our code, outside of tests, and
it is only exposed through libuplink in the
lib/uplink.(*Project).ListBuckets() API. We also don't know of any users
who might have a need for reverse listing through ListBuckets().
Since one of our prospective pointerdb backends can not support
backwards iteration, and because of the above considerations, we are
going to remove the reverse listing feature.
Change-Id: I8d2a1f33d01ee70b79918d584b8c671f57eef2a0
* set up redis support in live accounting
* move live.Service interface into accounting package and rename to Cache, pass into satellite
* refactor Cache to store one int64 total, add IncrBy method to redis client implementation
* add monkit tracing to live accounting
..although it ought to work for other storage.KeyValueStore needs as
well. it's just optimized to work pretty well for a largish hierarchy of
paths.
This includes the addition of "long benchmarks" for KeyValueStore
testing. These will only be run when -test-bench-long is added to the
test flags. In these benchmarks, a large corpus of paths matching a
natural ("real-life") hierarchy is read from paths.data.gz (which you
can get from https://github.com/storj/path-test-corpus) and imported
into a particular KeyValueStore. Recursive and non-recursive queries are
run on it to detect performance problems that arise only at scale.
This also includes alternate implementation of the postgreskv client,
which works in a less-bizarre way for non-recursive queries, but suffers
from poor performance in tests such as the long benchmarks. Once this
alternate impl is committed to the tree, we can remove it again; I just
want it to be available for future reference.
* Let's do it right this time
* Oh travis...
* Handle redis URL
* Travis... why u gotta be like this?
* Handle when address does not use redis scheme
* Start repairer
* Match provider.Responsibility interface
* Simplify if statement
* Config doesn't need to be a pointer
* Initialize doesn't need to be exported
* Don't run checker or repairer on startup
* Fix travis complaints
1. Added KeyValueStore.Iterate for implementing the different List, ListV2 etc. implementations. This allows for more efficient use of memory depending on the situation.
2. Implemented an inmemory teststore for running tests. This should allow to replace MockKeyValueStore in most places.
3. Rewrote tests
4. Pulled out logger from bolt implementation so it can be used for all other storage implementations.
5. Fixed multiple things in bolt and redis implementations.
Fixes go1.11 vet warnings.
Cancel on WithTimeout must always be called to avoid memory leak:
pkg/provider/provider.go:73: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak
Range over non-copyable things:
pkg/pool/connection_pool_test.go:32: range var v copies lock: struct{pool pool.ConnectionPool; key string; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:56: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:83: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected interface{}; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
zeebo/errs package always requires formatting directives:
pkg/peertls/peertls.go:50: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:47: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:87: Class.New call has arguments but no formatting directives
pkg/overlay/cache.go:94: Class.New call has arguments but no formatting directives
pkg/provider/certificate_authority.go:98: New call has arguments but no formatting directives
pkg/provider/identity.go:96: New call has arguments but no formatting directives
pkg/provider/utils.go:124: New call needs 1 arg but has 2 args
pkg/provider/utils.go:136: New call needs 1 arg but has 2 args
storage/redis/client.go:44: Class.New call has arguments but no formatting directives
storage/redis/client.go:64: Class.New call has arguments but no formatting directives
storage/redis/client.go:75: Class.New call has arguments but no formatting directives
storage/redis/client.go:80: Class.New call has arguments but no formatting directives
storage/redis/client.go:92: Class.New call has arguments but no formatting directives
storage/redis/client.go:96: Class.New call has arguments but no formatting directives
storage/redis/client.go:102: Class.New call has arguments but no formatting directives
storage/redis/client.go:126: Class.New call has arguments but no formatting directives
* adds netstate rpc server pagination, mocks pagination in test/util.go
* updates ns client example, combines ns client and server test to netstate_test, adds pagination to bolt client
* better organizes netstate test calls
* wip breaking netstate test into smaller tests
* wip modularizing netstate tests
* adds some test panics
* wip netstate test attempts
* testing bug in netstate TestDeleteAuth
* wip fixes global variable problem, still issues with list
* wip fixes get request params and args
* fixes bug in path when using MakePointers helper fn
* updates mockdb list func, adds test, changes Limit to int
* fixes merge conflicts
* fixes broken tests from merge
* remove unnecessary PointerEntry struct
* removes error when Get returns nil value from boltdb
* breaks boltdb client tests into smaller tests
* renames AssertNoErr test helper to HandleErr
* adds StartingKey and Limit parameters to redis list func, adds beginning of redis tests
* adds helper func for mockdb List function
* if no starting key provided for netstate List, the first value in storage will be used
* adds basic pagination for redis List function, adds tests
* adds list limit to call in overlay/server.go
* streamlines/fixes some nits from review
* removes use of obsolete EncryptedUnencryptedSize
* uses MockKeyValueStore instead of redis instance in redis client test
* changes test to expect nil returned for getting missing key
* remove error from `KeyValueStore#Get`
* fix bolt test
* Merge pull request #1 from bryanchriswhite/nat-pagination
remove error from `KeyValueStore#Get`
* adds Get returning error back to KeyValueStore interface and affected clients
* trying to appease travis: returns errors in Get calls in overlay/cache and cache_test
* handles redis get error when no key found
* internal/test: switch errors to error classes
if you construct an error directly at package init time, you
won't get useful stack traces or anything. zeebo/errs expects that
you always construct an error (.New) when the error actually
happens. instead, you call Class at init time, then use (Class).Has
to test for error type membership.
* fix linter
* fix test
