* add cache, update cache w/piece create/delete
* add service w/loop to cache to recalculate space used cache
* add piecestore cache to other sn svcs to use
* add table to persist the total space used
* rm cache where not needed
* rm stuff from sn svcs
* start fixing tests, changes per comments
* update commits
* add unit tests
* fix commiting before we write header bytes
* fix cache create test
* copy cache map, add started back to recalc
* fix test
* add test, update comments
Deprecate the pieceinfo database, and start storing piece info as a header to
piece files. Institute a "storage format version" concept allowing us to handle
pieces stored under multiple different types of storage. Add a piece_expirations
table which will still be used to track expiration times, so we can query it, but
which should be much smaller than the pieceinfo database would be for the
same number of pieces. (Only pieces with expiration times need to be stored in piece_expirations, and we don't need to store large byte blobs like the serialized
order limit, etc.) Use specialized names for accessing any functionality related
only to dealing with V0 pieces (e.g., `store.V0PieceInfo()`). Move SpaceUsed-
type functionality under the purview of the piece store. Add some generic
interfaces for traversing all blobs or all pieces. Add lots of tests.
* re-organizing into bandwidth service. re-enable rollup loop
* Prevent uniqueness failure in bandwidth rollup
* Add test to make sure the rollup select date range works correctly
* add bandwidth config for rollup interval
* storagenode: remove datetime calls in favor of UTC
datetime only has second level granularity whereas string
comparisons don't. Since we're wiping everything anyway, it's
easier to just use UTC everywhere rather than migrate to
datetime calls.
* add utcdb to check that arguments are utc
* storagenodedb: add trivial tests to ensure calls work
This at least tests that all of the timestamps passed in are
in the UTC timezone.
* fix truncated comment and change migrations to be UTC
- Drops some unused indexes
- Applies a computed index to timestamp columns
- Applies a partial index for expired pieces
- Uses BETWEEN to avoid some datetime calls
- Filters expired piece search by those that aren't NULL
* Add in-memory cache for bandwidth and space usage monitoring
* moved some structs around and added error handling for get piece size query
* added to existing bandwidth test. fixed typo
* added test, updates from PR review, added monkit for new methods
* PR review updates. renamed space used methods
* changed bw cache so that only Add updates the cache and it only overwrites when the date moves forward
* moved bandwidth usage to bw and space usage to pieceinfodb
* fixed interface comment
* removed pointer from sync.Once
* remove infodb locks and give a unique name for each in memory created.
* changed max idle and open to 1 for memory DBs. fixes table locking errors
* fixed race condition
* added file based infodb test
* added busy timeout parameter to the file based infodb for testing
* fixed imports
* removed db.locked() after merge from master
