* Added a gc package at satellite/gc, which contains the gc.Service, which runs garbage collection integrated with the metainfoloop, and the gc PieceTracker, which implements the metainfo loop Observer interface and stores all of the filters (about which pieces are good) for each node.
* Added a gc config located at satellite/gc/service.go (loop disabled by default in release)
* Creates bloom filters with pieces to be retained inside the metainfo loop
* Sends RetainRequests (or filters with good piece ids) to all storage nodes.
* storagenode: remove datetime calls in favor of UTC
datetime only has second level granularity whereas string
comparisons don't. Since we're wiping everything anyway, it's
easier to just use UTC everywhere rather than migrate to
datetime calls.
* add utcdb to check that arguments are utc
* storagenodedb: add trivial tests to ensure calls work
This at least tests that all of the timestamps passed in are
in the UTC timezone.
* fix truncated comment and change migrations to be UTC
If we verify that the size matches reality, we can then expect to use
the filesystem to store the piece size as used in the signed PieceHash
from the uplink. Otherwise, the uplink might send a garbage size value,
leaving the storagenode with no good way to verify the uplink signature
on the piece at a later date.
Also fix the code in uplink/piecestore/ so that it sends a valid size,
because it was being rude and sending 0.
- Drops some unused indexes
- Applies a computed index to timestamp columns
- Applies a partial index for expired pieces
- Uses BETWEEN to avoid some datetime calls
- Filters expired piece search by those that aren't NULL
* Add in-memory cache for bandwidth and space usage monitoring
* moved some structs around and added error handling for get piece size query
* added to existing bandwidth test. fixed typo
* added test, updates from PR review, added monkit for new methods
* PR review updates. renamed space used methods
* changed bw cache so that only Add updates the cache and it only overwrites when the date moves forward
* moved bandwidth usage to bw and space usage to pieceinfodb
* fixed interface comment
* removed pointer from sync.Once
* remove infodb locks and give a unique name for each in memory created.
* changed max idle and open to 1 for memory DBs. fixes table locking errors
* fixed race condition
* added file based infodb test
* added busy timeout parameter to the file based infodb for testing
* fixed imports
* removed db.locked() after merge from master
