Commit Graph

24 Commits

Author SHA1 Message Date
Yingrong Zhao
9f2f1527c5
storagenode/storagenodedb: add new tables for graceful exit (#3008)
* add database schema

* add migration

* change table name and update blueprint
2019-09-11 18:57:53 -04:00
Isaac Hess
7718802f0c
storagenode/storagenodedb: prepare for multiple databases (#3005)
* Migrate test: prepare for multiple databases

* Add copyright

* Fix unused variables

* Move data to testdata, split MultiDBSnapshot from MultiDBState
2019-09-11 14:31:46 -06:00
Yaroslav Vorobiov
f7403f97b0
storagenode/storageusage: add summary, rename timestamp to interval_start (#2911) 2019-09-04 17:13:43 +03:00
Yaroslav Vorobiov
b4d7d6778f
storagenode/reputation: add disqualified flag (#2862) 2019-08-28 23:54:12 +03:00
Cameron
1f3537d4a9 storagenode/vouchers: remove storagenode vouchers (#2873) 2019-08-26 19:35:19 +03:00
littleskunk
6615350188 initialize used space table with sum over pieceinfo (#2818) 2019-08-20 08:13:18 -04:00
Jess G
022f5d2e14
storagenode: add space used cache for pieces (#2753)
* add cache, update cache w/piece create/delete

* add service w/loop to cache to recalculate space used cache

* add piecestore cache to other sn svcs to use

* add table to persist the total space used

* rm cache where not needed

* rm stuff from sn svcs

* start fixing tests, changes per comments

* update commits

* add unit tests

* fix commiting before we write header bytes

* fix cache create test

* copy cache map, add started back to recalc

* fix test

* add test, update comments
2019-08-12 14:43:05 -07:00
Yaroslav Vorobiov
28a7778e9e
storagenode/nodestats: cache node stats (#2543) 2019-08-08 16:47:04 +03:00
paul cannon
17bdb5e9e5
move piece info into files (#2629)
Deprecate the pieceinfo database, and start storing piece info as a header to
piece files. Institute a "storage format version" concept allowing us to handle
pieces stored under multiple different types of storage. Add a piece_expirations
table which will still be used to track expiration times, so we can query it, but
which should be much smaller than the pieceinfo database would be for the
same number of pieces. (Only pieces with expiration times need to be stored in piece_expirations, and we don't need to store large byte blobs like the serialized
order limit, etc.) Use specialized names for accessing any functionality related
only to dealing with V0 pieces (e.g., `store.V0PieceInfo()`). Move SpaceUsed-
type functionality under the purview of the piece store. Add some generic
interfaces for traversing all blobs or all pieces. Add lots of tests.
2019-08-07 20:47:30 -05:00
Stefan Benten
38a40088c7 Remove orphaned tmp data from Storagenodes (#2582) 2019-07-17 16:00:37 +03:00
Stefan Benten
de300e9235 Network Wipe (Pre Beta) (#2566) 2019-07-16 18:31:29 +02:00
ethanadams
d044613679
SN DB Optimization: Add rollups to bandwidth usage (#2541)
* V3-2119: Add storagenode bandwidth usage rollup
2019-07-16 10:58:58 -04:00
Jeff Wendling
b9d8ddaad1
storagenode: remove datetime calls in favor of UTC (#2557)
* storagenode: remove datetime calls in favor of UTC

datetime only has second level granularity whereas string
comparisons don't. Since we're wiping everything anyway, it's
easier to just use UTC everywhere rather than migrate to
datetime calls.

* add utcdb to check that arguments are utc

* storagenodedb: add trivial tests to ensure calls work

This at least tests that all of the timestamps passed in are
in the UTC timezone.

* fix truncated comment and change migrations to be UTC
2019-07-15 13:38:08 -04:00
Jeff Wendling
a2418b22af storagenodedb: optimize index usage and queries (#2545)
- Drops some unused indexes
- Applies a computed index to timestamp columns
- Applies a partial index for expired pieces
- Uses BETWEEN to avoid some datetime calls
- Filters expired piece search by those that aren't NULL
2019-07-12 15:29:09 -04:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Jeff Wendling
02565db73a storagenode: migration to drop unused index and used_serials data (#2508) 2019-07-10 15:16:13 -04:00
Michal Niewrzal
bbc25a2bf7 Drop SN certifiates table from DB (#2498) 2019-07-09 17:33:45 -04:00
Fadila
3f4662598e
storagenode/piecestore: add piece_creation field (#2441) 2019-07-08 09:22:36 +02:00
Stefan Benten
1f58708910 Delete all Tardigrade Satellite Data from SNO's (#2324) 2019-06-25 13:10:56 +02:00
littleskunk
b1e5cf1200
add index on pieceinfo expireation for faster GetExpired calls (#2220)
* add index on pieceinfo expireation for faster GetExpired calls

* Add Migration File
2019-06-18 01:27:14 +02:00
Cameron
23587bba0c
Storagenode vouchers table (#2121)
* add vouchers table with methods
2019-06-07 16:20:34 -04:00
Egon Elbre
a2b61fd67c
storage node collector (#1913) 2019-05-08 14:11:59 +03:00
Stefan Benten
ac452a5819
Add Network Wipe for Storagenodes (#1909)
* Add Network Wipe Migration to InfoDB

* Remove New Data Section
2019-05-07 22:05:50 +02:00
Egon Elbre
fba9a5f945 migration tests for storagenodedb infodb (#1628) 2019-04-02 09:54:09 +02:00