Commit Graph

54 Commits

Author SHA1 Message Date
Moby von Briesen
39260b6fdc storagenode/storagenodedb: Add audit_history to storagenode reputationdb
This change will give us the ability to make audit windows visible via
the storagenode api.

Change-Id: Ifb3d044ce9b456c16f8ea3b3b162ba894426477a
2020-12-31 11:00:42 -05:00
Qweder93
9df74338a8 storagenode: secret db and service added
Change-Id: I91257e5adc4fc6711653f30c118e476ed1c95b6b
2020-10-16 13:24:33 +00:00
Qweder93
36d752e92d storagenode/reputation: offline_under_review_at added
Change-Id: Ia7ec79b2d6f20fe29de0c36223f9485380d2845c
2020-09-02 18:48:28 +03:00
Qweder93
7d9897b7af storagenode/nodestats: online_score added
Change-Id: I84b50a6cace306e5f10d53a2073fe8810d4d2960
2020-09-02 17:45:01 +03:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
Qweder93
9b90712aa0 storagenode/heldamount: payents added to db
Change-Id: Ib6c486251ca08d34003c35379d10314127edf103
2020-06-30 17:24:35 +03:00
Moby von Briesen
dc57640d9c storagenode/piecestore: switch usedserials db for in-memory usedserials store
Part 2 of moving usedserials in memory
* Drop usedserials table in storagenodedb
* Use in-memory usedserials store in place of db for order limit
verification
* Update order limit grace period to be only one hour - this means
uplinks must send their order limits to storagenodes within an hour of
receiving them

Change-Id: I37a0e1d2ca6cb80854a3ef495af2d1d1f92e9f03
2020-05-28 12:52:52 -04:00
crawter
f5ac678b0a storagenode/satellitesdb: added FK constraint to satelliteID
Change-Id: If5adf2b92627fcf80850670ba672b346320ddd87
2020-05-21 13:01:20 +00:00
Qweder93
49ad90dcd8 storagenode/reputation: unknown_score (unknown_alpha / unknown_alpha+unknown_beta) added to reputation stats, https://storjlabs.atlassian.net/browse/SG-326
Change-Id: I0b29ad736f7a11c7e57a846b6891f4b40755aa48
2020-05-20 11:25:14 +00:00
Jeff Wendling
efee3563a8 storagenode/storagenodedb: fix joined_at migration
Change-Id: I9091741bbec6e7a2fc6addc23249a40176ec040a
2020-05-05 15:01:07 -06:00
Qweder93
b754806b23 storagenode/reputation: unknown_audit_reputation_alpha and beta added to db, and reputation endpoint
Change-Id: I5d92268851bb9a202cc5d6b4d403467e6f692726
2020-05-05 15:48:04 +00:00
Jeff Wendling
4ad01e8170 storagenode/storagenodedb: backfill reputation.joined_at
it was being used in ways that implied it should be NOT NULL
even though it was possibly null. we used to get this data
from the satellite db's added_at column as seen in 30369b02,
so backfill it using that data where joined_at is NULL, and
then alter the table to constrain the column to be NOT NULL.

Fixes #3866.

Change-Id: If2d856189209740d985f71dada7b93525e625ef3
2020-05-01 17:59:53 +00:00
Qweder93
805e328c47 storagenode/heldamount payments removed
Change-Id: I87cc04f43d182a4190a571ef417be85d02db9d34
2020-04-21 17:15:31 +00:00
Qweder93
30369b027c storagenode/storagenodedb/reputation: add joined_at
Change-Id: Ic471fac97bf54b537f2c34f24b4069b0641c746d
2020-04-17 12:12:09 +00:00
Qweder93
743b3fb226 storagenode/nodestats: add pricing model, storagenode/cache: add paystub history storing
Change-Id: I9bc104a1407c8f286a964c796656d89b122bf752
2020-04-14 19:04:00 +03:00
Moby von Briesen
14b3704f56 storagenode: add suspended status to storagenode dashboard/api
* Add migration to storagenode reputation table to add suspended
timestamp
* Send suspended info to storagenode from satellite nodestats endpoint
* Add suspended status to storagenode api
* Add an indicator on the storagenode dashboard informing operator of
the satellites the node is suspended on

Change-Id: Ie3669f6069cc0258ba76ec99d17006e1b5fd9c8a
2020-04-09 13:36:23 +00:00
crawter
45507d285b storagenode/storagenodedb: heldamount tests added
Change-Id: I862b0b38349a11254426855ffafb1f2f0845cb4c
2020-03-15 14:55:11 +00:00
Qweder93
5ccce04338 storagenode/storagenodedb: heldamount added
Change-Id: I213e3abffd7356bbfccb3f33bcbafa558674b8d9
2020-03-13 16:23:59 +00:00
Isaac Hess
4dafd03f11 storagenode: Prevent negative values in piece_space_used, migrate negatives to 0
Change-Id: Ibd663db087058c928190aa52c520f22e9338dd04
2020-01-30 13:03:18 -05:00
Jeff Wendling
21b65ca3b0 storagenode/storagenodedb: migrate to set total to content_size
Change-Id: I4906c2fe9cdb3a32c045c98039d4bde6b8b809e3
2020-01-30 08:53:12 -07:00
Isaac Hess
14fd6a9ef0 storagenode/pieces: Track total piece size
This change updates the storagenode piecestore apis to expose access to
the full piece size stored on disk. Previously we only had access to
(and only kept a cache of) the content size used for all pieces. This
was inaccurate when reporting the amount of disk space used by nodes.

We now have access to the total content size, as well as the total disk
usage, of all pieces. The pieces cache also keeps a cache of the total
piece size along with the content size.

Change-Id: I4fffe7e1257e04c46021a2e37c5adc6fe69bee55
2020-01-23 11:00:24 -07:00
Egon Elbre
c7b846589e private/dbutil/sqliteutil: add ctx argument
Change-Id: If1caa9cde746817e62cae32a152eeec81959129c
2020-01-13 15:03:30 +02:00
Vitalii Shpital
53d9bc4530
storagenode/notifications: db created (#3707) 2019-12-16 19:59:01 +02:00
littleskunk
c2ea75208f
storagenode/orderdb: fix db lock
Change-Id: Id1add0ba7ae1b20bd98099bd4d3aff0fcfdd90c9
2019-12-15 23:41:22 +01:00
Isaac Hess
6aeddf2f53
storagenode/pieces: Add Trash and RestoreTrash to piecestore (#3575)
* storagenode/pieces: Add Trash and RestoreTrash to piecestore

* Add index for expiration trash
2019-11-20 09:28:49 -07:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Bill Thorp
89c59d06f9
storagenode/storagenodedb: add SQL receiver logic for graceful exit (#3067)
* added graceful exit db methods
2019-10-01 10:34:03 -04:00
Isaac Hess
2c5e169888
storagenode/storagenodedb: Vacuum info.db to prepare for splitting storagenodedbs (#3134) 2019-09-27 07:55:51 -06:00
Isaac Hess
580e511b4c
storagenode/storagenodedb: Migrate to separate dbs (#3081)
* storagenode/storagenodedb: Migrate to separate dbs

* storagenode/storagenodedb: Add migration to drop versions tables

* Put drop table statements into a transaction.

* Fix CI errors.

* Fix CI errors.

* Changes requested from PR feedback.

* storagenode/storagenodedb: fix tx commit
2019-09-23 12:36:46 -07:00
Simon Guindon
a2b1e9fa95
storagenode/storagenodedb: refactor both data access objects and migrations to support multiple DB connections (#3057)
* Split the info.db database into multiple DBs using Backup API.

* Remove location. Prev refactor assumed we would need this but don't.

* Added VACUUM to reclaim space after splitting storage node databases.

* Added unique names to SQLite3 connection hooks to fix testplanet.

* Moving DB closing to the migration step.

* Removing the closing of the versions DB. It's already getting closed.

* Swapping the database connection references on reconnect.

* Moved sqlite closing logic away from the boltdb closing logic.

* Moved sqlite closing logic away from the boltdb closing logic.

* Remove certificate and vouchers from DB split migration.

* Removed vouchers and bumped up the migration version.

* Use same constructor in tests for storage node databases.

* Use same constructor in tests for storage node databases.

* Adding method to access underlining SQL database connections and cleanup

* Adding logging for migration diagnostics.

* Moved migration closing database logic to minimize disk usage.

* Cleaning up error handling.

* Fix missing copyright.

* Fix linting error.

* Add test for migration 21 (#3012)

* Refactoring migration code into a nicer to use object.

* Refactoring migration code into a nicer to use object.

* Fixing broken migration test.

* Removed unnecessary code that is no longer needed now that we close DBs.

* Removed unnecessary code that is no longer needed now that we close DBs.

* Fixed bug where an invalid database path was being opened.

* Fixed linting errors.

* Renamed VersionsDB to LegacyInfoDB and refactored DB lookup keys.

* Renamed VersionsDB to LegacyInfoDB and refactored DB lookup keys.

* Fix migration test. NOTE: This change does not address new tables satellites and satellite_exit_progress

* Removing v22 migration to move into it's own PR.

* Removing v22 migration to move into it's own PR.

* Refactored schema, rebind and configure functions to be re-useable.

* Renamed LegacyInfoDB to DeprecatedInfoDB.

* Cleaned up closeDatabase function.

* Renamed storageNodeSQLDB to migratableDB.

* Switched from using errs.Combine() to errs.Group in closeDatabases func.

* Removed constructors from storage node data access objects.

* Reformatted usage of const.

* Fixed broken test snapshots.

* Fixed linting error.
2019-09-18 12:17:28 -04:00
Yingrong Zhao
9f2f1527c5
storagenode/storagenodedb: add new tables for graceful exit (#3008)
* add database schema

* add migration

* change table name and update blueprint
2019-09-11 18:57:53 -04:00
Isaac Hess
7718802f0c
storagenode/storagenodedb: prepare for multiple databases (#3005)
* Migrate test: prepare for multiple databases

* Add copyright

* Fix unused variables

* Move data to testdata, split MultiDBSnapshot from MultiDBState
2019-09-11 14:31:46 -06:00
Yaroslav Vorobiov
f7403f97b0
storagenode/storageusage: add summary, rename timestamp to interval_start (#2911) 2019-09-04 17:13:43 +03:00
Yaroslav Vorobiov
b4d7d6778f
storagenode/reputation: add disqualified flag (#2862) 2019-08-28 23:54:12 +03:00
Cameron
1f3537d4a9 storagenode/vouchers: remove storagenode vouchers (#2873) 2019-08-26 19:35:19 +03:00
littleskunk
6615350188 initialize used space table with sum over pieceinfo (#2818) 2019-08-20 08:13:18 -04:00
Jess G
022f5d2e14
storagenode: add space used cache for pieces (#2753)
* add cache, update cache w/piece create/delete

* add service w/loop to cache to recalculate space used cache

* add piecestore cache to other sn svcs to use

* add table to persist the total space used

* rm cache where not needed

* rm stuff from sn svcs

* start fixing tests, changes per comments

* update commits

* add unit tests

* fix commiting before we write header bytes

* fix cache create test

* copy cache map, add started back to recalc

* fix test

* add test, update comments
2019-08-12 14:43:05 -07:00
Yaroslav Vorobiov
28a7778e9e
storagenode/nodestats: cache node stats (#2543) 2019-08-08 16:47:04 +03:00
paul cannon
17bdb5e9e5
move piece info into files (#2629)
Deprecate the pieceinfo database, and start storing piece info as a header to
piece files. Institute a "storage format version" concept allowing us to handle
pieces stored under multiple different types of storage. Add a piece_expirations
table which will still be used to track expiration times, so we can query it, but
which should be much smaller than the pieceinfo database would be for the
same number of pieces. (Only pieces with expiration times need to be stored in piece_expirations, and we don't need to store large byte blobs like the serialized
order limit, etc.) Use specialized names for accessing any functionality related
only to dealing with V0 pieces (e.g., `store.V0PieceInfo()`). Move SpaceUsed-
type functionality under the purview of the piece store. Add some generic
interfaces for traversing all blobs or all pieces. Add lots of tests.
2019-08-07 20:47:30 -05:00
Stefan Benten
38a40088c7 Remove orphaned tmp data from Storagenodes (#2582) 2019-07-17 16:00:37 +03:00
Stefan Benten
de300e9235 Network Wipe (Pre Beta) (#2566) 2019-07-16 18:31:29 +02:00
ethanadams
d044613679
SN DB Optimization: Add rollups to bandwidth usage (#2541)
* V3-2119: Add storagenode bandwidth usage rollup
2019-07-16 10:58:58 -04:00
Jeff Wendling
b9d8ddaad1
storagenode: remove datetime calls in favor of UTC (#2557)
* storagenode: remove datetime calls in favor of UTC

datetime only has second level granularity whereas string
comparisons don't. Since we're wiping everything anyway, it's
easier to just use UTC everywhere rather than migrate to
datetime calls.

* add utcdb to check that arguments are utc

* storagenodedb: add trivial tests to ensure calls work

This at least tests that all of the timestamps passed in are
in the UTC timezone.

* fix truncated comment and change migrations to be UTC
2019-07-15 13:38:08 -04:00
Jeff Wendling
a2418b22af storagenodedb: optimize index usage and queries (#2545)
- Drops some unused indexes
- Applies a computed index to timestamp columns
- Applies a partial index for expired pieces
- Uses BETWEEN to avoid some datetime calls
- Filters expired piece search by those that aren't NULL
2019-07-12 15:29:09 -04:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Jeff Wendling
02565db73a storagenode: migration to drop unused index and used_serials data (#2508) 2019-07-10 15:16:13 -04:00
Michal Niewrzal
bbc25a2bf7 Drop SN certifiates table from DB (#2498) 2019-07-09 17:33:45 -04:00
Fadila
3f4662598e
storagenode/piecestore: add piece_creation field (#2441) 2019-07-08 09:22:36 +02:00
Stefan Benten
1f58708910 Delete all Tardigrade Satellite Data from SNO's (#2324) 2019-06-25 13:10:56 +02:00
littleskunk
b1e5cf1200
add index on pieceinfo expireation for faster GetExpired calls (#2220)
* add index on pieceinfo expireation for faster GetExpired calls

* Add Migration File
2019-06-18 01:27:14 +02:00