Commit Graph

15 Commits

Author SHA1 Message Date
Jeff Wendling
074649835b satellite/satellitedb: add some docs and improve some snapshots
This attempts to add a README.md to help create consistent migrations
that maximize our test coverage and do not include unnecessary
statements.

It also adds a feature to have an `-- OLD DATA --` section as well
as a `-- NEW DATA --` section so that we can fix mistakes made in
previous snapshots (like a row that was forgotten to be added when a
table was created) without editing them going forward.

Change-Id: I28a786f8ef163cae1de1bb08f61af1e1104b0a88
2020-05-22 21:27:36 +00:00
Jeff Wendling
d20db90cff private/dbutil/txutil: create new transactions for retries
it was noticed that if you had a long lived transaction A that
was blocking some other transaction B and A was being aborted
due to retriable errors, then transaction B was never given
priority. this was due to using savepoints to do lightweight
retries.

this behavior was problematic becaue we had some queries blocked
for over 16 hours, so this commit addresses the issue with two
prongs:

    1. bound the amount of time we will retry a transaction
    2. create new transactions when a retry is needed

the first ensures that we never wait for 16 hours, and the value
chosen is 10 minutes. that should be long enough for an ample
amount of retries for small queries, and huge queries probably
shouldn't be retried, even if possible: it's more preferrable to
find a way to make them smaller.

the second ensures that even in the case of retries, queries that
are blocked on the aborted transaction gain priority to run.

between those two changes, the maximum stall time due to retries
should be bounded to around 10 minutes.

Change-Id: Icf898501ef505a89738820a3fae2580988f9f5f4
2020-02-01 18:34:28 +00:00
Egon Elbre
1279eeae39 private/tagsql,storage: fixes to context cancellation
Replace all the remaining uses of sql.DB with tagsql.DB to
fix issues with context cancellation.

Introduce tagsql.Open which helps to get rid of all tagsql.Wrap-s.
Use tagsql in cockroachkv and postgreskv.

Change-Id: I8946d203341cb85a25976896fc7881e1f704e779
2020-01-20 15:44:39 +02:00
Egon Elbre
ee0293c212 private/dbutil/sqliteutil: add missing err check
Change-Id: Ie18c76d0e6d02a5c55e2d6503437b8a07b47a64e
2020-01-19 19:24:58 +00:00
Egon Elbre
59d06644b9 private/migrate: switch to tagsql
Also added temporary types withRebind and withTagTx,
which will be later removed. Currently they help to avoid
changing the whole codebase at the same time.

Change-Id: I7f07ba8f4709a23a463bfa67464628665a05808f
2020-01-19 14:39:16 +02:00
Egon Elbre
5fd833b108 private/dbutil: remove basic Query
dbschema.Query is used only for testing and sqlite,
so this won't cause us problems in production.

Change-Id: Ib296a7daf161a9d3de23a7dfdc4f505d47ac4a37
2020-01-19 14:39:16 +02:00
Egon Elbre
64fb2d3d2f Revert "dbutil: statically require all databases accesses to use contexts"
This reverts commit 8e242cd012.

Revert because lib/pq has known issues with context cancellation.
These issues need to be resolved before these changes can be merged.

Change-Id: I160af51dbc2d67c5449aafa406a403e5367bb555
2020-01-15 07:28:00 +00:00
JT Olio
8e242cd012 dbutil: statically require all databases accesses to use contexts
this will allow for some nice runtime analysis down the road.
also, this allows for wrapping database handles in a way that
can interact with these contexts

requires https://review.dev.storj.io/c/storj/dbx/+/514

Change-Id: Ib087b7cd73296dd2c1e0331314da34d861f61d2b
2020-01-14 18:20:47 -05:00
Egon Elbre
64f056bee4 private/dbutil/sqlutil: use context in queries
Change-Id: Icb92daa483d13e6d57013f3917571d476126bfd2
2020-01-14 20:27:09 +00:00
Egon Elbre
c7b846589e private/dbutil/sqliteutil: add ctx argument
Change-Id: If1caa9cde746817e62cae32a152eeec81959129c
2020-01-13 15:03:30 +02:00
Egon Elbre
082ec81714
uplink: move to storj.io/uplink (#3746) 2020-01-08 15:40:19 +02:00
paul cannon
0c88a7b475 private/migrate: use transactional helpers and not Begin()
This code needs to work against cockroachDB, so transactions must be retried
when a retryable error is returned. This change puts migrate
transactions into the dbutil.WithTx transactional helpers to achieve
this in the easiest way.

Change-Id: Ib930e82d55cb0257357a222ce9131e6e53372c03
2020-01-07 18:25:38 +00:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Jeff Wendling
fb8e78132d storagenodedb: reenable utccheck in tests
Change-Id: If7d64dd4ae58e4b656ff9122ae3195b2a5173cb3
2019-12-10 23:17:14 +00:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00