it was noticed that if you had a long lived transaction A that
was blocking some other transaction B and A was being aborted
due to retriable errors, then transaction B was never given
priority. this was due to using savepoints to do lightweight
retries.
this behavior was problematic becaue we had some queries blocked
for over 16 hours, so this commit addresses the issue with two
prongs:
1. bound the amount of time we will retry a transaction
2. create new transactions when a retry is needed
the first ensures that we never wait for 16 hours, and the value
chosen is 10 minutes. that should be long enough for an ample
amount of retries for small queries, and huge queries probably
shouldn't be retried, even if possible: it's more preferrable to
find a way to make them smaller.
the second ensures that even in the case of retries, queries that
are blocked on the aborted transaction gain priority to run.
between those two changes, the maximum stall time due to retries
should be bounded to around 10 minutes.
Change-Id: Icf898501ef505a89738820a3fae2580988f9f5f4
We don't do a lot of panicking in our main code, so hopefully this won't
matter much, but we /do/ call panic a lot in our tests (t.Fatal,
require.NoError, etc). And when that happens, we need pending
transactions to be aborted or we can get into a deadlock situation when
something else tries to /Close/ that connection.
Change-Id: Idaf0d543ac95afea34f9b2393d1187f5322e9f0f
This reverts commit 8e242cd012.
Revert because lib/pq has known issues with context cancellation.
These issues need to be resolved before these changes can be merged.
Change-Id: I160af51dbc2d67c5449aafa406a403e5367bb555
this will allow for some nice runtime analysis down the road.
also, this allows for wrapping database handles in a way that
can interact with these contexts
requires https://review.dev.storj.io/c/storj/dbx/+/514
Change-Id: Ib087b7cd73296dd2c1e0331314da34d861f61d2b
These helpers will work similar to the WithTx method we have added to
our dbx.DB instances, but it will use crdb.ExecuteTx or crdb.ExecuteInTx
when the backend is CockroachDB, so that transactions are retried
correctly.
Anything that uses transactions and might need to work against
CockroachDB needs to handle "RetriableError" from cockroachdb by
restarting the transaction. This will probably be a large pain if not
using these helpers or something very like them.
Subsequent changes will undertake transforming all db-transaction uses
in satellite code so that they are cockroach-safe.
Change-Id: I648b8de2168612c67b9d6eb8402bccf8286249a9
