Commit Graph

21 Commits

Author SHA1 Message Date
Egon Elbre
0835b9024c private/dbutil/pgutil: add ctx argument
Change-Id: Icfd56ca8c1f831ad56c0195a0b883e8f0618daaf
2020-01-13 15:27:06 +02:00
Egon Elbre
c7b846589e private/dbutil/sqliteutil: add ctx argument
Change-Id: If1caa9cde746817e62cae32a152eeec81959129c
2020-01-13 15:03:30 +02:00
Egon Elbre
8d8d57c3b5 mod: update sqlite module to v2.0.2
This updates SQLite amalgamation from 3.29.0 to 3.30.1.

The module contains fixes for races.

Change-Id: Ic6a06a43ba404de0091d8a2f7444a8f4b1d5d54c
2020-01-08 21:21:15 +02:00
Egon Elbre
082ec81714
uplink: move to storj.io/uplink (#3746) 2020-01-08 15:40:19 +02:00
paul cannon
0c88a7b475 private/migrate: use transactional helpers and not Begin()
This code needs to work against cockroachDB, so transactions must be retried
when a retryable error is returned. This change puts migrate
transactions into the dbutil.WithTx transactional helpers to achieve
this in the easiest way.

Change-Id: Ib930e82d55cb0257357a222ce9131e6e53372c03
2020-01-07 18:25:38 +00:00
paul cannon
6231842422 private/dbutil: add WithTx transaction helpers
These helpers will work similar to the WithTx method we have added to
our dbx.DB instances, but it will use crdb.ExecuteTx or crdb.ExecuteInTx
when the backend is CockroachDB, so that transactions are retried
correctly.

Anything that uses transactions and might need to work against
CockroachDB needs to handle "RetriableError" from cockroachdb by
restarting the transaction. This will probably be a large pain if not
using these helpers or something very like them.

Subsequent changes will undertake transforming all db-transaction uses
in satellite code so that they are cockroach-safe.

Change-Id: I648b8de2168612c67b9d6eb8402bccf8286249a9
2020-01-06 20:06:45 +00:00
paul cannon
a33734bee7 satellite/satellitedb/dbx: add cockroach driver type
Change-Id: I7a0da6e066c67a521fc1b23b085ab8554eee0d4c
2020-01-06 18:01:03 +00:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Egon Elbre
acb4435a67 satellite/satellitedb: improve Cockroach migrate test
Load schemas in parallel instead of one-by-one.

Optimizes from 2m30s to 1m15s.

Change-Id: I0bf6381a0ae99b44271fe55d4ee658683064c097
2019-12-21 10:58:43 +00:00
Cameron Ayer
a4f9865b47 satellite: adds and enables cockroachdb compatibility for tests
Change-Id: I85a3ad8c3b9d7e15ea8675b6c55af0002933db57
2019-12-16 22:29:25 +00:00
paul cannon
2f7465c294 private/dbutil: register "cockroach" as sql.DB driver
this will allow us to inspect the type of `db.Driver()` on *sql.DB
connections to correctly differentiate between pg and crdb conns.

as a bonus, this moves all concerns about when to replace "cockroach://"
with "postgres://" out of view, letting the thin shim "driver" take care
of that.

Change-Id: Ib24103ab7c508231e681f89a7321b623e4e125e9
2019-12-16 19:10:00 +00:00
Vitalii Shpital
53d9bc4530
storagenode/notifications: db created (#3707) 2019-12-16 19:59:01 +02:00
Jeff Wendling
fb8e78132d storagenodedb: reenable utccheck in tests
Change-Id: If7d64dd4ae58e4b656ff9122ae3195b2a5173cb3
2019-12-10 23:17:14 +00:00
Jessica Grebenschikov
d8a8f92e30 private/dbutil/cockroachutil: keep crdb connstr for tests
Change-Id: Icad19d6b0093e7bf0fff709330164bfcbd733911
2019-12-10 17:24:35 +00:00
Jeff Wendling
48da8baab5 storj-sim: work with cockroach:// urls for satellite databases
for storj-sim to work, we need to avoid schemas in cockroach urls
so we have storj-sim create namespaced databases instead of schemas
and we have the migrate command create the database in the same way
that it would create a schema for postgres. then it works!

a follow up commit will move the creation of the database/schemas
into storj-sim's setup step so that we can avoid doing these icky
creations during normal migration calls. it will also make the
pointerdb have an explicit call to migrate instead of just doing
it every time it's opened.

Change-Id: If69ef5cb96b6866b0438c761bd445afb3597ae5f
2019-12-09 23:44:00 +00:00
Jeff Wendling
1df7b360d7 satellite/metainfo: Use cockroachdb client for metainfo db
Change-Id: I3cf7a00de4f654eacaffbb494f4841c64a2d9ce6
2019-12-05 10:33:54 -07:00
paul cannon
378b863b2b private,satellite: unite all the "temp db schema" things
first, so that they all work the same way, because it's getting
complicated, and second, so that we can do the appropriate thing
instead of CREATE SCHEMA for cockroachdb.

Change-Id: I27fbaeeb6223a3e06d97bcf692a2d014b31465f7
2019-12-05 15:36:59 +00:00
paul cannon
850c358087 private/dbutil/pgutil: make QuerySchema work on crdb
Adjust the pg_constraint query so that it works without a LATERAL JOIN,
since CockroachDB doesn't like that.

This isn't hooked up to a cockroach test yet, but that's coming.

Change-Id: I0df6b477d958996b673fc121eaa1f7c35e5cc504
2019-12-04 18:55:26 +00:00
Jennifer Johnson
ecb960f506 private/dbutil: distinguishes between db drivers and implementations to allow for different implementations of SQL queries.
Change-Id: I2dc8d1d371139aa8bc805e92a2b80b71f580fd64
2019-12-04 18:31:26 +00:00
Jess G
388f33b84d
satellitedb: add support to testplanet for cockroachdb (#3634)
* update migration steps, add crdb support to testplanet

* add crdb support

* have jenkins run a bares bones crdb compat test

* skip crdb tests

* skip crdb tests

* fix root_piece_id column

* write crdb store to tmp dir

* escape
2019-11-22 11:59:46 -08:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00