Commit Graph

346 Commits

Author SHA1 Message Date
Cameron Ayer
a4f9865b47 satellite: adds and enables cockroachdb compatibility for tests
Change-Id: I85a3ad8c3b9d7e15ea8675b6c55af0002933db57
2019-12-16 22:29:25 +00:00
paul cannon
2f7465c294 private/dbutil: register "cockroach" as sql.DB driver
this will allow us to inspect the type of `db.Driver()` on *sql.DB
connections to correctly differentiate between pg and crdb conns.

as a bonus, this moves all concerns about when to replace "cockroach://"
with "postgres://" out of view, letting the thin shim "driver" take care
of that.

Change-Id: Ib24103ab7c508231e681f89a7321b623e4e125e9
2019-12-16 19:10:00 +00:00
paul cannon
b5ddfc6fa5 satellite/satellitedb: unexport satellitedb.DB
Backstory: I needed a better way to pass around information about the
underlying driver and implementation to all the various db-using things
in satellitedb (at least until some new "cockroach driver" support makes
it to DBX). After hitting a few dead ends, I decided I wanted to have a
type that could act like a *dbx.DB but which would also carry
information about the implementation, etc. Then I could pass around that
type to all the things in satellitedb that previously wanted *dbx.DB.

But then I realized that *satellitedb.DB was, essentially, exactly that
already.

One thing that might have kept *satellitedb.DB from being directly
usable was that embedding a *dbx.DB inside it would make a lot of dbx
methods publicly available on a *satellitedb.DB instance that previously
were nicely encapsulated and hidden. But after a quick look, I realized
that _nothing_ outside of satellite/satellitedb even needs to use
satellitedb.DB at all. It didn't even need to be exported, except for
some trivially-replaceable code in migrate_postgres_test.go. And once
I made it unexported, any concerns about exposing new methods on it were
entirely moot.

So I have here changed the exported *satellitedb.DB type into the
unexported *satellitedb.satelliteDB type, and I have changed all the
places here that wanted raw dbx.DB handles to use this new type instead.
Now they can just take a gander at the implementation member on it and
know all they need to know about the underlying database.

This will make it possible for some other pending code here to
differentiate between postgres and cockroach backends.

Change-Id: I27af99f8ae23b50782333da5277b553b34634edc
2019-12-16 19:09:30 +00:00
Jessica Grebenschikov
c5116cb2a0 satellitedb: fix migration cockroach test
Change-Id: Ie3b4a4b0795d156238d50a58078282cc0918a334
2019-12-16 18:02:31 +00:00
Vitalii Shpital
53d9bc4530
storagenode/notifications: db created (#3707) 2019-12-16 19:59:01 +02:00
Yaroslav Vorobiov
8cf1aa6e4f
satellite/accounting: fix project limits migration (#3717) 2019-12-10 18:12:49 +02:00
Jeff Wendling
48da8baab5 storj-sim: work with cockroach:// urls for satellite databases
for storj-sim to work, we need to avoid schemas in cockroach urls
so we have storj-sim create namespaced databases instead of schemas
and we have the migrate command create the database in the same way
that it would create a schema for postgres. then it works!

a follow up commit will move the creation of the database/schemas
into storj-sim's setup step so that we can avoid doing these icky
creations during normal migration calls. it will also make the
pointerdb have an explicit call to migrate instead of just doing
it every time it's opened.

Change-Id: If69ef5cb96b6866b0438c761bd445afb3597ae5f
2019-12-09 23:44:00 +00:00
Natalie Villasana
c3c02bec3c
satellite/satellitedb: reset storage node reputations to re-enable disqualification (#3693) 2019-12-09 12:04:00 -05:00
Egon Elbre
56a3b62bef satellite/satellitedb: ensure migration tests run (#3706)
satellitedb migration tests ran against multiple base versions, however after the merging all the steps the base versions didn't exists anymore - which meant none of the migration tests were actually running.
2019-12-09 09:26:58 -06:00
Vitalii Shpital
fa5288c254
satellitedb: bucket search fixed (#3594) 2019-12-09 14:46:30 +02:00
Jennifer Johnson
8e9532dbd9 satellitedb/repairqueue: use errs.New() instead of fmt.Errorf() to retain stack trace
Change-Id: I47f1985aaeace556e3e0f7a20d2718410936db17
2019-12-05 15:52:25 +00:00
paul cannon
378b863b2b private,satellite: unite all the "temp db schema" things
first, so that they all work the same way, because it's getting
complicated, and second, so that we can do the appropriate thing
instead of CREATE SCHEMA for cockroachdb.

Change-Id: I27fbaeeb6223a3e06d97bcf692a2d014b31465f7
2019-12-05 15:36:59 +00:00
Cameron
28a66747c9 satellite/satellitedb: cockroachDB compatible transaction in offersDB.Create (#3686)
* use crdb wrapper for create/update offer transaction, ensure times returned are UTC

* round to nearest microsecond to compare times
2019-12-04 21:54:43 -06:00
Jennifer Johnson
7c5f777a4f satellitedb/repairqueue: runs a different implementation of the query within Select() for postgres vs cockroach
Change-Id: Ie34dbdb9d870d7d9f8f269702b6b3bad0c55b98e
2019-12-04 21:32:02 +00:00
Jennifer Johnson
e9eff54473 satellitedb/attribution: updates valueAttrQuery to work for both postgres and cockroach
Change-Id: I55d6c21950a66bf782596e9152622efd700c324a
2019-12-04 18:42:02 +00:00
Jennifer Johnson
ecb960f506 private/dbutil: distinguishes between db drivers and implementations to allow for different implementations of SQL queries.
Change-Id: I2dc8d1d371139aa8bc805e92a2b80b71f580fd64
2019-12-04 18:31:26 +00:00
Cameron
c00f688930 format db name in query to match actual db name (#3688) 2019-12-04 08:55:10 -05:00
paul cannon
4d2881b711 satellite/satellitedb: return latest coupon time as UTC
it doesn't necessarily _have_ to be UTC; the time is correct as returned
either way, but this will make it a little less prone to variance.

also, there is a test that depends on the time being returned in UTC.

Change-Id: Ia71e24ecd9973ba70a1cfb5621a3030a5c82d004
2019-12-03 17:36:41 +00:00
Jess G
e0b9b5b317
satellitedb: fixes for cockroachdb compatibility (#3682)
* crdb compat fixes

* update sn accounting query
2019-12-03 08:24:46 -08:00
paul cannon
b1fa7cdfbb
satellite/satellitedb/satellitedbtest: flag to run cockroach tests (#3674)
This will make it so we don't need to comment out those lines every time
we want to enable the cockroachdb tests during development.

Once it's ready this flag can go away.
2019-12-02 14:06:21 -06:00
Yehor Butko
ae05fa0874
satellitedb/accounting: fixed query (#3672) 2019-11-29 17:53:57 +02:00
Yehor Butko
0f523e82a6
missing line returned (#3670) 2019-11-28 23:42:04 +02:00
Yehor Butko
b10eee6e7f
satellitedb/accounting: project usage optimization (#3668) 2019-11-28 20:45:31 +02:00
Yehor Butko
756b9b9e2b
satellite/payments: coupons and coupon usage (#3648) 2019-11-26 19:58:51 +02:00
Jess G
854e5507ab
crdb uses namespaced db for each test (#3646)
* crdb uses namespaced db for each test

* add test for me test

* fix lint and tests

* updates per cr comments

* rm all replaceall
2019-11-26 08:39:57 -08:00
Cameron
7abad3c6bb
refactor sql to be compatible with pq and cockroach (#3647) 2019-11-26 09:26:22 -05:00
Yaroslav Vorobiov
8a002e8c8e
satellite/accounting: separate project limit from project entity (#3632) 2019-11-25 16:18:04 +02:00
Jess G
388f33b84d
satellitedb: add support to testplanet for cockroachdb (#3634)
* update migration steps, add crdb support to testplanet

* add crdb support

* have jenkins run a bares bones crdb compat test

* skip crdb tests

* skip crdb tests

* fix root_piece_id column

* write crdb store to tmp dir

* escape
2019-11-22 11:59:46 -08:00
Matt Robinson
9af97d366a Make sed a little more cross platformable (#3629) 2019-11-22 11:17:02 -07:00
Nikolai Siedov
6a4389d3e1 satellite/console: apiKeys case-insensitive search added (#3621) 2019-11-21 00:56:55 +02:00
Yingrong Zhao
b995406ff9
satellite/satellitedb: separate uuid creation from db layer (#3600) 2019-11-20 14:16:27 -05:00
Jess G
e9c3194c82
satellitedb: merge migration into one step (#3551)
* merge migration

* rm migration versions

* rm unneeded migration test data

* create index w/postgres + crdb compatible syntax

* add default to offers.invitee_credit_duration_days

* changes so that schema matches from master to branch

* change to be crdb compatible

* add check to confirm db version

* mv version check to migration

* update tests

* add minversion to sadb migration, update tests

* confirm min version for all dbs in a migration

* add validate migration to sadb

* fix lint err

* rm min version check from migrate

* change sadb check

* hard code min db version

* fix comment
2019-11-19 12:52:57 -08:00
littleskunk
8b3444e088
satellite/nodeselection: don't select nodes that haven't checked in for a while (#3567)
* satellite/nodeselection: dont select nodes that havent checked in for a while

* change testplanet online window to one minute

* remove satellite reconfigure online window = 0 in repair tests

* pass timestamp into UpdateCheckIn

* change timestamp to timestamptz

* edit tests to set last_contact_success to 4 hours ago

* fix syntax error

* remove check for last_contact_success > last_contact_failure in IsOnline
2019-11-15 23:43:06 +01:00
Yaroslav Vorobiov
53c6741ba6
satellite/payments: add API for retrieving conversion ratio, convert tokens to USD before applying to balance (#3530) 2019-11-15 16:59:39 +02:00
Yehor Butko
a8e4e9cb03
satellite/payments: project usage charges (#3512) 2019-11-15 16:27:44 +02:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Egon Elbre
1e64006e32 lint: add staticcheck as a separate step (#3569) 2019-11-14 10:31:30 +02:00
Natalie Villasana
1a9757a7f2 satellite/gracefulexit: add count for order limits sent from satellite to exiting node (#3544) 2019-11-13 09:54:50 -05:00
Yaroslav Vorobiov
36311a3a05
satellite/console: add token deposit API, populate billing history with transactions (#3500) 2019-11-12 13:14:34 +02:00
Egon Elbre
4b85d3d739
internal/testplanet: better error message when postgres is not defined (#3539) 2019-11-11 17:05:21 +02:00
Yingrong Zhao
6331f839ae
satellite/gracefulexit: not allow disqualified node to graceful exit (#3493) 2019-11-07 12:19:34 -05:00
Ethan Adams
f3dccb56b1
satellite/gracefulexit: Check if pointer has been overwritten or deleted before sending transfer message. (#3481) 2019-11-07 11:13:05 -05:00
Natalie Villasana
68a7790069 satellite/gracefulexit: select new node filtered by Distinct IP (#3435) 2019-11-06 16:38:51 -05:00
Egon Elbre
23c556ae15
satellite/rewards: fixes from review comments (#3495) 2019-11-06 20:37:53 +02:00
Yaroslav Vorobiov
35edc2bcc3 satellite/payments: invoice creation (#3468) 2019-11-05 15:16:02 +02:00
Yehor Butko
0c2e498f09
satellite/satellitedb: console tables archview comments updated (#3465) 2019-11-04 16:37:39 +02:00
Yehor Butko
761cec5ea3
satellite/payments: archview comments updated (#3464) 2019-11-04 14:30:07 +02:00
Jess G
8d92c288e2
satellitedb: separate migration into subcommand (#3436)
* separate sadb migration, add version check

* update checkversion to do same validation as migration

* changes per CR

* add sa migration to storj-sim

* add different debug port in storj-sim for migration

* add wait for exit for storj-sim migration

* update sa docker entrypoint to support migration

* storj-sim satellite parts all wait for migration

* upgrade golang-migrate/migrate to v4 because bug

* fix go mod tidy
2019-11-02 13:09:07 -07:00
Egon Elbre
aa761700af satellite/satellitedb: update nodes in sorted order (#3446) 2019-11-01 18:07:23 +01:00
Maximillian von Briesen
54594e79c3 satellite/gracefulexit: add metrics on satellite for graceful exit (#3355) 2019-10-29 16:22:20 -04:00