Commit Graph

357 Commits

Author SHA1 Message Date
paul cannon
a33734bee7 satellite/satellitedb/dbx: add cockroach driver type
Change-Id: I7a0da6e066c67a521fc1b23b085ab8554eee0d4c
2020-01-06 18:01:03 +00:00
Ethan
05b406e992 satellite:{downtime,overlay}: Implement offline node detection chore
https://storjlabs.atlassian.net/browse/V3-3398

Change-Id: I598c3bad819026377d1d113c099dc9bba8b02742
2020-01-03 17:10:03 +00:00
Ethan
8859c36234 satellite/{downtime,contact}: Add CheckNodeAvailability for use within the downtime tracking chores.
https://storjlabs.atlassian.net/browse/V3-2545

Change-Id: I1dd54a0c77cb4905bb1f350beeb82c6f7700ee70
2020-01-02 18:24:11 +00:00
Moby von Briesen
ff74b44c5f satellite/overlay: Add ability for overlay to get offline nodes ordered by last checked time
This is required for the downtime tracking service: https://storjlabs.atlassian.net/browse/V3-2545

Change-Id: I286cdc07d802393948eb10c25c45ba78cc3ceafc
2020-01-02 16:39:38 +00:00
Egon Elbre
3528c56c6f satellite/satellitedb/satellitedbtest: skip unconfigured db
Change-Id: Ib6ea58208ef19410146845e82eb08724888e85ae
2020-01-02 10:51:59 +00:00
Moby von Briesen
bb3baf5a4e satellite/satellitedb: Add nodes_offline_times table for downtime tracking
Change-Id: If6b80fe0a20d88cedacaf4b76b75aa21d0af2465
2019-12-30 15:45:02 -05:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Egon Elbre
acb4435a67 satellite/satellitedb: improve Cockroach migrate test
Load schemas in parallel instead of one-by-one.

Optimizes from 2m30s to 1m15s.

Change-Id: I0bf6381a0ae99b44271fe55d4ee658683064c097
2019-12-21 10:58:43 +00:00
Yingrong Zhao
6e71591b9b satellitedb;storagenodedb: remove unnecessary use of DB transactions in graceful exit
Change-Id: Ief0a28c6750c130896b48bfebfbea7fb3caa810f
2019-12-20 21:24:38 +00:00
Jeff Wendling
ffbc43d170 satellite/satellitedb/dbx: monitor the database calls
Change-Id: I9fbdc4a35fedfc7a1b4e1b630d2a3664b3218e67
2019-12-18 21:57:36 +00:00
Kaloyan Raev
5ee1a00857 satellite/overlay: filter reliable nodes from a list
Adds the KnownReliable method to Overlay Service that filters all nodes
from the given list to be only reliable nodes (online and qualified).
The method return []*pb.Node of reliable nodes. The pb.Node values are
ready for dialing.

The first use case is when deleting an object to efficiently dial all
reliable nodes holding a piece of that object and send them a delete
request.

Change-Id: I13e0a8666f3807c5c31ef1a1087476018a5d3acb
2019-12-17 21:20:08 +00:00
Cameron Ayer
a4f9865b47 satellite: adds and enables cockroachdb compatibility for tests
Change-Id: I85a3ad8c3b9d7e15ea8675b6c55af0002933db57
2019-12-16 22:29:25 +00:00
paul cannon
2f7465c294 private/dbutil: register "cockroach" as sql.DB driver
this will allow us to inspect the type of `db.Driver()` on *sql.DB
connections to correctly differentiate between pg and crdb conns.

as a bonus, this moves all concerns about when to replace "cockroach://"
with "postgres://" out of view, letting the thin shim "driver" take care
of that.

Change-Id: Ib24103ab7c508231e681f89a7321b623e4e125e9
2019-12-16 19:10:00 +00:00
paul cannon
b5ddfc6fa5 satellite/satellitedb: unexport satellitedb.DB
Backstory: I needed a better way to pass around information about the
underlying driver and implementation to all the various db-using things
in satellitedb (at least until some new "cockroach driver" support makes
it to DBX). After hitting a few dead ends, I decided I wanted to have a
type that could act like a *dbx.DB but which would also carry
information about the implementation, etc. Then I could pass around that
type to all the things in satellitedb that previously wanted *dbx.DB.

But then I realized that *satellitedb.DB was, essentially, exactly that
already.

One thing that might have kept *satellitedb.DB from being directly
usable was that embedding a *dbx.DB inside it would make a lot of dbx
methods publicly available on a *satellitedb.DB instance that previously
were nicely encapsulated and hidden. But after a quick look, I realized
that _nothing_ outside of satellite/satellitedb even needs to use
satellitedb.DB at all. It didn't even need to be exported, except for
some trivially-replaceable code in migrate_postgres_test.go. And once
I made it unexported, any concerns about exposing new methods on it were
entirely moot.

So I have here changed the exported *satellitedb.DB type into the
unexported *satellitedb.satelliteDB type, and I have changed all the
places here that wanted raw dbx.DB handles to use this new type instead.
Now they can just take a gander at the implementation member on it and
know all they need to know about the underlying database.

This will make it possible for some other pending code here to
differentiate between postgres and cockroach backends.

Change-Id: I27af99f8ae23b50782333da5277b553b34634edc
2019-12-16 19:09:30 +00:00
Jessica Grebenschikov
c5116cb2a0 satellitedb: fix migration cockroach test
Change-Id: Ie3b4a4b0795d156238d50a58078282cc0918a334
2019-12-16 18:02:31 +00:00
Vitalii Shpital
53d9bc4530
storagenode/notifications: db created (#3707) 2019-12-16 19:59:01 +02:00
Yaroslav Vorobiov
8cf1aa6e4f
satellite/accounting: fix project limits migration (#3717) 2019-12-10 18:12:49 +02:00
Jeff Wendling
48da8baab5 storj-sim: work with cockroach:// urls for satellite databases
for storj-sim to work, we need to avoid schemas in cockroach urls
so we have storj-sim create namespaced databases instead of schemas
and we have the migrate command create the database in the same way
that it would create a schema for postgres. then it works!

a follow up commit will move the creation of the database/schemas
into storj-sim's setup step so that we can avoid doing these icky
creations during normal migration calls. it will also make the
pointerdb have an explicit call to migrate instead of just doing
it every time it's opened.

Change-Id: If69ef5cb96b6866b0438c761bd445afb3597ae5f
2019-12-09 23:44:00 +00:00
Natalie Villasana
c3c02bec3c
satellite/satellitedb: reset storage node reputations to re-enable disqualification (#3693) 2019-12-09 12:04:00 -05:00
Egon Elbre
56a3b62bef satellite/satellitedb: ensure migration tests run (#3706)
satellitedb migration tests ran against multiple base versions, however after the merging all the steps the base versions didn't exists anymore - which meant none of the migration tests were actually running.
2019-12-09 09:26:58 -06:00
Vitalii Shpital
fa5288c254
satellitedb: bucket search fixed (#3594) 2019-12-09 14:46:30 +02:00
Jennifer Johnson
8e9532dbd9 satellitedb/repairqueue: use errs.New() instead of fmt.Errorf() to retain stack trace
Change-Id: I47f1985aaeace556e3e0f7a20d2718410936db17
2019-12-05 15:52:25 +00:00
paul cannon
378b863b2b private,satellite: unite all the "temp db schema" things
first, so that they all work the same way, because it's getting
complicated, and second, so that we can do the appropriate thing
instead of CREATE SCHEMA for cockroachdb.

Change-Id: I27fbaeeb6223a3e06d97bcf692a2d014b31465f7
2019-12-05 15:36:59 +00:00
Cameron
28a66747c9 satellite/satellitedb: cockroachDB compatible transaction in offersDB.Create (#3686)
* use crdb wrapper for create/update offer transaction, ensure times returned are UTC

* round to nearest microsecond to compare times
2019-12-04 21:54:43 -06:00
Jennifer Johnson
7c5f777a4f satellitedb/repairqueue: runs a different implementation of the query within Select() for postgres vs cockroach
Change-Id: Ie34dbdb9d870d7d9f8f269702b6b3bad0c55b98e
2019-12-04 21:32:02 +00:00
Jennifer Johnson
e9eff54473 satellitedb/attribution: updates valueAttrQuery to work for both postgres and cockroach
Change-Id: I55d6c21950a66bf782596e9152622efd700c324a
2019-12-04 18:42:02 +00:00
Jennifer Johnson
ecb960f506 private/dbutil: distinguishes between db drivers and implementations to allow for different implementations of SQL queries.
Change-Id: I2dc8d1d371139aa8bc805e92a2b80b71f580fd64
2019-12-04 18:31:26 +00:00
Cameron
c00f688930 format db name in query to match actual db name (#3688) 2019-12-04 08:55:10 -05:00
paul cannon
4d2881b711 satellite/satellitedb: return latest coupon time as UTC
it doesn't necessarily _have_ to be UTC; the time is correct as returned
either way, but this will make it a little less prone to variance.

also, there is a test that depends on the time being returned in UTC.

Change-Id: Ia71e24ecd9973ba70a1cfb5621a3030a5c82d004
2019-12-03 17:36:41 +00:00
Jess G
e0b9b5b317
satellitedb: fixes for cockroachdb compatibility (#3682)
* crdb compat fixes

* update sn accounting query
2019-12-03 08:24:46 -08:00
paul cannon
b1fa7cdfbb
satellite/satellitedb/satellitedbtest: flag to run cockroach tests (#3674)
This will make it so we don't need to comment out those lines every time
we want to enable the cockroachdb tests during development.

Once it's ready this flag can go away.
2019-12-02 14:06:21 -06:00
Yehor Butko
ae05fa0874
satellitedb/accounting: fixed query (#3672) 2019-11-29 17:53:57 +02:00
Yehor Butko
0f523e82a6
missing line returned (#3670) 2019-11-28 23:42:04 +02:00
Yehor Butko
b10eee6e7f
satellitedb/accounting: project usage optimization (#3668) 2019-11-28 20:45:31 +02:00
Yehor Butko
756b9b9e2b
satellite/payments: coupons and coupon usage (#3648) 2019-11-26 19:58:51 +02:00
Jess G
854e5507ab
crdb uses namespaced db for each test (#3646)
* crdb uses namespaced db for each test

* add test for me test

* fix lint and tests

* updates per cr comments

* rm all replaceall
2019-11-26 08:39:57 -08:00
Cameron
7abad3c6bb
refactor sql to be compatible with pq and cockroach (#3647) 2019-11-26 09:26:22 -05:00
Yaroslav Vorobiov
8a002e8c8e
satellite/accounting: separate project limit from project entity (#3632) 2019-11-25 16:18:04 +02:00
Jess G
388f33b84d
satellitedb: add support to testplanet for cockroachdb (#3634)
* update migration steps, add crdb support to testplanet

* add crdb support

* have jenkins run a bares bones crdb compat test

* skip crdb tests

* skip crdb tests

* fix root_piece_id column

* write crdb store to tmp dir

* escape
2019-11-22 11:59:46 -08:00
Matt Robinson
9af97d366a Make sed a little more cross platformable (#3629) 2019-11-22 11:17:02 -07:00
Nikolai Siedov
6a4389d3e1 satellite/console: apiKeys case-insensitive search added (#3621) 2019-11-21 00:56:55 +02:00
Yingrong Zhao
b995406ff9
satellite/satellitedb: separate uuid creation from db layer (#3600) 2019-11-20 14:16:27 -05:00
Jess G
e9c3194c82
satellitedb: merge migration into one step (#3551)
* merge migration

* rm migration versions

* rm unneeded migration test data

* create index w/postgres + crdb compatible syntax

* add default to offers.invitee_credit_duration_days

* changes so that schema matches from master to branch

* change to be crdb compatible

* add check to confirm db version

* mv version check to migration

* update tests

* add minversion to sadb migration, update tests

* confirm min version for all dbs in a migration

* add validate migration to sadb

* fix lint err

* rm min version check from migrate

* change sadb check

* hard code min db version

* fix comment
2019-11-19 12:52:57 -08:00
littleskunk
8b3444e088
satellite/nodeselection: don't select nodes that haven't checked in for a while (#3567)
* satellite/nodeselection: dont select nodes that havent checked in for a while

* change testplanet online window to one minute

* remove satellite reconfigure online window = 0 in repair tests

* pass timestamp into UpdateCheckIn

* change timestamp to timestamptz

* edit tests to set last_contact_success to 4 hours ago

* fix syntax error

* remove check for last_contact_success > last_contact_failure in IsOnline
2019-11-15 23:43:06 +01:00
Yaroslav Vorobiov
53c6741ba6
satellite/payments: add API for retrieving conversion ratio, convert tokens to USD before applying to balance (#3530) 2019-11-15 16:59:39 +02:00
Yehor Butko
a8e4e9cb03
satellite/payments: project usage charges (#3512) 2019-11-15 16:27:44 +02:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Egon Elbre
1e64006e32 lint: add staticcheck as a separate step (#3569) 2019-11-14 10:31:30 +02:00
Natalie Villasana
1a9757a7f2 satellite/gracefulexit: add count for order limits sent from satellite to exiting node (#3544) 2019-11-13 09:54:50 -05:00
Yaroslav Vorobiov
36311a3a05
satellite/console: add token deposit API, populate billing history with transactions (#3500) 2019-11-12 13:14:34 +02:00