Commit Graph

16 Commits

Author SHA1 Message Date
Egon Elbre
f2d8e97d97 satellite/satellitedb: simplify select nodes query construction
Change-Id: I07009b28762d4485929a2a999e8f4be8179bee51
2021-10-22 07:41:07 +00:00
Egon Elbre
51f488fc4b satellite/satellitedb: fix selection query with AOST
Union query for both reputable and new nodes didn't properly work. The
top level query is required to have an `AS OF SYSTEM TIME` statement as
well.

Change-Id: I8ee6dd5b700c2b1ed2aa562962bfa72be7eec30a
2021-10-19 16:59:40 +03:00
Egon Elbre
0858c3797a satellite/{metabase,satellitedb}: deduplicate AS OF SYSTEM TIME code
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.

As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.

Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.

Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
2021-05-11 12:40:36 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Cameron Ayer
05f8d2d0b1 satellite/satellitedb: filter offline suspended nodes from selection
Change-Id: I5a6f413453332238d579a7bf50eb30e9156f96c2
2021-03-27 23:36:46 +00:00
Moby von Briesen
6e2ef3b9ee Revert "satellite/satellitedb: Do not consider nodes with offline_suspended as reputable."
This reverts commit e24262c2c9.

Change-Id: I287deb2e52d03bbd698ed055f0f216b0b5bf2798
2021-01-04 14:28:37 +00:00
Moby von Briesen
e24262c2c9 satellite/satellitedb: Do not consider nodes with offline_suspended as reputable.
Nodes which are offline_suspended will no longer be considered for new
uploads. The current threshold that enters a node into offline
suspension is 0.6. Disqualification for offline suspension is still
disabled.

Change-Id: I0da9abf47167dd5bf6bb21e0bc2186e003e38d1a
2020-12-29 17:59:09 +00:00
Ethan Adams
6070018021
satellite/overlay: use AS OF SYSTEM TIME with Cockroach
Query nodes table using AS OF SYSTEM TIME '-10s' (by default) when on CRDB to alleviate contention on the nodes table and minimize CRDB retries. Queries for standard uploads are already cached, and node lookups for graceful exit uploads has retry logic so it isn't necessary for the nodes returned to be current.
2020-12-22 21:07:07 +02:00
Stefan Benten
494bd5db81
all: golangci-lint v1.33.0 fixes (#3985) 2020-12-05 17:01:42 +01:00
Egon Elbre
c23a8e3b81 go.mod: update pgx to v4.9.0
Fix query to use TextArray instead of VarcharArray.
Fix queries to use the correct type.

Change-Id: Ibb7e55adba277d05778118d81ca697470e72c374
2020-09-29 19:03:08 +00:00
Jennifer Johnson
4e2413a99d satellite/satellitedb: uses vetted_at field to select for reputable nodes
Additionally, this PR changes NewNodeFraction devDefault and testplanet config from 0.05 to 1.
This is because many tests relied on selecting nodes that were reputable based on audit and uptime
counts of 0, in effect, selecting new nodes as reputable ones.
However, since reputation is now indicated by a vetted_at db field that is explicitly set
rather than implied by audit and uptime counts, it would be more complicated to try to
update all of the nodes' reputations before selecting nodes for tests.
Now we just allow all test nodes to be new if needed.

Change-Id: Ib9531be77408662315b948fd029cee925ed2ca1d
2020-09-04 16:45:32 +00:00
paul cannon
bbdb351e5e all: use jackc/pgx in place of lib/pq
What:

Use the github.com/jackc/pgx postgresql driver in place of
github.com/lib/pq.

Why:

github.com/lib/pq has some problems with error handling and context
cancellations (i.e. it might even issue queries or DML statements more
than once! see https://github.com/lib/pq/issues/939). The
github.com/jackx/pgx library appears not to have these problems, and
also appears to be better engineered and implemented (in particular, it
doesn't use "exceptions by panic"). It should also give us some
performance improvements in some cases, and even more so if we can use
it directly instead of going through the database/sql layer.

Change-Id: Ia696d220f340a097dee9550a312d37de14ed2044
2020-07-13 15:54:41 +00:00
Cameron Ayer
3b4b5f45c7 satellite: replace references to Suspended with UnknownAuditSuspended
Change-Id: I3d2d00c95954c0546ad077702617895f262926ef
2020-06-23 14:19:22 +00:00
Egon Elbre
d86cce202c satellite/satellitedb: use arrays for arguments in node selection
This simplifies the code and makes queries faster:

name                               old time/op  new time/op  delta
SelectStorageNodes-32              7.72ms ± 6%  7.22ms ± 3%  -6.44%  (p=0.016 n=5+5)
SelectNewStorageNodes-32           7.75ms ± 2%  7.37ms ± 1%  -4.89%  (p=0.008 n=5+5)
SelectStorageNodesExclusion-32     16.9ms ± 0%  16.6ms ± 0%  -2.15%  (p=0.008 n=5+5)
SelectNewStorageNodesExclusion-32  17.2ms ± 0%  16.6ms ± 2%  -3.69%  (p=0.008 n=5+5)
FindStorageNodes-32                45.5ms ± 0%  45.1ms ± 1%    ~     (p=0.056 n=5+5)
FindStorageNodesExclusion-32       77.4ms ± 0%  75.9ms ± 0%  -1.91%  (p=0.008 n=5+5)

Change-Id: I38f77f6282b9738e8416113d42c6acb46c03da7b
2020-04-09 21:16:10 +03:00
Egon Elbre
ccf4f9ed2d satellite/satellitedb: node selection code cleanup
Reduce the number of non-methods to reduce funcs in the namespace also
combine a func to slightly condense the code more.

Change-Id: Ifbe728eb8c8ca4c981df648decd259c2097b6b40
2020-04-09 20:41:29 +03:00
Natalie Villasana
cf80b3caf3
satellite/overlay: combine SelectStorageNodes and SelectNewStorageNodes (#3831) 2020-04-09 11:19:44 -04:00