JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ad87d1de74
storj/satellite
History
Márton Elek ad87d1de74
satellite/satellitedb/overlaycache: fill node tags with join for limited number of nodes 
The easiest way to get node information WITH node tags is executing two queries:

 1. select all nodes
 2. select all tags

And we can pair them with a loop, using the in-memory data structures.

But this approach does work only, if we select all nodes, which is true when we use cache (upload, download, repair checker).

But repair process selects only the required nodes, where this approach is suboptimal. (full table scan for all tags, even if we need only tags for a few dozens nodes).

Possible solutions:

 1. We can introduce a cache for repair (similar to upload cache)
 2. Or we can select both node and tag information with one query (join).

This patch implements the second approach.

Note: repair itself is quite slow (10-20 seconds per segements to repair). With 15 seconds execution time and 3 minutes cache staleness, we would use the cache only 12 times per worker. Probably we don't need cache for now.

https://github.com/storj/storj/issues/6198

Change-Id: I0364d94306e9815a1c280b71e843b8f504e3d870
2023-09-07 19:27:53 +02:00
..
abtesting console/abTesting: add support for AB testing 2022-10-27 10:57:12 +00:00
accounting satellite/accounting/live: ensure we don't panic when we get nil 2023-08-10 19:41:57 +03:00
admin satellite/admin: support more options for passing project ID 2023-09-07 08:53:41 +00:00
analytics satellite/analytics: Update lifecyclestage for personal users 2023-08-24 09:47:26 -04:00
attribution satellite/{db, admin}: added endpoints to update user's and project's user_agent 2023-06-16 19:41:05 +00:00
audit satellite/metainfo: remove ServerSideCopyDuplicateMetadata 2023-08-08 12:15:10 +00:00
buckets satellite/buckets: handle bucket exists better 2023-08-23 14:40:31 +00:00
compensation payments: don't redefine compensation rates twice 2023-06-10 12:40:43 +00:00
console satellite/console/consolewasm: no direct cast to grant.Permission 2023-09-07 13:27:53 +03:00
contact go.mod: bump lang to 1.19 and common 2023-08-10 18:41:15 +03:00
gc satellite/metainfo: remove ServerSideCopyDuplicateMetadata 2023-08-08 12:15:10 +00:00
geoip satellite/geoip: exclude nodes with represented_contry from geofencing 2023-08-14 12:36:33 +00:00
gracefulexit satellite/gracefulexit: use placement when gracefulexit pick new nodes 2023-08-21 15:40:05 +00:00
internalpb satellite/metabase: adjust BeginObjectNextVersion to use pending_objects 2023-08-02 14:42:26 +00:00
mailservice satellite/consoleweb: fix flaky TestAuth tests 2022-08-04 19:06:07 +00:00
metabase satellite/metabase: adjust BucketEmpty to use pending_objects table 2023-08-21 15:01:59 +00:00
metainfo satellite/metainfo: better metabase errors handling 2023-09-01 11:32:10 +00:00
metrics satellite/metabase/rangedloop: move Segment definition 2023-05-16 12:37:17 +00:00
nodeapiversion satellite/nodeapiversion: new table for tracking node api usage 2020-07-09 15:02:25 +00:00
nodeevents satellite/satellitedb/dbx: document node tables 2023-02-24 14:05:55 +00:00
nodeselection satellite/nodeselection: rename (NodeFilter).MatchInclude to Match 2023-08-31 16:17:52 +02:00
nodestats payments: don't redefine compensation rates twice 2023-06-10 12:40:43 +00:00
oidc satellite: add get user paged projects http endpoint 2023-08-08 14:02:53 +00:00
orders satellite/overlay: change Reliable and KnownReliable 2023-09-02 23:34:50 +00:00
overlay go.mod: bump dependencies (private,uplink,common) 2023-09-06 13:28:22 +02:00
payments satellite/payments: filter out draft invoices 2023-09-05 14:22:30 +00:00
repair go.mod: bump dependencies (private,uplink,common) 2023-09-06 13:28:22 +02:00
reputation satellite/audit: remove pieces when audits fail 2023-06-22 14:19:00 +00:00
revocation satellite/satellitedb: move tests to their domains 2021-02-19 17:29:15 +02:00
satellitedb satellite/satellitedb/overlaycache: fill node tags with join for limited number of nodes 2023-09-07 19:27:53 +02:00
snopayouts all: fix error naming 2021-04-29 15:38:21 +03:00
admin.go satellite/payments: Remove expired package credits 2023-05-22 11:58:37 +00:00
api_test.go satellite/api: configuration option to set additional node tag authorities 2023-08-03 08:34:02 +00:00
api.go satellite/api: configuration option to set additional node tag authorities 2023-08-03 08:34:02 +00:00
auditor.go satellite: switch to use nodefilters instead of old placement.AllowedCountry 2023-07-07 16:55:45 +00:00
configlock_test.go all: fix deprecated ioutil commands 2022-10-11 15:27:29 +00:00
core.go satellite/{console,payments}: freeze/warn storjscan users 2023-08-31 13:22:21 +00:00
gc-bf_test.go satellite/gc/bloomfilter: remove segments loop parts 2023-05-15 11:46:26 +00:00
gc-bf.go satellite/metabase: remove segmentsloop package 2023-05-18 19:08:29 +00:00
gc.go gc: remove segment loop dependency 2023-02-03 10:31:16 +00:00
peer.go satellite/api: configuration option to set additional node tag authorities 2023-08-03 08:34:02 +00:00
rangedloop.go satellite/repair/repairer: fix NumHealthyInExcludedCountries calculation 2023-07-10 12:01:19 +02:00
repairer.go satellite/repair/repairer: fix NumHealthyInExcludedCountries calculation 2023-07-10 12:01:19 +02:00
ui.go satellite/console: optional separate web app server 2023-07-11 12:17:35 -04:00