JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
59ebb0ef27
storj/satellite/metabase
History
Michal Niewrzal 06b51258be satellite/metabase: use table stats if are up to date 
Currently, to get number of entries in segments table we are doing
heavy SELECT count(*) operation. For biggest satellite it's taking
25min now. We are using this method to get stat before and after
segments loop so it adds almost 1h to overall loop time.

With current version of crdb we are using this additional code won't be
used because global configuration for stats refresh rate is inaccurate
for such large table like `segments`. Soon we should be able to upgrade
crdb and be able to adjust refresh rate per table and configure it to
satisfy defined threshold.

https://github.com/storj/storj/issues/5544

Change-Id: I05cfd9154f08894d2bc56bf716b436d1b03b87f1
2023-03-13 14:54:13 +00:00
..
metabasetest satellite/metabase/metabasetest: fix race while running tests 2023-02-02 10:35:28 +00:00
rangedloop satellite/rangedloop: migrate segments verification from segment loop 2023-03-08 17:00:11 +00:00
segmentloop satellite/metabase: add piece size calculation to segment 2023-02-22 11:04:02 +00:00
zombiedeletion satellite/metainfo/zombiedeletion: enable chore by default 2021-12-21 13:26:18 +01:00
alias_test.go ci: fix lint errors 2021-05-17 13:37:31 +00:00
alias.go satellite/metabase,cmd/tools/segment-verify: simplify interface 2022-10-06 13:42:00 +00:00
aliascache_test.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
aliascache.go cmd/tools/segment-verify: tool for checking duplicate net 2022-12-15 22:58:36 +00:00
aliaspiece_test.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
aliaspiece.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
bench_expired_test.go all: fix nolint directives 2022-10-11 18:31:20 +00:00
bench_test.go all: fix nolint directives 2022-10-11 18:31:20 +00:00
commit_object_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
commit_object.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
commit_test.go satellite/metabase: always try to remove old version on commit 2022-12-09 13:45:03 +00:00
commit.go satellite/metabase: always try to remove old version on commit 2022-12-09 13:45:03 +00:00
common_test.go satellite/metabase: add piece size calculation to segment 2023-02-22 11:04:02 +00:00
common.go satellite/metainfo: fix duplicates while listing committed objects 2023-02-14 14:47:27 +01:00
consts_test.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
copy_object_test.go satellite/metainfo: enable multiple versions fix by default 2023-01-26 09:35:20 +00:00
copy_object.go satellite/metabase: copy handles version when overwriting target location 2022-09-28 11:39:24 +00:00
db_internal_test.go satellite/metabase/segmentloop: limit max interval 2021-06-10 17:06:33 +00:00
db_test.go satellite/metabase: drop alias migration code 2021-05-07 12:13:28 +03:00
db.go satellite/metabase: add database comments 2023-02-07 11:42:21 +02:00
delete_bucket_test.go satellite: adjust code to handle context cancelation for SQL queries 2023-01-27 10:07:43 +01:00
delete_bucket.go satellite/metabase: add missing error check 2023-01-26 11:18:14 +00:00
delete_objects_test.go satellite/metabase: delete migrated zombie objects 2022-08-10 10:16:47 +02:00
delete_objects.go satellite/metabase: delete migrated zombie objects 2022-08-10 10:16:47 +02:00
delete_test.go satellite/metainfo: use multiple object versions internally 2022-10-06 15:19:02 +00:00
delete.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
encoding.go satellite/metabase: deletion query as explicit transaction 2022-03-31 11:40:36 +00:00
get_test.go satellite/metainfo: use multiple object versions internally 2022-10-06 15:19:02 +00:00
get.go satellite/metabase: add piece size calculation to segment 2023-02-22 11:04:02 +00:00
iterator_test.go satellite/metabase: make UploadID stable for different options 2022-11-10 17:35:36 +00:00
iterator_util_test.go satellite/metabase: fix listing prefixes with cursor set 2021-07-07 14:30:46 +00:00
iterator.go satellite/metabase: make UploadID stable for different options 2022-11-10 17:35:36 +00:00
list_objects_test.go satellite/metabase: ListObjects 2022-09-20 11:15:44 +00:00
list_objects.go satellite/metainfo: fix duplicates while listing committed objects 2023-02-14 14:47:27 +01:00
list_segments_test.go satellite/metabase: flatten copy references when copying 2022-03-20 10:29:37 +00:00
list_segments.go satellite/metabase/metabasetest: detect full scan table queries 2023-01-23 19:40:20 +00:00
list_verify_segments_test.go satellite/metabase: add logic for verifying segments in given buckets 2022-12-08 09:45:15 +00:00
list_verify_segments.go satellite/metabase: add logic for verifying segments in given buckets 2022-12-08 09:45:15 +00:00
list.go satellite/metabase: drop IterateObjectsAllVersions 2022-05-26 14:12:20 +00:00
loop_test.go satellite/audit: use NodeAlias instead pure NodeID 2023-02-21 15:31:59 +00:00
loop.go satellite/audit: use NodeAlias instead pure NodeID 2023-02-21 15:31:59 +00:00
metadata_test.go satellite/metabase: update UpdateObjectMetadata to work with latest commited object 2022-08-24 10:28:57 +03:00
metadata.go satellite/metabase: update UpdateObjectMetadata to work with latest commited object 2022-08-24 10:28:57 +03:00
migrate_test.go metabase: use predefined snapshot for unit tests avoid migration 2022-06-01 12:07:00 +00:00
move_object_test.go satellite/metabase: handle target pending/committed objects while move 2023-01-04 08:50:51 +00:00
move_object.go satellite/metabase: handle target pending/committed objects while move 2023-01-04 08:50:51 +00:00
raw.go satellite/metabase/metabasetest: detect full scan table queries 2023-01-23 19:40:20 +00:00
stats_test.go satellite/metabase: use table stats if are up to date 2023-03-13 14:54:13 +00:00
stats.go satellite/metabase: use table stats if are up to date 2023-03-13 14:54:13 +00:00
streamstat_test.go satellite/metabase: handle copies with GetStreamPieceCountByNodeID 2022-12-13 12:32:56 +01:00
streamstat.go satellite/metabase: fix GetStreamPieceCountByNodeID full table scan 2023-01-10 23:35:22 +00:00
update_test.go satellite/metabase/test: extend default time diff for comparison 2022-05-23 10:24:58 +00:00
update.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
util.go satellite/metabase: add intLimitRange 2021-07-19 07:51:16 +00:00