JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
bed2af3e87
storj/satellite/metabase
History
Michał Niewrzał 7390f389cd satellite/metabase: non recursive listing optimization 
With pointerdb listing objects operation was optimized to skip
objects from prefixes for non recursive listing. This change it
adopting this optimiaztion from old code.

Main change is to drop current page results if we detect a prefix
that needs to be skipped and jump with next listing query after this
prefix by setting cursor to "some-prefix(byte('/')+1) which is
effectively "some-prefix0".

Benchmark:
name                                                 old time/op  new time/op  delta
NonRecursiveListing/Postgres/listing_no_prefix-8      960µs ±11%   257µs ±12%  -73.19%  (p=0.008 n=5+5)
NonRecursiveListing/Postgres/listing_with_prefix-8    945µs ±11%   671µs ±12%  -28.97%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_no_prefix-8    4.31ms ± 8%  1.19ms ± 7%  -72.44%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8  4.97ms ± 8%  3.35ms ±15%  -32.67%  (p=0.008 n=5+5)

Fixes https://github.com/storj/team-metainfo/issues/115

Change-Id: Iafdf3600d058abbaf441f792d32a7fc17cc08696
2022-07-27 11:26:35 +00:00
..
metabasetest satellite/metabase: fix copy to ancestor location 2022-06-28 03:53:01 +02:00
segmentloop satellite/metabase/segmentloop: start immediately on manual trigger 2021-09-28 18:32:03 +00:00
zombiedeletion satellite/metainfo/zombiedeletion: enable chore by default 2021-12-21 13:26:18 +01:00
alias_test.go ci: fix lint errors 2021-05-17 13:37:31 +00:00
alias.go satellite/metabase: drop alias migration code 2021-05-07 12:13:28 +03:00
aliascache_test.go satellite/metabase/metabasetest: move test utils to a separate package 2021-04-26 13:12:58 +03:00
aliascache.go satellite/metabase: deletion query as explicit transaction 2022-03-31 11:40:36 +00:00
aliaspiece_test.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
aliaspiece.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
bench_expired_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
bench_test.go satellite/metabase: drop IterateObjectsAllVersions 2022-05-26 14:12:20 +00:00
commit_object_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
commit_object.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
commit_test.go satellite/metabase: fix flaky TestCommitInlineSegment 2022-05-20 10:20:56 +00:00
commit.go satellite/metabase: Classify pending object error 2022-05-10 10:54:17 +00:00
common_test.go segment/{metabase,repair}: add dedicated methods on metabase.Pieces 2021-08-03 15:12:03 +00:00
common.go satellite/metabase: don't return expired objects 2022-07-04 21:32:30 +03:00
consts_test.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
copy_object_test.go satellite/metabase: error if source object changed 2022-07-19 13:27:55 +02:00
copy_object.go satellite/metainfo: usage limits for copy 2022-07-25 12:35:27 +02:00
db_internal_test.go satellite/metabase/segmentloop: limit max interval 2021-06-10 17:06:33 +00:00
db_test.go satellite/metabase: drop alias migration code 2021-05-07 12:13:28 +03:00
db.go metabase: support disabling serverside copy when already enabled 2022-06-24 17:56:43 +00:00
delete_bucket_test.go satellite/metabase: delete bucket with cross-bucket copies 2022-04-05 01:32:02 +02:00
delete_bucket.go satellite/metabase: deduplicate delete query 2022-04-11 14:12:18 +00:00
delete_objects_test.go satellite/metabase/metabasetest/create: return segments 2022-02-28 13:40:46 +00:00
delete_objects.go satellite/metabase: send metric about zombie segment deletion 2021-10-04 16:25:23 +00:00
delete_test.go satellite/metabase: simplify delete query 2022-05-27 20:09:50 +00:00
delete.go satellite/metabase: fix copy to ancestor location 2022-06-28 03:53:01 +02:00
encoding.go satellite/metabase: deletion query as explicit transaction 2022-03-31 11:40:36 +00:00
get_test.go satellite/metabase: don't return expired objects 2022-07-04 21:32:30 +03:00
get.go satellite/metabase: don't return expired objects 2022-07-04 21:32:30 +03:00
iterator_test.go satellite/metabase: non recursive listing optimization 2022-07-27 11:26:35 +00:00
iterator_util_test.go satellite/metabase: fix listing prefixes with cursor set 2021-07-07 14:30:46 +00:00
iterator.go satellite/metabase: non recursive listing optimization 2022-07-27 11:26:35 +00:00
list_segments_test.go satellite/metabase: flatten copy references when copying 2022-03-20 10:29:37 +00:00
list_segments.go satellite/metabase: copy expires_at to segments while FinishCopyObject 2022-04-27 18:39:54 +00:00
list.go satellite/metabase: drop IterateObjectsAllVersions 2022-05-26 14:12:20 +00:00
loop_test.go satellite/metabase: add option to override metadata with CommitObject 2022-04-26 08:03:52 +00:00
loop.go satellite/metainfo: propagate geofencing between buckets and stream id 2021-11-24 08:05:05 +00:00
metadata_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
metadata.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
migrate_test.go metabase: use predefined snapshot for unit tests avoid migration 2022-06-01 12:07:00 +00:00
move_object_test.go satellite/metabase: error if source object changed 2022-07-19 13:27:55 +02:00
move_object.go satellite/metabase: error if source object changed 2022-07-19 13:27:55 +02:00
raw.go metabasetest: use DROP instead of TRUNCATE 2022-05-31 12:16:02 +00:00
stats_test.go satellite/metabase: add method to get table statistics 2021-06-16 18:34:55 +03:00
stats.go satellite/metabase: use db methods that pass context correctly 2021-07-29 09:32:31 +00:00
streamstat_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
streamstat.go satellite/metabase: use db methods that pass context correctly 2021-07-29 09:32:31 +00:00
update_test.go satellite/metabase/test: extend default time diff for comparison 2022-05-23 10:24:58 +00:00
update.go satellite/metabase: use db methods that pass context correctly 2021-07-29 09:32:31 +00:00
util.go satellite/metabase: add intLimitRange 2021-07-19 07:51:16 +00:00