storj/satellite/metabase
Michal Niewrzal db1409eea6 satellite/metabase: use SUBSTRING with objects iterator
Until this change we were stripping prefix from object key on satellite side. Because of that we were transferring over network unnecessary data
from DB. This change adjusts iterator SQL queries to use SUBSTRING to
remove prefix on DB side and avoid sending it to satellite.

Benchmark against 'main':
unfortunately "time/op" is very unstable while doing local bench in this
case and sometimes  there is no difference in time and sometimes its up to 18%. I never saw results when old solution is faster then new one. Results for "alloc/op" and "allocs/op" are rather consistent.

name                                                 old time/op    new time/op    delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      1.98ms ± 6%    2.05ms ±23%     ~     (p=1.000 n=9+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8    3.97ms ± 8%    3.42ms ±20%  -13.86%  (p=0.005 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8    8.42ms ±16%    7.58ms ± 5%   -9.91%  (p=0.002 n=10+10)

name                                                 old alloc/op   new alloc/op   delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      16.7kB ± 0%    16.9kB ± 0%   +1.16%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8    27.3kB ± 0%    28.2kB ± 0%   +3.31%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8    60.0kB ± 0%    62.4kB ± 0%   +3.93%  (p=0.000 n=10+8)

name                                                 old allocs/op  new allocs/op  delta
NonRecursiveListing/Cockroach/listing_no_prefix-8         312 ± 0%       315 ± 0%   +0.96%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8       526 ± 0%       541 ± 0%   +2.85%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8     1.16k ± 0%     1.23k ± 0%   +5.24%  (p=0.000 n=10+10)

Change-Id: I23e501494ededafb2dd5ea903e8e4e313b42e956
2022-10-10 14:27:26 +00:00
..
metabasetest satellite/metabase: add tests for ListVerifySegments 2022-09-29 11:52:51 +00:00
segmentloop satellite/metabase/segmentloop: remove unused monkit calls 2022-10-03 14:07:18 +00:00
zombiedeletion satellite/metainfo/zombiedeletion: enable chore by default 2021-12-21 13:26:18 +01:00
alias_test.go ci: fix lint errors 2021-05-17 13:37:31 +00:00
alias.go satellite/metabase,cmd/tools/segment-verify: simplify interface 2022-10-06 13:42:00 +00:00
aliascache_test.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
aliascache.go cmd/tools/segment-verify: a few fixes 2022-10-05 15:51:38 +00:00
aliaspiece_test.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
aliaspiece.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
bench_expired_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
bench_test.go satellite/metabase: add metabase.GetObjectLatestVersion back 2022-08-11 22:14:38 +00:00
commit_object_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
commit_object.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
commit_test.go satellite/metainfo: use multiple object versions internally 2022-10-06 15:19:02 +00:00
commit.go satellite/metainfo: use multiple object versions internally 2022-10-06 15:19:02 +00:00
common_test.go segment/{metabase,repair}: add dedicated methods on metabase.Pieces 2021-08-03 15:12:03 +00:00
common.go satellite/metainfo: use multiple object versions internally 2022-10-06 15:19:02 +00:00
consts_test.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
copy_object_test.go satellite/metainfo: use multiple object versions internally 2022-10-06 15:19:02 +00:00
copy_object.go satellite/metabase: copy handles version when overwriting target location 2022-09-28 11:39:24 +00:00
db_internal_test.go satellite/metabase/segmentloop: limit max interval 2021-06-10 17:06:33 +00:00
db_test.go satellite/metabase: drop alias migration code 2021-05-07 12:13:28 +03:00
db.go satellite: optional migration for integration tests 2022-10-04 11:35:03 +02:00
delete_bucket_test.go satellite/metabase: delete bucket with cross-bucket copies 2022-04-05 01:32:02 +02:00
delete_bucket.go satellite/metabase: measure delete object batch 2022-08-08 12:24:26 +00:00
delete_objects_test.go satellite/metabase: delete migrated zombie objects 2022-08-10 10:16:47 +02:00
delete_objects.go satellite/metabase: delete migrated zombie objects 2022-08-10 10:16:47 +02:00
delete_test.go satellite/metainfo: use multiple object versions internally 2022-10-06 15:19:02 +00:00
delete.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
encoding.go satellite/metabase: deletion query as explicit transaction 2022-03-31 11:40:36 +00:00
get_test.go satellite/metainfo: use multiple object versions internally 2022-10-06 15:19:02 +00:00
get.go {satellite/metabase, satellite/metainfo, satellite/inspector} : Use metabase.GetObjectLastCommitted instead metabase.GetObjectExactVersion 2022-09-08 07:27:22 +00:00
iterator_test.go satellite/metabase: test list objects with version >1 2022-09-26 08:07:11 +00:00
iterator_util_test.go satellite/metabase: fix listing prefixes with cursor set 2021-07-07 14:30:46 +00:00
iterator.go satellite/metabase: use SUBSTRING with objects iterator 2022-10-10 14:27:26 +00:00
list_objects_test.go satellite/metabase: ListObjects 2022-09-20 11:15:44 +00:00
list_objects.go satellite/metabase: ListObjects 2022-09-20 11:15:44 +00:00
list_segments_test.go satellite/metabase: flatten copy references when copying 2022-03-20 10:29:37 +00:00
list_segments.go satellite/metabase: copy expires_at to segments while FinishCopyObject 2022-04-27 18:39:54 +00:00
list_verify_segments_test.go satellite/metabase: add tests for ListVerifySegments 2022-09-29 11:52:51 +00:00
list_verify_segments.go satellite/metabase: add tests for ListVerifySegments 2022-09-29 11:52:51 +00:00
list.go satellite/metabase: drop IterateObjectsAllVersions 2022-05-26 14:12:20 +00:00
loop_test.go satellite/metainfo: use multiple object versions internally 2022-10-06 15:19:02 +00:00
loop.go satellite/metainfo: propagate geofencing between buckets and stream id 2021-11-24 08:05:05 +00:00
metadata_test.go satellite/metabase: update UpdateObjectMetadata to work with latest commited object 2022-08-24 10:28:57 +03:00
metadata.go satellite/metabase: update UpdateObjectMetadata to work with latest commited object 2022-08-24 10:28:57 +03:00
migrate_test.go metabase: use predefined snapshot for unit tests avoid migration 2022-06-01 12:07:00 +00:00
move_object_test.go satellite/metainfo: use multiple object versions internally 2022-10-06 15:19:02 +00:00
move_object.go satellite/{metabase, metainfo}: merge logic for BeginCopyObject and BeginMoveObject 2022-09-22 09:02:20 +00:00
raw.go metabasetest: use DROP instead of TRUNCATE 2022-05-31 12:16:02 +00:00
stats_test.go satellite/metabase: don't count objects for loop 2022-09-22 16:02:08 +00:00
stats.go satellite/metabase: don't count objects for loop 2022-09-22 16:02:08 +00:00
streamstat_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
streamstat.go satellite/metabase: use db methods that pass context correctly 2021-07-29 09:32:31 +00:00
update_test.go satellite/metabase/test: extend default time diff for comparison 2022-05-23 10:24:58 +00:00
update.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
util.go satellite/metabase: add intLimitRange 2021-07-19 07:51:16 +00:00