JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
c3ca98f552
storj/satellite/metabase
History
Fadila Khadar 7788170234 satellite/metabase: ListObjects 
Current metainfo.ListObjects implementation is using metabase iterator to list objects.
In the non-recursive case, it used to retrieve all the corresponding rows and then discarded the entries that did not fit the listing request.
This can lead in some edge cases (each prefix contains more than batchsize objects/sub-prefixes) to make unecessary calls to the db.

This change defines the metabase.ListObjects and aims at retrieving only prefixes (but not objects under it) and objects by modifying the SQL query.

In this version, it is not optimized on the database side. Cockroach will still have to go through all rows under a prefix, so there is still room for improvement.

metainfo.ListObjects is not currently using this method as we would like to assess its performance on the QA satellite first.

Fixes https://github.com/storj/storj/issues/5088

Change-Id: Ied3a9210871871d9d4a3096888d3e40c2dceed61
2022-09-20 11:15:44 +00:00
..
metabasetest satellite/metabase: ListObjects 2022-09-20 11:15:44 +00:00
segmentloop satellite/metabase/segmentloop: add missing monitoring 2022-08-26 14:01:22 +02:00
zombiedeletion satellite/metainfo/zombiedeletion: enable chore by default 2021-12-21 13:26:18 +01:00
alias_test.go ci: fix lint errors 2021-05-17 13:37:31 +00:00
alias.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
aliascache_test.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
aliascache.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
aliaspiece_test.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
aliaspiece.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
bench_expired_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
bench_test.go satellite/metabase: add metabase.GetObjectLatestVersion back 2022-08-11 22:14:38 +00:00
commit_object_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
commit_object.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
commit_test.go satellite/metabase: fix flaky TestCommitInlineSegment 2022-05-20 10:20:56 +00:00
commit.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
common_test.go segment/{metabase,repair}: add dedicated methods on metabase.Pieces 2021-08-03 15:12:03 +00:00
common.go satellite/metabase: don't return expired objects 2022-07-04 21:32:30 +03:00
consts_test.go satellite/metabase: move package one level higher 2021-04-21 15:54:22 +03:00
copy_object_test.go satellite/metabase: copy when source and destination are the same 2022-09-13 19:21:13 +00:00
copy_object.go satellite/metabase: copy when source and destination are the same 2022-09-13 19:21:13 +00:00
db_internal_test.go satellite/metabase/segmentloop: limit max interval 2021-06-10 17:06:33 +00:00
db_test.go satellite/metabase: drop alias migration code 2021-05-07 12:13:28 +03:00
db.go metabase: support disabling serverside copy when already enabled 2022-06-24 17:56:43 +00:00
delete_bucket_test.go satellite/metabase: delete bucket with cross-bucket copies 2022-04-05 01:32:02 +02:00
delete_bucket.go satellite/metabase: measure delete object batch 2022-08-08 12:24:26 +00:00
delete_objects_test.go satellite/metabase: delete migrated zombie objects 2022-08-10 10:16:47 +02:00
delete_objects.go satellite/metabase: delete migrated zombie objects 2022-08-10 10:16:47 +02:00
delete_test.go satellite/metabase: added method DeleteObjectLastCommitted 2022-08-31 12:26:56 +00:00
delete.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
encoding.go satellite/metabase: deletion query as explicit transaction 2022-03-31 11:40:36 +00:00
get_test.go satellite/metabase: use GetObjectLastCommitted with move/copy 2022-08-30 06:36:02 +00:00
get.go {satellite/metabase, satellite/metainfo, satellite/inspector} : Use metabase.GetObjectLastCommitted instead metabase.GetObjectExactVersion 2022-09-08 07:27:22 +00:00
iterator_test.go satellite/metabase: jump after prefix at the end of iteration page 2022-08-23 13:32:30 +00:00
iterator_util_test.go satellite/metabase: fix listing prefixes with cursor set 2021-07-07 14:30:46 +00:00
iterator.go satellite/metabase: jump after prefix at the end of iteration page 2022-08-23 13:32:30 +00:00
list_objects_test.go satellite/metabase: ListObjects 2022-09-20 11:15:44 +00:00
list_objects.go satellite/metabase: ListObjects 2022-09-20 11:15:44 +00:00
list_segments_test.go satellite/metabase: flatten copy references when copying 2022-03-20 10:29:37 +00:00
list_segments.go satellite/metabase: copy expires_at to segments while FinishCopyObject 2022-04-27 18:39:54 +00:00
list_verify_segments_test.go satellite/metabase: add ListVerifySegments 2022-09-12 17:32:29 +03:00
list_verify_segments.go satellite/metabase: add ListVerifySegments 2022-09-12 17:32:29 +03:00
list.go satellite/metabase: drop IterateObjectsAllVersions 2022-05-26 14:12:20 +00:00
loop_test.go satellite/metabase: add option to override metadata with CommitObject 2022-04-26 08:03:52 +00:00
loop.go satellite/metainfo: propagate geofencing between buckets and stream id 2021-11-24 08:05:05 +00:00
metadata_test.go satellite/metabase: update UpdateObjectMetadata to work with latest commited object 2022-08-24 10:28:57 +03:00
metadata.go satellite/metabase: update UpdateObjectMetadata to work with latest commited object 2022-08-24 10:28:57 +03:00
migrate_test.go metabase: use predefined snapshot for unit tests avoid migration 2022-06-01 12:07:00 +00:00
move_object_test.go satellite/metabase: use GetObjectLastCommitted with move/copy 2022-08-30 06:36:02 +00:00
move_object.go satellite/metabase: use GetObjectLastCommitted with move/copy 2022-08-30 06:36:02 +00:00
raw.go metabasetest: use DROP instead of TRUNCATE 2022-05-31 12:16:02 +00:00
stats_test.go satellite/metabase: add method to get table statistics 2021-06-16 18:34:55 +03:00
stats.go satellite/metabase: use db methods that pass context correctly 2021-07-29 09:32:31 +00:00
streamstat_test.go Revert "satellite/metabase: use storj.Nonce instead []byte" 2021-10-12 15:39:54 +02:00
streamstat.go satellite/metabase: use db methods that pass context correctly 2021-07-29 09:32:31 +00:00
update_test.go satellite/metabase/test: extend default time diff for comparison 2022-05-23 10:24:58 +00:00
update.go satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 2022-09-15 13:56:10 +00:00
util.go satellite/metabase: add intLimitRange 2021-07-19 07:51:16 +00:00