Commit Graph

17 Commits

Author SHA1 Message Date
Michal Niewrzal
d1d98af098 satellite/metabase: jump after prefix at the end of iteration page
For non-recursive listing if cursor to the next iteration page
is a prefix then jump after this prefix to avoid listing objects from
this prefix.

Benchmark against 'main' - 7e5025cac0
name                                                 old time/op    new time/op    delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      11.7ms ±29%     1.7ms ± 7%  -85.52%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8    4.60ms ± 6%    3.23ms ± 3%  -29.72%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8    7.38ms ± 7%    8.47ms ±34%     ~     (p=0.421 n=5+5)

name                                                 old alloc/op   new alloc/op   delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      92.3kB ± 0%    16.9kB ± 0%  -81.72%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8    38.0kB ± 0%    27.4kB ± 0%  -28.02%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8    59.8kB ± 0%    60.1kB ± 0%   +0.38%  (p=0.008 n=5+5)

name                                                 old allocs/op  new allocs/op  delta
NonRecursiveListing/Cockroach/listing_no_prefix-8       1.78k ± 0%     0.31k ± 0%  -82.48%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8       734 ± 0%       526 ± 0%  -28.34%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8     1.15k ± 0%     1.16k ± 0%   +0.87%  (p=0.008 n=5+5)

Discussed here https://github.com/storj/team-metainfo/issues/116

Change-Id: Iff671f062d9af83ec419334089163dd204d0cac0
2022-08-23 13:32:30 +00:00
Michał Niewrzał
71c1d0fff2 satellite/metabase: reduce queries for iterator boundaries tests
This is longest metabase at the moment and would nice to speed
up is a bit to improve overal tests execution time.

Change-Id: I86da8e0e593d20024b3ec778cbeab34a4613151f
2022-08-22 10:47:28 +00:00
Michal Niewrzal
5458cbdad8 satellite/metabase/metabasetest: use interface for tests and benchmarks
metabasetest package utils can be used by both tests and benchmarks
if we will use interface TestingT from require package. This change
adjusts metabasettest.CreateObject method

Change-Id: I3c138e2ef9873b804ab5b3402804efa409397a9f
2022-08-03 13:08:06 +00:00
Michał Niewrzał
7390f389cd satellite/metabase: non recursive listing optimization
With pointerdb listing objects operation was optimized to skip
objects from prefixes for non recursive listing. This change it
adopting this optimiaztion from old code.

Main change is to drop current page results if we detect a prefix
that needs to be skipped and jump with next listing query after this
prefix by setting cursor to "some-prefix(byte('/')+1) which is
effectively "some-prefix0".

Benchmark:
name                                                 old time/op  new time/op  delta
NonRecursiveListing/Postgres/listing_no_prefix-8      960µs ±11%   257µs ±12%  -73.19%  (p=0.008 n=5+5)
NonRecursiveListing/Postgres/listing_with_prefix-8    945µs ±11%   671µs ±12%  -28.97%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_no_prefix-8    4.31ms ± 8%  1.19ms ± 7%  -72.44%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8  4.97ms ± 8%  3.35ms ±15%  -32.67%  (p=0.008 n=5+5)

Fixes https://github.com/storj/team-metainfo/issues/115

Change-Id: Iafdf3600d058abbaf441f792d32a7fc17cc08696
2022-07-27 11:26:35 +00:00
Qweder93
417df4d73c satellite/metabase: don't return expired objects
Resolves: https://github.com/storj/team-metainfo/issues/112

Change-Id: If48f865826c22764fc1ff83745c34524cd396aac
2022-07-04 21:32:30 +03:00
Michał Niewrzał
c934f45bfc satellite/metabase: drop IterateObjectsAllVersions
We are not using this method and most probably we
won't need to list objects with all statuses at once.
Removing for now.

Change-Id: I7aa0468c5f635ee2fb1fe51db382595c6343dd9c
2022-05-26 14:12:20 +00:00
Michał Niewrzał
6e5a94698e satellite/metabase: add option to override metadata with CommitObject
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.

https://github.com/storj/team-metainfo/issues/105

Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
2022-04-26 08:03:52 +00:00
Michał Niewrzał
289ae3325a satellite/metabase: always include encryption in listed object
We implemented filtering of system metadata to
reduce DB load in case user will need only list of
objects but we missed that encryption configuration
is needed to decrypt object key.

This change is always including encryption configuration into
list objects results.

Change-Id: Iaf7b3d8441480e3ad787f19a28f1b648584d5f7d
2021-11-12 05:08:01 +00:00
Erik van Velzen
91158fd3bf satellite: omit system metadata fields if not requested
To reduce database load, add option to omit as many object properties
as possible when listing objects.

Change-Id: I817633801b00629a4042d1d1bd2389ee581953de
2021-11-08 09:19:44 +01:00
Michał Niewrzał
8293c9659e satellite/metabase: speedup iterator tests
Main speedup is from reducing number of cases to iterate
in 'boundaries' test case.

Change-Id: I5bbf1311790f3d0e553d59e6a73e1cf3fa601a55
2021-10-19 11:52:07 +00:00
Michał Niewrzał
1fdb0eaa5b Revert "satellite/metabase: use storj.Nonce instead []byte"
This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.

This reverts commit 1ef06fae99.

Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
2021-10-12 15:39:54 +02:00
Michał Niewrzał
1ef06fae99 satellite/metabase: use storj.Nonce instead []byte
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.

Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
2021-10-11 16:13:34 +00:00
Clement Sam
6613133a1c satellite/metabase: return metadata from IterateObjectsAllVersionsWithStatus on demand
Change-Id: I6bb717cf941e2b5fd7c411bd1e5a77fad0835995
2021-08-06 16:20:39 +00:00
Michał Niewrzał
2e9d3a737c satellite/metabase: fix listing prefixes with cursor set
We were not skipping the initial prefix from the cursor.

Change-Id: I2bb472e960b92cae77fd1226de0b26fac79c429b
2021-07-07 14:30:46 +00:00
Michał Niewrzał
033006403f satellite/metainfo: fix setting object ZombieDeletionDeadline
So far we were setting ZombieDeletionDeadline alwasy as nil and because of that DB default was never set. This change adds separate query for inserting object if deadline is not set.

Change-Id: I3d6a16570e7c74b5304e13edad8c7adcd021340c
2021-05-11 07:26:42 +00:00
Egon Elbre
948529a600 satellite/metabase/metabasetest: move test utils to a separate package
Change-Id: Iaa607996478497d5c97c6a6973d099b315b3abf0
2021-04-26 13:12:58 +03:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00