Commit Graph

14 Commits

Author SHA1 Message Date
Michal Niewrzal
b5c5c62d7b satellite/metabase: add missing error check
Change-Id: I6891f0647cb8e4a8dd6534eaa3588bbe76e2721d
2023-01-26 11:18:14 +00:00
Erik van Velzen
9e64a87f7e satellite/metabase: measure delete object batch
Split out the function to delete a batch of objects from a bucket, so
that we get metrics which give a rough indication how long this operation
takes.

Part of https://github.com/storj/storj/issues/4957

Change-Id: I20a4ed5894217f4cd0b2f25aee297f0ecda57ab5
2022-08-08 12:24:26 +00:00
Erik van Velzen
16480c97da satellite/metabase: deduplicate delete query
Use the same query when deleting a single object or multiple.
I have chosen not to deduplicate the row "scan" logic because
it is less complicated code and this change would expand to other
parts of the codebase.

Part of https://github.com/storj/storj/issues/4700

Change-Id: I7a958c78c903b2bddd72ca217971f7e8e02a0d0c
2022-04-11 14:12:18 +00:00
Erik van Velzen
54c3bafb5b satellite/metabase: delete bucket with cross-bucket copies
When deleting a bucket, make sure that object copies in other buckets are
promoted to new ancestor and left in a working state.

Closes https://github.com/storj/storj/issues/4591

Change-Id: I019d916cd6de5ed51dd0dd25f47c35d0ec666af6
2022-04-05 01:32:02 +02:00
Erik van Velzen
a9bd983f04 sql: capitalize keywords
Capitalize some keywords which were overlooked

Change-Id: Ie2ad283669e2ca2650fcddfd8c7395a81bac09a8
2022-03-01 15:19:38 +00:00
Michał Niewrzał
260a673327 satellite/metabase: don't delete pieces until query results are received
To optimize memory consumption we where consuming
segment data during processing results from delete
query. Turns out that there is a chance that query will be
rolled-back if something will go wrong while reading
results. In such case its possible to delete pices but
object/segment will be still in DB.
This change removed piece deletion from problematic
place. Pieces are still deleted in batches but are not
limited at the moment. To avoid memory issues object
deletion batch was decreased.

Change-Id: Icb3667220f9c25f64b73cf71d0cf3fdc7e5107c5
2021-08-08 03:43:17 +00:00
Michał Niewrzał
be92ff1751 satellite/metabase: fix finishing bucket objects deletion
In case when number of deleted segments in a single
batch was multiplication of opts.DeletePiecesBatchSize it
was possible to finish deletion of objects to early.

Change-Id: I9181d4a3c64053d9df46a11a6e0cd22153c73ee9
2021-08-02 09:00:02 +00:00
Michał Niewrzał
ae345320fe satellite/metabase: use db methods that pass context correctly
It turns out that some DB methods are not
passing context to to lower levels even if have
context as a parameter.

Change-Id: I82f4fc1a47c362687a91364d85e4468057e53134
2021-07-29 09:32:31 +00:00
Egon Elbre
5a56021bca satellite/metabase: add intLimitRange
We have quite a bit code duplication in ensuring valid int ranges.

Change-Id: Ib31db8c4b2be6a677625cab5bcdddefb60401fe3
2021-07-19 07:51:16 +00:00
Michał Niewrzał
1f5fbbd24a satellite/metabase: while deleting bucket objects delete pieces in batches
It's possible that single object will have a lot of segments and
at the moment we are trying to collect all pieces at once and
send to storage nodes information about deletion. Such
approach my lead to using extensive amount of memory. This
change is handling this problem by calling DeletePieces
function multiple times with only part of pieces to delete for
a single call.

Change-Id: Ie1e66cd9d86d130eb89a61cf6e23f38b8cb8859e
2021-06-25 13:27:00 +00:00
Egon Elbre
5044337440 satellite/metabase: add monitoring for objects and segments
Currently it's difficult to gather how many objects and segments are
being inserted. Adding separate monitoring counters make this easier.

Change-Id: I986cd82f03e99d2aa6fc76028255ee1090d1b294
2021-06-04 20:01:10 +03:00
Egon Elbre
0858c3797a satellite/{metabase,satellitedb}: deduplicate AS OF SYSTEM TIME code
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.

As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.

Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.

Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
2021-05-11 12:40:36 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00