Commit Graph

23 Commits

Author SHA1 Message Date
Egon Elbre
9b520b2114 satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes
They are needed for segment-verify tool.

Also rename some of the conversion methods to make clear,
which of them have side-effects.

Change-Id: Ie9a0952548e9ed5068c7a30c2fd2134b07139bca
2022-09-15 13:56:10 +00:00
Qweder93
df5b0d0044 satellite/metabase: added method DeleteObjectLastCommitted
method similar to metabase.DeleteObjectExactVersion which will delete last committed object

Closes 	https://github.com/storj/storj/issues/4872

Change-Id: Ia9f8c227dc59575bf8ed297886b35536097028b4
2022-08-31 12:26:56 +00:00
Erik van Velzen
74f4f6e765 satellite/metabase: fix copy to ancestor location
Previously copying an object to it's ancestor location (copy of copy)
broke the object and all copies.

This fixes this by calling the existing delete method rather than a
custom one when there is an existing object at the copy destination.

The check for existing object at destination has been moved to an
earlier point in FinishCopy.

metabase.DeleteObject exposes a transaction parameter so that it can be
reused within metabase.

Closes https://github.com/storj/storj/issues/4707

Uplink test at https://review.dev.storj.io/c/storj/uplink/+/7557

Change-Id: I418fc3337fa9f30146ccc1db456af168ae41c326
2022-06-28 03:53:01 +02:00
Erik van Velzen
4da61aa257 satellite/metabase: segment_copies full table scan
Part of the delete query cause a full table scan of segment_copies. This
slowed down the system. This change should have the same semantics but
improved performance.

Part of https://github.com/storj/storj/issues/4898

Change-Id: I4afe23df05467eafc9c91591f47a7251a0f3dd31
2022-06-27 11:03:15 +00:00
Egon Elbre
11dd0687c9 satellite/metabase: add missing monitoring
Change-Id: I8b1049aafc0d2c6029a68cac0e6fdc3d7b25b30d
2022-06-08 12:34:00 +00:00
Fadila Khadar
792c49e5ad satellite/metabase: simplify delete query
when deleting an object that has been copied multiple times, we look for an ancestor_stream_id by taking the min of all copies stream_id.

This change simplifies this process by picking any stream_id as a new ancestor by using 'distinct on'.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Iffb519b82d2ae2ed73af48fa0e86f87384e0158f
2022-05-27 20:09:50 +00:00
Qweder93
384118ee1e satellite/metabase: use pgutil.NullTimestampTZArray
This code cleanup to use helper DB method.

Change-Id: Idc438234cb4f7da23d6ade2d69f5d889f15e9009
2022-05-03 19:30:21 +00:00
Erik van Velzen
16480c97da satellite/metabase: deduplicate delete query
Use the same query when deleting a single object or multiple.
I have chosen not to deduplicate the row "scan" logic because
it is less complicated code and this change would expand to other
parts of the codebase.

Part of https://github.com/storj/storj/issues/4700

Change-Id: I7a958c78c903b2bddd72ca217971f7e8e02a0d0c
2022-04-11 14:12:18 +00:00
Michał Niewrzał
25e92fe443 satellite/metabase: deletion query as explicit transaction
Latest CRDB version did't work with our server-side copy deletion
query and we had to rewrite it.

Fixes https://github.com/storj/storj/issues/4655

Change-Id: I66d5c4abfc3c3f53dea8bc01ae11cd2b020e4289
2022-03-31 11:40:36 +00:00
Michał Niewrzał
ba74cb17a9 satellite: adjust to latest CRDB version
We have an issue with latest CRDB. Single query cannot modify
the same table multiple times. Now build is blocked.
This change is unblocking build by:
* adjusting query for inserting into repair queue
* temporary removing code for deletion for server-side copy
* temporary disable backward compatibility tests for CRDB

Change-Id: Idd9744ebd228e5dc05bdaf65cfc8f779472a975d
2022-03-23 07:16:38 +00:00
Michał Niewrzał
0bde845a17 satellite/metabase: don't delete pieces when deleting ancestor object
Fixes https://github.com/storj/storj/issues/4613

Change-Id: I3d6217a618a2a685256471f0394a143a323ac044
2022-03-21 09:32:26 +00:00
Michał Niewrzał
35efb6462c satellite/metainfo: adjust deleting to server side copy
For server-side copy we adjusted one method DeleteObjectExactVersion.
Other deletion methods won't be used directly in code at the moment.
We will adjust other methods later or decide if we will need them at
all.

To handle deletion of objects with copies or just copies correctly we
need to use DeleteObjectExactVersion method in two places while:
* removing object before upload
* explicit object deletion

This change is also changing DeleteObjectExactVersion method to
delete pending objects because we need this functionality to
delete object before new upload.

https://github.com/storj/storj/issues/4481

Change-Id: Ieff5cc95732bb70ed8cc0ecdd62e03c929857c02
2022-03-17 13:57:01 +01:00
Erik van Velzen
53e851bfb2 satellite/metabase/delete: adjust delete for copy feature
Adjust deletion of committed exact version so that it takes into account
copied objects.

Change-Id: I5daaa94ff8ad87417d52c79f305bc3275b683efa
2022-03-04 13:50:58 +00:00
Erik van Velzen
a9bd983f04 sql: capitalize keywords
Capitalize some keywords which were overlooked

Change-Id: Ie2ad283669e2ca2650fcddfd8c7395a81bac09a8
2022-03-01 15:19:38 +00:00
Michał Niewrzał
bc161794fc satellite/metabase: drop DeleteObjectLatestVersion method
This method was never used, except tests.

Change-Id: Idc1e69b2e2971995b5c4e6cf78a2b5fc69f39ad2
2022-02-02 14:33:48 +00:00
Egon Elbre
187941ff86 satellite/{metabase,metainfo}: add some missing monitoring
Change-Id: Ie0018f6385bb145d0a514dfe2792f3f299b55b79
2021-10-21 09:15:04 +00:00
Michał Niewrzał
aa80d37053 satellite/metabase: add automatic conversion to DB value for ObjectKey fields
We were manually converting ObjectKey fields to []byte to use it with
SQL query but we can just implement Value method to convert it automatically.

Change-Id: I6d346f4b59718e1e8ef37cd9f95e613b864b42cd
2021-10-05 07:34:00 +00:00
Michał Niewrzał
ae345320fe satellite/metabase: use db methods that pass context correctly
It turns out that some DB methods are not
passing context to to lower levels even if have
context as a parameter.

Change-Id: I82f4fc1a47c362687a91364d85e4468057e53134
2021-07-29 09:32:31 +00:00
Egon Elbre
5044337440 satellite/metabase: add monitoring for objects and segments
Currently it's difficult to gather how many objects and segments are
being inserted. Adding separate monitoring counters make this easier.

Change-Id: I986cd82f03e99d2aa6fc76028255ee1090d1b294
2021-06-04 20:01:10 +03:00
Egon Elbre
0858c3797a satellite/{metabase,satellitedb}: deduplicate AS OF SYSTEM TIME code
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.

As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.

Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.

Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
2021-05-11 12:40:36 +03:00
Egon Elbre
5269596c7d satellite/{metabase,metainfo}: use ObjectStream as argument
Change-Id: I5a7f096002b4d7a6162b24d7a64346b058c4c89c
2021-05-04 15:30:59 +00:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00