Commit Graph

11 Commits

Author SHA1 Message Date
Michal Niewrzal
61933bc6f0 satellite/{metainfo,metabase}: optimize expired/zombie objects deletion
* optimize SQL for zombie objects deletion query by reducing some direct
selects to segments table
* set AOST for expired/zombie object deletion (was 0 since now)

https://github.com/storj/storj/issues/5881

Change-Id: I50482151d056a86fe0e31678a463f413d410759d
2023-06-09 11:22:46 +00:00
Michal Niewrzal
917925bc11 satellite/metabase: delete migrated zombie objects
We noticed that in the system we have undeleted very old pending
objects. General rule is to delete them after some inactivity. Turns
out that all those objects are objects migrated to metabase from
previous DB schema. During this migration we didn't set
zombie_deletion_deadline to any value.

This change takes into account pending objects with zombie deletion
deadline set to nil during zombie deletion process.

I also checked accross all production satellites and youngest pending
objects with nil zombie_deletion_deadline are from 2021 so it is safe
to delete them.

Change-Id: Ie2b6a4b4e203c1750cf8408ee281c0631b263082
2022-08-10 10:16:47 +02:00
Ivan Fraixedes
938efd7601 satellite/metabase: Don't stop on error expired & zombie objects
Don't terminate the expired objects loop or the zombie objects loop when
there is a DB error when selecting the objects for deleting them because
it isn't critical and the loops will pick them up again in the next
iteration.

The exception is if the DB rows scan method returns an error because
that's a symptom of the passed arguments to the method don't match with
the columns order, number, or type of the query, or there is invalid
data in the DB.

Don't also terminate these loops if the there is a DB error when
deleting the objects because the loops will pick them up in the next
iteration.

Because we don't return those errors now for not terminating the loop,
we have to log them.

Change-Id: I86bcf83d619345255840ae8f3db61620f044d2af
2022-08-08 10:18:49 +00:00
Michał Niewrzał
f2f8a9ca1e satellite/metabase: send metric about zombie segment deletion
We would like to verify if zombie object/segment works fine.
We need some metric for that. Figuring out number of deleted
objects is harder so let's for that later.

Change-Id: Ic99e2ce93256130b7c51f514824fddc009655075
2021-10-04 16:25:23 +00:00
Egon Elbre
707c4a7b0e satellite/metabase: use implicit tx when batch deleting
Change-Id: I2dc577b95209674b626a76633dedf5b5b1641db0
2021-08-30 15:04:35 +03:00
Michał Niewrzał
0344790c20 satellite/metabase: delete zombie object that has no new segments for specific period of time
Added options flag to define after which object won't be marked as inactive. All segments CreatedAt
time needs to be bellow this flag to treat object as inactive.

Change-Id: Ib5cffc776c6ee1b62b51eb8595438f968b42528c
2021-08-23 08:25:38 +00:00
Egon Elbre
5a56021bca satellite/metabase: add intLimitRange
We have quite a bit code duplication in ensuring valid int ranges.

Change-Id: Ib31db8c4b2be6a677625cab5bcdddefb60401fe3
2021-07-19 07:51:16 +00:00
Egon Elbre
5044337440 satellite/metabase: add monitoring for objects and segments
Currently it's difficult to gather how many objects and segments are
being inserted. Adding separate monitoring counters make this easier.

Change-Id: I986cd82f03e99d2aa6fc76028255ee1090d1b294
2021-06-04 20:01:10 +03:00
Egon Elbre
e7cf369cd8 satellite/metabase: use pgxutil.Conn helper
Clean up underlying pgx.Conn extraction with helper.

Change-Id: Ie43f663a6036c439e675df54fccd3605c4d8d69e
2021-05-25 05:56:35 +00:00
Egon Elbre
0858c3797a satellite/{metabase,satellitedb}: deduplicate AS OF SYSTEM TIME code
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.

As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.

Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.

Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
2021-05-11 12:40:36 +03:00
Michał Niewrzał
0d3865950f satellite/metabase: rename delete_expired.go to delete_objects.go
Logic for deleting specific object type can be reused so file name can be more general.

Change-Id: I75182c0d5e1b653eaa454c895f347dd33b4bebbc
2021-05-05 08:57:32 +00:00