Commit Graph

10 Commits

Author SHA1 Message Date
Michał Niewrzał
6b88a675c5 metabase-createdat-migration: add migration tool
We recently added create_at column to segments table.
Old segments needs to get this value from objects table.
This tool will iterate over all objects and update corresponding
segments if create_at column is not set.

Change-Id: Ib5aedc384637e739ee9af84454af0639e2559416
2021-04-08 14:20:09 +00:00
Michał Niewrzał
a5224e7a6c satellite/metainfo/metaloop: use segment CreatedAt and RepairedAt
Repair checker expects to have information about CreatedAt and RepairedAt fields to calculate segment age metric.

Change-Id: I6b41df880d77133be541e14d10d91cc75759b339
2021-04-02 08:46:54 +00:00
Egon Elbre
b5e95ec79f satellite/metainfo/metabase: add extra validation to IterateLoopSegments
Change-Id: I377bf9f661dcff6ba29d10f2909eb59d6d361c5d
2021-03-06 11:19:36 +02:00
Egon Elbre
ec67413776 satellite/metainfo: add as of system time to object iteration
Change-Id: Idc1d5c7d983f331c60f5b75c5a977a25e197faf9
2021-03-03 13:39:47 +02:00
Egon Elbre
b0b7b81105 satellite/metainfo/metabase: streams iterator
Iterate over streams/segments rather than loading all of them into
memory. This reduces the memory overhead of metainfo loop.

Change-Id: I9e98ab98f0d5f6e80668677269b62d6549526e57
2021-03-03 13:16:49 +02:00
Michał Niewrzał
c51ea68ad3 satellite/metainfo/metabase: reduce number of fields for LoopSegmentEntry
For metainfo loop we need only some of Segment fields. By removing some of them we will reduce memory consumption during loop.

Change-Id: I4af8baab58f7de8ddf5e142380180bb70b1b442d
2021-03-02 15:04:54 +01:00
Michał Niewrzał
c223c2d845 satellite/metainfo/metabase: rename ListObjectsSegments to ListLoopSegmentEntries
This method will be used only with metainfo loop and we need to customize query to consume less memory.

Change-Id: Iaa97392f483c5df5609d501b3847b80eb1ea2583
2021-03-02 12:24:59 +00:00
Michał Niewrzał
8e97111513 satellite/metainfo/metabase: reduce fields number for LoopObjectEntry
We want to read from DB only those fields that are used by metainfo loop so we need to remove most of fields from LoopObjectEntry.

Change-Id: I14ecae288f631dc0ff54f4c560ce43b736eccdcf
2021-03-02 12:21:19 +00:00
Egon Elbre
7e72a231c2 satellite/metainfo/metabase: cast bucket_name to []byte
Currently our metabase assumption is that it may contain arbitrary
bucket names and endpoint applies the naming constraints as it sees fit.
However by passing bucket_name as TEXT pg and crdb automatically try to
convert it to []byte, which may or not may work as intended... or in
some cases not work at all.

Cast all bucket name arguments to []byte to make it work.

Change-Id: I44650f5c873010997398bb0163d7f56ff6d9b5cf
2021-03-02 13:22:49 +02:00
Michał Niewrzał
646cf229a2 satellite/metainfo/metabase: rename FullIterateObjects method to IterateLoopObjects
We want to have custom loop iterator to avoid reading all object fields to reduce memory consumpion. This is first step to just rename existing iterator to IterateLoopObjects.

Change-Id: I8878ff21a49ba224db2d497cc8f9076e75c7609e
2021-03-01 17:19:05 +01:00