Commit Graph

11 Commits

Author SHA1 Message Date
Michał Niewrzał
5dbc91622f metainfo-migration: fix flaky TestMigrator_SingleSegmentObj test
Test had two issues:
* sub test was using parent test to fail (ctx.Check)
* pointerDB and metabaseDB were using the same unique DB and pointerDB was closing/deleting it first

Change-Id: I5741b76d518663d80c4e6448c76e4ee9dd86c8e1
2021-03-29 17:31:58 +00:00
Michał Niewrzał
50b08a5b00 metainfo-migrator: handle 'omit' conn string
Uplink tests are running storj/storj tests only against postgres. Without this change integration test on uplink will fail as 'omit' is not supported in migrator tests code.

Change-Id: Ic72406f52439e98683d050508fb42aa41632e183
2021-03-23 13:33:22 +00:00
Michał Niewrzał
e0f07ef04e metainfo-migrator: ignore invalid segment keys
Old pointerdbs can have key with just project id, segment index and bucket, without object key. We need to ignore such keys.

Change-Id: I80a466a94e317a229da236fe6bc9e762e6f7ced6
2021-03-10 17:23:36 +00:00
Michał Niewrzał
80ae7787ee metainfo-migration: add simple retry logic for queries
Change-Id: I2fd93493f9f540a39583d4a2a2a22d25453cb0e5
2021-03-10 14:00:18 +01:00
Michał Niewrzał
ce4024a521 metainfo-migrator: handle objects with missing segments
We can have objects with missing segments. Such objects will be removed by segment reaper but between executions we can have it. We should not interrupt migration but collect such objects to cleanup migrated DB later.

Change-Id: I5cc4a66395c1773a6430f34cc25a0f2449133f80
2021-03-08 13:43:26 +00:00
Michał Niewrzał
79d6294dbe metainfo-migration: fix data race
Change-Id: Idc266509fe7a6a717689f090a1409db09c8e9daa
2021-02-16 09:53:17 +01:00
Michał Niewrzał
341a4b7f59 metainfo-migration: extend test case
Change-Id: I9d1ac41a18bb08200ef20f7ff2f8df5531140f99
2021-02-11 19:28:41 +01:00
Michał Niewrzał
5a23d9c167 metainfo-migrator: improve performance
Change-Id: I258605737c37dfdd8f0070dbadf0e3a9c6358cc3
2021-02-11 17:33:22 +01:00
Egon Elbre
da68f0cda7 cmd/metainfo-migration: add fastpb for faster unmarshaling
Change-Id: I7f9856fd48477cf54ade62b74935c26c411998aa
2021-01-27 16:26:12 +02:00
Michał Niewrzał
2d087c54b1 satellite/metainfo/metabase: use PlainSize for fixed segment calculation
In the past we were storing fixed segment size with StreamInfo, encrypted in metadata. The value was unencrypted size of segment, not encrypted one.

Change-Id: Id6b18440c674223eabbb152b1636c83e1ab6462c
2021-01-20 16:40:27 +00:00
Michał Niewrzał
ac058e5ecc metainfo-migration: basic pointerdb->metabase migrator
Change-Id: If183b9c99ce816521fcae6e7189a6f85ddd4eb48
2021-01-12 12:59:53 +00:00