Commit Graph

84 Commits

Author SHA1 Message Date
Michał Niewrzał
8293c9659e satellite/metabase: speedup iterator tests
Main speedup is from reducing number of cases to iterate
in 'boundaries' test case.

Change-Id: I5bbf1311790f3d0e553d59e6a73e1cf3fa601a55
2021-10-19 11:52:07 +00:00
Michał Niewrzał
f5f2398431 satellite/metabase: remove code related to part deletion
It is not used currently.

Change-Id: I32e9fa357814a3159ee8659a607cd6986f361fbf
2021-10-19 10:11:49 +00:00
Jeff Wendling
91ce70e4ed satellite/metabase: allow overwriting segments in pending objects
this is so that multipart upload allows overwriting parts. our
consistency model is more relaxed because part overwrites
won't be atomic if they consist of multiple segments, but at
least this allows overwriting at all.

Change-Id: I21dac4c24046e05efe74e6c6fd189a02c95eb6c8
2021-10-14 10:33:52 +02:00
Michał Niewrzał
1fdb0eaa5b Revert "satellite/metabase: use storj.Nonce instead []byte"
This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.

This reverts commit 1ef06fae99.

Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
2021-10-12 15:39:54 +02:00
Michał Niewrzał
1ef06fae99 satellite/metabase: use storj.Nonce instead []byte
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.

Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
2021-10-11 16:13:34 +00:00
Michał Niewrzał
aa80d37053 satellite/metabase: add automatic conversion to DB value for ObjectKey fields
We were manually converting ObjectKey fields to []byte to use it with
SQL query but we can just implement Value method to convert it automatically.

Change-Id: I6d346f4b59718e1e8ef37cd9f95e613b864b42cd
2021-10-05 07:34:00 +00:00
Michał Niewrzał
f2f8a9ca1e satellite/metabase: send metric about zombie segment deletion
We would like to verify if zombie object/segment works fine.
We need some metric for that. Figuring out number of deleted
objects is harder so let's for that later.

Change-Id: Ic99e2ce93256130b7c51f514824fddc009655075
2021-10-04 16:25:23 +00:00
Egon Elbre
6d3fd33ca3 satellite/metabase/segmentloop: start immediately on manual trigger
Currently loops wait for the coalesce duration for TriggerWait.
Let's skip the coalesce when we trigger it manually.

Change-Id: If5bacd4e263d233f1f3ea41b989922d2ed5a48d4
2021-09-28 18:32:03 +00:00
Qweder93
4fefa36a79 satellite/metabase: NewBucket added to metabase/metainfo FinishMoveObject methods
server-side move extended with moving between buckets, for this reason
we change bucket name for object in db.

Change-Id: Ie21bcccc170e6ff14dcd8053fdb86fdf6d8438a0
2021-09-24 17:53:02 +03:00
Qweder93
cae08d816c satellite/metabase: FinishMoveObject segment query improved
Change-Id: I198033d5763e6f386d3090fed49edcbfd95ff536
2021-09-22 12:51:48 +03:00
Egon Elbre
32cee1e572 satellite/metabase/segmentloop: ensure we shutdown on ctx err
Not all errors from RunOnce can be retried. The context can be cancelled
with several different errors, e.g. timeout. Ensure we stop the loop
when context has errored, because none of the queries will succeed
when it has failed.

Change-Id: If3ff11f11a6f43c0d67633be1cfaf23e3e9e55f3
2021-09-21 10:02:00 +00:00
crawter
5e4b196b11 satellite/metainfo: finish move object
Second method needed to perform server-side move. It updates
metadata key and nonce and all segments key and nonces.

Change-Id: Ia43b26622a13048269f0ae9e1524b345db112adb
2021-09-20 16:21:01 +03:00
crawter
07311cdd6f satellite/metainfo: begin move object added
First from two methods needed to perform server-side move. It gets
metadata key and nonce and all segments key and nonces and returns
all of that to uplink.

Change-Id: Ied2c79559e77d3f63091c4d61948f2d6a2147d67
2021-09-14 13:18:21 +00:00
Michał Niewrzał
84a383638b satellite/metabase: increase maximum batch size for loop
Current default is equal to hardcoded maximum of batch size
for loop listing. With this change we will bump maximum
batch size but current default won't change. We will be able to
increase it later without need for source code change.

Change-Id: I2744a87be28af4157f58ede73455682f61733bc1
2021-09-10 09:33:55 +00:00
Michał Niewrzał
67a33e4c06 satellite/metabase: drop unused IterateLoopStreams method
We were using this method for metainfo loop but now
segment loop is not using it so we can drop it.

Change-Id: I60c5b4f86a619259906d8c2ba76e665b8715be75
2021-09-09 14:12:44 +00:00
Michał Niewrzał
c258f4bbac private/testplanet: move Metabase outside Metainfo for satellite
At some point we moved metabase package outside Metainfo
but we didn't do that for satellite structure. This change
refactors only tests.
When uplink will be adjusted we can remove old entries in
Metainfo struct.

Change-Id: I2b66ed29f539b0ec0f490cad42c72840e0351bcb
2021-09-09 07:15:51 +00:00
Egon Elbre
707c4a7b0e satellite/metabase: use implicit tx when batch deleting
Change-Id: I2dc577b95209674b626a76633dedf5b5b1641db0
2021-08-30 15:04:35 +03:00
Michał Niewrzał
2340429ea8 satellite/metabase/zombiedeletion: add missing test case
Additional test case where user is uploading 3 segments
each within 23h interval and when zombie deletion process
is executed then nothing is deleted because last segment
was uploaded in less then 24h.

Change-Id: I7426d6fe2c7e9b88c054a01408910c986bcf8d5f
2021-08-27 08:57:06 +00:00
Michał Niewrzał
b7555980c4 satellite/metainfo: add zombiedeletion chore
This chore will be deleting zombie objects and it's segments.

Change-Id: Ia5d92dd6c3f52e178d4d43d9603732310761e027
2021-08-24 08:22:06 +00:00
Michał Niewrzał
0344790c20 satellite/metabase: delete zombie object that has no new segments for specific period of time
Added options flag to define after which object won't be marked as inactive. All segments CreatedAt
time needs to be bellow this flag to treat object as inactive.

Change-Id: Ib5cffc776c6ee1b62b51eb8595438f968b42528c
2021-08-23 08:25:38 +00:00
Artur M. Wolff
e452f85163 satellite/metabase: sync batchSizeLimit and ListLimit constants
This change syncs batchSizeLimit and ListLimit constants to prevent
throwing away results returned while listing with a maximum returns
limit.

Change-Id: Ie2425542d945cb88653dcc34c079737bb32320d4
2021-08-20 11:01:46 +00:00
Artur M. Wolff
a5371353bf satellite/metabase/segmentloop: fix rate.NewLimiter call
This change fixes the call to rate.NewLimiter to account for the
"infinite" case.

Change-Id: Ib3f914ca33ad8b981157fd224f077e6ad8d8c644
2021-08-19 17:23:13 +00:00
Michał Niewrzał
260a673327 satellite/metabase: don't delete pieces until query results are received
To optimize memory consumption we where consuming
segment data during processing results from delete
query. Turns out that there is a chance that query will be
rolled-back if something will go wrong while reading
results. In such case its possible to delete pices but
object/segment will be still in DB.
This change removed piece deletion from problematic
place. Pieces are still deleted in batches but are not
limited at the moment. To avoid memory issues object
deletion batch was decreased.

Change-Id: Icb3667220f9c25f64b73cf71d0cf3fdc7e5107c5
2021-08-08 03:43:17 +00:00
Clement Sam
6613133a1c satellite/metabase: return metadata from IterateObjectsAllVersionsWithStatus on demand
Change-Id: I6bb717cf941e2b5fd7c411bd1e5a77fad0835995
2021-08-06 16:20:39 +00:00
Clement Sam
1f353f3231 segment/{metabase,repair}: change segment created_at column to not accept nulls
This change adds a NOT NULL constraint to the created_at column in the segment table.
All occurrences of CreatedAt as a pointer are changed to non pointer version (metabase, segment loop, etc)

Change-Id: I3efd476ebd1edd3327b69c9223d9edc800e1cc52
2021-08-06 08:16:28 +00:00
Clement Sam
f06e7c5f60 segment/{metabase,repair}: add dedicated methods on metabase.Pieces
This change adds dedicated methods on metabase.Pieces to be able to add, remove pieces and also to check duplicates.

Change-Id: I21aaeff40c017c2ebe1cc85a864ae546754769cc
2021-08-03 15:12:03 +00:00
Michał Niewrzał
be92ff1751 satellite/metabase: fix finishing bucket objects deletion
In case when number of deleted segments in a single
batch was multiplication of opts.DeletePiecesBatchSize it
was possible to finish deletion of objects to early.

Change-Id: I9181d4a3c64053d9df46a11a6e0cd22153c73ee9
2021-08-02 09:00:02 +00:00
Michał Niewrzał
55d7bcc59b satellite/metabase/segmentloop: don't shutdown satellite on loop error
We made decision to avoid satellite shutdown when segment loop
will return error. Loop still can reeturn error but it will be logged
and we will make monitoring/alert around that error.

Change-Id: I6aa8e284406edf644a09d6b1fe00c3155c5430c9
2021-07-30 06:49:10 +00:00
Michał Niewrzał
ae345320fe satellite/metabase: use db methods that pass context correctly
It turns out that some DB methods are not
passing context to to lower levels even if have
context as a parameter.

Change-Id: I82f4fc1a47c362687a91364d85e4468057e53134
2021-07-29 09:32:31 +00:00
Michał Niewrzał
b12d29935a satellite/metabase: remove metaloop package
We moved everything to segment loop so we can now
remove metaloop from code.

Change-Id: I9bd8d2349e5638d7cdad50f2f313f9bd89a8165c
2021-07-22 13:00:45 +00:00
Egon Elbre
5a56021bca satellite/metabase: add intLimitRange
We have quite a bit code duplication in ensuring valid int ranges.

Change-Id: Ib31db8c4b2be6a677625cab5bcdddefb60401fe3
2021-07-19 07:51:16 +00:00
Kaloyan Raev
a767aed591
satellite/{metabase,metainfo}: require StreamID for UpdateObjectMetadata
This avoids corrupting objects if reuploads and metadata updates happen
concurrently.
2021-07-08 17:50:37 +03:00
Michał Niewrzał
2e9d3a737c satellite/metabase: fix listing prefixes with cursor set
We were not skipping the initial prefix from the cursor.

Change-Id: I2bb472e960b92cae77fd1226de0b26fac79c429b
2021-07-07 14:30:46 +00:00
Egon Elbre
74af44c12e satellite/metabase: make IteratePendingObjectsByKey recursive
Change-Id: I593164302559bec69393947e19820f8eef338830
2021-07-06 07:27:22 +00:00
Kaloyan Raev
a6086d421f satellite/metabase: implement SetObjectMetadataLatestVersion
It replaces UpdateMetadataObject that hasn't been used anywhere yet.

Change-Id: I1b2131acd75924c487a1c3ca099674251c5c9b56
2021-07-06 01:06:16 +00:00
Egon Elbre
c248651f3f satellite/metabase/{meta,segment}loop: fix test flakiness
When the delta is very small from the bounds then the ratio calculation
doesn't work that well. Let's allow 100 from the bounds, since that
would be expected in any case.

We won't add a configuration for it, since it's not that useful.

Change-Id: I049066a42470b825f430b7f32ebe92d544c6cc8b
2021-07-02 19:16:11 +03:00
Michał Niewrzał
4335b21332 satellite/metabase: add TotalEncryptedSize to LoopObjectEntry
We gonna need TotalEncryptedSize for buckets tally migrated to objects
iterator from metaloop.

Change-Id: I64774fdbda47abdaf5fbd370bd015b14bac9afdf
2021-06-30 16:43:31 +00:00
Michał Niewrzał
8ce619706b satellite/audit: migrate to new segment_pending_audit table
Currently, pending audit is finding segment by segment location
(path) because we want to move audit to segmentloop and we will
have only StreamID and Position we need to add columns for those
fields. Altering existing table can cause issues while
migration and deployment. Cleaner choise is to make new table.
This change contains migration with new segment_pending_audit
table that will replace pending_audits table and adjustments
to use new table in the code.

Table pending_audits will be dropped with next release.

Change-Id: Id507e29c152da594bac1fd812c78d7ecf45ec51f
2021-06-28 13:19:49 +02:00
Michał Niewrzał
1f5fbbd24a satellite/metabase: while deleting bucket objects delete pieces in batches
It's possible that single object will have a lot of segments and
at the moment we are trying to collect all pieces at once and
send to storage nodes information about deletion. Such
approach my lead to using extensive amount of memory. This
change is handling this problem by calling DeletePieces
function multiple times with only part of pieces to delete for
a single call.

Change-Id: Ie1e66cd9d86d130eb89a61cf6e23f38b8cb8859e
2021-06-25 13:27:00 +00:00
Egon Elbre
5f901c8066 satellite/metabase/metaloop: add as of system interval to stats
Table stats query should use AS OF SYSTEM INTERVAL in the query to optimize
the behavior.

Change-Id: Ie14f5441062d3d264bb3c626081e77d6edc7b724
2021-06-17 23:43:41 +03:00
Egon Elbre
9640cc2c06 satellite/metabase/segmentloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.segment-loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M segments is 3M segments.

Change-Id: I5ee03e913ddc4e67e94010ced126a2a9ea51f41b
2021-06-17 19:46:55 +03:00
Egon Elbre
341033dda1 satellite/metabase/metaloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M objects is 3M objects.

Change-Id: Ife5522ecc97bcc5a55667f36868a0f1fc8e4c561
2021-06-17 15:23:53 +03:00
Egon Elbre
0df4a27bf7 satellite/metabase: add method to get table statistics
In loop we need to start verifying it's correctness. This allows to
gather these stats.

Change-Id: I146fb50e2b3658b6f3c2682cdc1983e6abd73c29
2021-06-16 18:34:55 +03:00
Michał Niewrzał
1a974c327d satellite/metabase: set Segment.ExpiresAt while getting segment
Change-Id: I6504efd19c3516ffcf97ea3923e7f52e9832613b
2021-06-15 13:57:51 +00:00
JT Olio
6949dc0bac satellite/metaloop: missing monitoring on observers
Change-Id: I630fbb0448c8d08b426486b3e49abfbca03332a6
2021-06-15 13:39:13 +00:00
Michał Niewrzał
9a113da361 satellite/metabase: expose expires_at with loops
Loops are using custom structs to provide
segments while iterating/looping so we need
to expose new field there too.

Change-Id: I12c8f4a01afeac171bf638d278253999fa90a8cb
2021-06-11 16:51:37 +00:00
Egon Elbre
f3a52d1da5 satellite/metabase/segmentloop: limit max interval
Ensure that we don't query too far in the history, which slows things
down.

Change-Id: Ia77aa522f7f4c5d43629d51bb9a51a49fab6fa14
2021-06-10 17:06:33 +00:00
Michał Niewrzał
bc79f01aaa satellite/metabase: set expires_at while committing segment
We added expires_at column to segments table and now
we need to populate this column while committing segment.
We still need to migrate existing segments with
separate tool.

Change-Id: Ibac8c63d97201dd98cc2cb9db385f4cb73bc3f7e
2021-06-10 16:32:28 +00:00
Fadila Khadar
21731ff8d0 satellite/metabase: add expires_at column to segments
Change-Id: If798198d87edb68cda8859ce2277fdfbd6f8380b
2021-06-07 19:11:20 +00:00
Egon Elbre
347f5f87e0 satellite/metabase/metaloop: limit as of system time
Currently we did not limit the "as of system time" for iterating over
objects table. Using just an interval would cause problems with the
tests. That could be overcome skipping that interval for tests
altogether, however, we should probably test those more to ensure that
GC stays working as intended.

This is a safer code, however, maybe not as straigthforward as it could
be.

Change-Id: I374f77783b2af42bb6da846735ceea20a7ce5e60
2021-06-07 13:01:06 +00:00