This new method will be used with mechanism to limit number
of segments per project, similar to limiting buckets or bandwidth.
This is only one of multiple changes we will do to implement this
limitation.
Change-Id: Ia516c2a006ad1d7b4431d780679be9d809848f4b
We implemented filtering of system metadata to
reduce DB load in case user will need only list of
objects but we missed that encryption configuration
is needed to decrypt object key.
This change is always including encryption configuration into
list objects results.
Change-Id: Iaf7b3d8441480e3ad787f19a28f1b648584d5f7d
To resolve problem with lack of ability to set metadata while MPU on
gateway we are adding setting metadata with BeginObject. This
change makes also metadata optional while CommitObject. We need
this functionality to not override metadata set with BeginObject in
case when metadata is not set with CommitObject.
Another reson is that we would like to not set metadata at all if user
didn't specify metadata. At the moment we always setting some bytes
for metadata fields e.g. empty EncryptedMetadata field can have key
and nonce set.
Change-Id: Ifee25b7718eb1f919119db9b698b29d8b5ebe2ec
To reduce database load, add option to omit as many object properties
as possible when listing objects.
Change-Id: I817633801b00629a4042d1d1bd2389ee581953de
Turns out that S3 protocol is setting object metadata with initial
request, in our case it's BeginObject, so we need to modify metabase
methods to accept metadata at the beginning and at the end.
BeginObject methods will set metadata always (niled or not niled).
One additional improvment for metadata fields was introduced and it's
validating fields as optional if EncryptedMetadata was not set.
This change currently doesn't have any implications and it's a base for
other changes. Metadata is not set with BeginObject metainfo endpoint
yet.
Change-Id: I1f768407bc3428500b0d30ee188257420d953001
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB
Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
this is so that multipart upload allows overwriting parts. our
consistency model is more relaxed because part overwrites
won't be atomic if they consist of multiple segments, but at
least this allows overwriting at all.
Change-Id: I21dac4c24046e05efe74e6c6fd189a02c95eb6c8
This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.
This reverts commit 1ef06fae99.
Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.
Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
We were manually converting ObjectKey fields to []byte to use it with
SQL query but we can just implement Value method to convert it automatically.
Change-Id: I6d346f4b59718e1e8ef37cd9f95e613b864b42cd
We would like to verify if zombie object/segment works fine.
We need some metric for that. Figuring out number of deleted
objects is harder so let's for that later.
Change-Id: Ic99e2ce93256130b7c51f514824fddc009655075
Currently loops wait for the coalesce duration for TriggerWait.
Let's skip the coalesce when we trigger it manually.
Change-Id: If5bacd4e263d233f1f3ea41b989922d2ed5a48d4
server-side move extended with moving between buckets, for this reason
we change bucket name for object in db.
Change-Id: Ie21bcccc170e6ff14dcd8053fdb86fdf6d8438a0
Not all errors from RunOnce can be retried. The context can be cancelled
with several different errors, e.g. timeout. Ensure we stop the loop
when context has errored, because none of the queries will succeed
when it has failed.
Change-Id: If3ff11f11a6f43c0d67633be1cfaf23e3e9e55f3
Second method needed to perform server-side move. It updates
metadata key and nonce and all segments key and nonces.
Change-Id: Ia43b26622a13048269f0ae9e1524b345db112adb
First from two methods needed to perform server-side move. It gets
metadata key and nonce and all segments key and nonces and returns
all of that to uplink.
Change-Id: Ied2c79559e77d3f63091c4d61948f2d6a2147d67
Current default is equal to hardcoded maximum of batch size
for loop listing. With this change we will bump maximum
batch size but current default won't change. We will be able to
increase it later without need for source code change.
Change-Id: I2744a87be28af4157f58ede73455682f61733bc1
We were using this method for metainfo loop but now
segment loop is not using it so we can drop it.
Change-Id: I60c5b4f86a619259906d8c2ba76e665b8715be75
At some point we moved metabase package outside Metainfo
but we didn't do that for satellite structure. This change
refactors only tests.
When uplink will be adjusted we can remove old entries in
Metainfo struct.
Change-Id: I2b66ed29f539b0ec0f490cad42c72840e0351bcb
Additional test case where user is uploading 3 segments
each within 23h interval and when zombie deletion process
is executed then nothing is deleted because last segment
was uploaded in less then 24h.
Change-Id: I7426d6fe2c7e9b88c054a01408910c986bcf8d5f
Added options flag to define after which object won't be marked as inactive. All segments CreatedAt
time needs to be bellow this flag to treat object as inactive.
Change-Id: Ib5cffc776c6ee1b62b51eb8595438f968b42528c
This change syncs batchSizeLimit and ListLimit constants to prevent
throwing away results returned while listing with a maximum returns
limit.
Change-Id: Ie2425542d945cb88653dcc34c079737bb32320d4
To optimize memory consumption we where consuming
segment data during processing results from delete
query. Turns out that there is a chance that query will be
rolled-back if something will go wrong while reading
results. In such case its possible to delete pices but
object/segment will be still in DB.
This change removed piece deletion from problematic
place. Pieces are still deleted in batches but are not
limited at the moment. To avoid memory issues object
deletion batch was decreased.
Change-Id: Icb3667220f9c25f64b73cf71d0cf3fdc7e5107c5
This change adds a NOT NULL constraint to the created_at column in the segment table.
All occurrences of CreatedAt as a pointer are changed to non pointer version (metabase, segment loop, etc)
Change-Id: I3efd476ebd1edd3327b69c9223d9edc800e1cc52
This change adds dedicated methods on metabase.Pieces to be able to add, remove pieces and also to check duplicates.
Change-Id: I21aaeff40c017c2ebe1cc85a864ae546754769cc
In case when number of deleted segments in a single
batch was multiplication of opts.DeletePiecesBatchSize it
was possible to finish deletion of objects to early.
Change-Id: I9181d4a3c64053d9df46a11a6e0cd22153c73ee9
We made decision to avoid satellite shutdown when segment loop
will return error. Loop still can reeturn error but it will be logged
and we will make monitoring/alert around that error.
Change-Id: I6aa8e284406edf644a09d6b1fe00c3155c5430c9
It turns out that some DB methods are not
passing context to to lower levels even if have
context as a parameter.
Change-Id: I82f4fc1a47c362687a91364d85e4468057e53134
When the delta is very small from the bounds then the ratio calculation
doesn't work that well. Let's allow 100 from the bounds, since that
would be expected in any case.
We won't add a configuration for it, since it's not that useful.
Change-Id: I049066a42470b825f430b7f32ebe92d544c6cc8b
Currently, pending audit is finding segment by segment location
(path) because we want to move audit to segmentloop and we will
have only StreamID and Position we need to add columns for those
fields. Altering existing table can cause issues while
migration and deployment. Cleaner choise is to make new table.
This change contains migration with new segment_pending_audit
table that will replace pending_audits table and adjustments
to use new table in the code.
Table pending_audits will be dropped with next release.
Change-Id: Id507e29c152da594bac1fd812c78d7ecf45ec51f
It's possible that single object will have a lot of segments and
at the moment we are trying to collect all pieces at once and
send to storage nodes information about deletion. Such
approach my lead to using extensive amount of memory. This
change is handling this problem by calling DeletePieces
function multiple times with only part of pieces to delete for
a single call.
Change-Id: Ie1e66cd9d86d130eb89a61cf6e23f38b8cb8859e
This adds verification for the processed count and before and after
segment/objects table counts.
This adds new flag:
metainfo.segment-loop.suspicious-processed-ratio: 0.03
This defaults to 3%, which at 100M segments is 3M segments.
Change-Id: I5ee03e913ddc4e67e94010ced126a2a9ea51f41b
