This condenses the precommit constraint into a single function, which
allows to cleanup a bunch of logic elsewhere. Also, avoid delete in the
first place when we are not allowed to remove the uncommited object.
This also fixes submitting the metrics and now it submits the correct
metrics.
Change-Id: If91dfa3b19bce5b24ff2a19d7c34b57a200db1dd
There are several different object types in a versioned table,
which will determine the exact behaviour.
The object type states are:
* Pending - the object is yet to be committed and is being uploaded.
* Committed - the object has been finished and can be read.
* DeleteMarker - indicates that the object should be treated as not
present when is at the top of the version stack.
There are also versioning states:
* Unversioned - only one unversioned object is allowed per object key.
* Versioned - multiple objects with the same key are allowed.
Change-Id: I65dfa781e8da253a4e5d572b799d53c351196eee
With this change we are removing code responsible for deleting objects
and supporting server side copies created with references. In practice
we are restoring delete queries that we had before server side copy
implementation (with small exception, see bellow).
From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.
https://github.com/storj/storj/issues/5891
Change-Id: Iee4e7a9688cff27a60fb95d60dd233d996f41c85
..instead of using segment_copies and ancestor_stream_id, etc.
This bypasses reference counting entirely, depending on our mark+sweep+
bloomfilter garbage collection strategy to get rid of pieces once they
are no longer part of a segment.
This is only safe to do after we have stopped passing delete requests on
to storage nodes.
Refs: https://github.com/storj/storj/issues/5889
Change-Id: I37bdcffaa752f84fd85045235d6875b3526b5ecc
In case some invalid characters in bucket name we need to cast
bucket name to byte array for query argument. This change is
doing this for some missed cases.
Change-Id: I47d0d8e3c85a69bdf63de1137adcd533dcfe50a8
This flag was in general one time switch to enable versions internally.
New we can remove it as it makes code more complex.
Change-Id: I740b6e8fae80d5fac51d9425793b02678357490e
When doing server-side copy, deletes the committed version of the target location if it already exists. It does not touch pending versions. The version of the copy is set to the highest already existing version + 1.
Fixes: https://github.com/storj/storj/issues/5071
Change-Id: I1d91ac17054834b1f4f0970a9fa5d58198c58a37
If source and destination are the same, do nothing and return the original object.
Later, we will have to handle the case when metadata is changed, but for now it's not possible through libuplink.
An issue has been created for this (https://github.com/storj/storj/issues/5168)
Change-Id: I91cf48afeec498d3b2c219fa0d3baf2163cff384
We are preparing to use object versions internally and to do
that we need to prepare different parts of the system to handle
object versions different than '1'. This change adjust code
responsible for server-side move and copy.
What was done:
* begin methods for move and copy are now using GetObjectLastCommitted
to find object
* results from begin move and copy operation contains now version to
be able to map object correctly with finish operation
* begin methods are putting version into satellite stream id and
finish methods are using this version as parameter instead hardcoded
value
Fixes https://github.com/storj/storj/issues/4867
Change-Id: I1380911279c21e10a3fff0342793efd2e73eafad
Previously there was no realtime administration of the storage usage
during copies. Now there is.
Closes https://github.com/storj/storj/issues/4719
Change-Id: I0d536bf551d16208116c3aceac89ed590ec473bf
An object copy/move is done by 2 DRPC calls. It's possible a new object was uploaded ore moved to the source location between these calls. For copy, in that case the segments end up with the wrong keys. This change adds an explicit check for that by comparing the streamId supplied by the user with the streamId in the database.
Fixes https://github.com/storj/storj/issues/4930
Change-Id: Id600456ce78fb4069b93644828a0b3eb85e23e16
Previously copying an object to it's ancestor location (copy of copy)
broke the object and all copies.
This fixes this by calling the existing delete method rather than a
custom one when there is an existing object at the copy destination.
The check for existing object at destination has been moved to an
earlier point in FinishCopy.
metabase.DeleteObject exposes a transaction parameter so that it can be
reused within metabase.
Closes https://github.com/storj/storj/issues/4707
Uplink test at https://review.dev.storj.io/c/storj/uplink/+/7557
Change-Id: I418fc3337fa9f30146ccc1db456af168ae41c326
- instead of closing over the outer err variable, potentially
overwriting some errors or something, declare local variables.
- double check that we got the number of rows we expected to get
and error otherwise. this prevents a possible source of inserting
bogus rows into the database.
Change-Id: I30662be2727afe0a90e4215a182fedc2648d1169
Read the source object and write the destination object in the same
transaction, to prevent breaking the object because it was deleted
simultaneously.
This is probably the root cause of the metainfo loop halting from
2022-06-21 onwards, where 2 objects lost their root_piece_id during
copying.
Part of https://github.com/storj/storj/issues/4930
Change-Id: I9c45d56a7bfb48ecd5f4906ee1cca42922901e90
s3 allows for overwriting an object when using server-side copy.
This change makes overwriting the destination part of the atomic server-side copy operation so that
if copy fails, the old object is still available.
All the segments of the existing destination are deleted. If this destination object is an ancestor of another object, a new ancestor is promoted.
Fixes https://github.com/storj/storj/issues/4607
Change-Id: I85d1250850bb71867586ac230c8275a0cc1b63c3
Uplink is fixed and now we should always get both key and nonce
or both empty.
Fixes https://github.com/storj/storj/issues/4646
Change-Id: I65dca2d4d5a10787c2fecad39e301121f1ae242a
So far we assumes that metadata key/nonce cannot be empty at all
but at some point we adjusted code to accept empty metadata/key/nonce
to save DB space.
This change is adjusting how we are processing nonce while
FinishMoveObject/FinishCopyObject. We can use storj.Nonce directly
which makes code cleaner. It's also fixing issue in FinishMoveObject
where we didn't convert nonce correctly to []byte.
Part of change is disabling validation for key and nonce until
uplink will be adjusted. We need change uplink to send always
both key and nonce or non of them. Validation will be restored
as soon as change for uplink will be merged.
https://github.com/storj/storj/issues/4646
Change-Id: Ia1772bc430ae591f54c6a9ae0308a4968aa30bed
If B is a copy of A, and C is a copy of B, then in the segment_copies table, it should appear that C is a copy of A.
Fixes https://github.com/storj/storj/issues/4538
Change-Id: I7e6b03f7584597cf616cd1e0cd0156386771d207
In the server-side copy initial implementation, we are inserting segments one by one. This PR inserts them all at once.
Fixes https://github.com/storj/storj/issues/4476
Change-Id: I776dba99be38a0eef73366e8e9287cbb794003dc
We were not checking if we were provided an empty StreamID.
Furthermore, this changes returns the object copy with the correct createdAt field.
Change-Id: Iefc563c34ae9d8c1e233895155c1718bf905df91
Copy object functionality should support setting new metadata for
copy. This change is adjusting FinishCopyObject method to set new
metadata when OverrideMetadata field is set to true.
Fixes https://github.com/storj/storj/issues/4483
Change-Id: Ica37cb57e8edae301cdc483fbda4f3ddba5d2702
Updates metadata and metainfo to return object metadata with
FinishCopyObject request.
https://github.com/storj/storj/issues/4474
Change-Id: I32cba5c20a943272e9b5964df1b3d6463ad212dc
Part of server-side copy implementation.
Creates the methods BeginCopyObject and FinishCopyObject in satellite/metabase/copy_object.go.
This commit makes it possible to copy objects and their segments in metabase.
https://github.com/storj/team-metainfo/issues/79
Change-Id: Icc64afce16e5e0e15b83c43cff36797bb9bd39bc
