Commit Graph

21 Commits

Author SHA1 Message Date
Michal Niewrzal
4d9c9138ce satellite/metainfo: use multiple object versions internally
With this change we are switching methods to begin object, from
BeginObjectExactVersion to BeginObjectNextVersion. Main implication
is that from now it will be possible to have object with version
different than 1. New object will always get first available version.

Main reason to do this it to avoid deleting existing object during
reuploading object. Now we can create multiple pending objects but
only last committed will be available to the user. Any previous
committed object will be deleted.Because of that we moved logic to
delete existing object from BeginObject to CommitoObject request.

New logic is behind feature flat to be able to test it well first
before enablng on production.

Fixes https://github.com/storj/storj/issues/4871

Change-Id: I2dd9c7364fd93796a05ef607bda9c39a741e6a89
2022-10-06 15:19:02 +00:00
Fadila Khadar
f58129786b satellite/metabase: copy handles version when overwriting target location
When doing server-side copy, deletes the committed version of the target location if it already exists. It does not touch pending versions. The version of the copy is set to the highest already existing version + 1.

Fixes: https://github.com/storj/storj/issues/5071

Change-Id: I1d91ac17054834b1f4f0970a9fa5d58198c58a37
2022-09-28 11:39:24 +00:00
Fadila Khadar
601cadb92c satellite/metabase: copy when source and destination are the same
If source and destination are the same, do nothing and return the original object.
Later, we will have to handle the case when metadata is changed, but for now it's not possible through libuplink.

An issue has been created for this (https://github.com/storj/storj/issues/5168)

Change-Id: I91cf48afeec498d3b2c219fa0d3baf2163cff384
2022-09-13 19:21:13 +00:00
Michal Niewrzal
fa76dfe577 satellite/metabase: use GetObjectLastCommitted with move/copy
We are preparing to use object versions internally and to do
that we need to prepare different parts of the system to handle
object versions different than '1'. This change adjust code
responsible for server-side move and copy.

What was done:
* begin methods for move and copy are now using GetObjectLastCommitted
to find object
* results from begin move and copy operation contains now version to
be able to map object correctly with finish operation
* begin methods are putting version into satellite stream id and
finish methods are using this version as parameter instead hardcoded
value

Fixes https://github.com/storj/storj/issues/4867

Change-Id: I1380911279c21e10a3fff0342793efd2e73eafad
2022-08-30 06:36:02 +00:00
Erik van Velzen
87e1528104 satellite/metabase: error if source object changed
An object copy/move is done by 2 DRPC calls. It's possible a new object was uploaded ore moved to the source location between these calls. For copy, in that case the segments end up with the wrong keys. This change adds an explicit check for that by comparing the streamId supplied by the user with the streamId in the database.

Fixes https://github.com/storj/storj/issues/4930

Change-Id: Id600456ce78fb4069b93644828a0b3eb85e23e16
2022-07-19 13:27:55 +02:00
Erik van Velzen
74f4f6e765 satellite/metabase: fix copy to ancestor location
Previously copying an object to it's ancestor location (copy of copy)
broke the object and all copies.

This fixes this by calling the existing delete method rather than a
custom one when there is an existing object at the copy destination.

The check for existing object at destination has been moved to an
earlier point in FinishCopy.

metabase.DeleteObject exposes a transaction parameter so that it can be
reused within metabase.

Closes https://github.com/storj/storj/issues/4707

Uplink test at https://review.dev.storj.io/c/storj/uplink/+/7557

Change-Id: I418fc3337fa9f30146ccc1db456af168ae41c326
2022-06-28 03:53:01 +02:00
Qweder93
04b22f24eb satellite/metabase: copy expires_at to segments while FinishCopyObject
Resolves: https://github.com/storj/storj/issues/4726

Change-Id: I0bb222b7d5d3c66fe96c78a8aa1ea8f3b0a33f2b
2022-04-27 18:39:54 +00:00
Michał Niewrzał
6e5a94698e satellite/metabase: add option to override metadata with CommitObject
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.

https://github.com/storj/team-metainfo/issues/105

Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
2022-04-26 08:03:52 +00:00
Fadila Khadar
3d9329a5b1 satellite/metabase: overwrite existing destination on copy
s3 allows for overwriting an object when using server-side copy.
This change makes overwriting the destination part of the atomic server-side copy operation so that
if copy fails, the old object is still available.

All the segments of the existing destination are deleted. If this destination object is an ancestor of another object, a new ancestor is promoted.

Fixes https://github.com/storj/storj/issues/4607

Change-Id: I85d1250850bb71867586ac230c8275a0cc1b63c3
2022-04-07 06:28:51 +00:00
Michał Niewrzał
2014e4655a satellite/metabase: bring back copy/move validation for key and nonce
Uplink is fixed and now we should always get both key and nonce
or both empty.

Fixes https://github.com/storj/storj/issues/4646

Change-Id: I65dca2d4d5a10787c2fecad39e301121f1ae242a
2022-04-01 07:23:22 +00:00
Michał Niewrzał
4da3dc8170 satellite/metabase: improve processing of key/nonce while finish copy/move
So far we assumes that metadata key/nonce cannot be empty at all
but at some point we adjusted code to accept empty metadata/key/nonce
to save DB space.

This change is adjusting how we are processing nonce while
FinishMoveObject/FinishCopyObject. We can use storj.Nonce directly
which makes code cleaner. It's also fixing issue in FinishMoveObject
where we didn't convert nonce correctly to []byte.

Part of change is disabling validation for key and nonce until
uplink will be adjusted. We need change uplink to send always
both key and nonce or non of them. Validation will be restored
as soon as change for uplink will be merged.

https://github.com/storj/storj/issues/4646

Change-Id: Ia1772bc430ae591f54c6a9ae0308a4968aa30bed
2022-03-28 14:34:21 +00:00
Fadila Khadar
cb37971ce4 satellite/metabase: copy of inline segments only object not in segment_copies
Fixes  https://github.com/storj/storj/issues/4561

Change-Id: I8db0f984cc94ac9b95c1d746310a3b1d49728c7a
2022-03-23 22:41:06 +01:00
Fadila Khadar
f6b4d522be satellite/metabase: flatten copy references when copying
If B is a copy of A, and C is a copy of B, then in the segment_copies table, it should appear that C is a copy of A.

Fixes https://github.com/storj/storj/issues/4538

Change-Id: I7e6b03f7584597cf616cd1e0cd0156386771d207
2022-03-20 10:29:37 +00:00
Fadila Khadar
93f8c6a159 satellite/metabase: require non-zero newStreamID for FinishCopyObject
We were not checking if we were provided an empty StreamID.
Furthermore, this changes returns the object copy with the correct createdAt field.

Change-Id: Iefc563c34ae9d8c1e233895155c1718bf905df91
2022-03-16 15:47:52 +00:00
Michał Niewrzał
ccd16bbef6 satellite/metabase: handle NewEncryptedMetadata while coping object
Copy object functionality should support setting new metadata for
copy. This change is adjusting FinishCopyObject method to set new
metadata when OverrideMetadata field is set to true.

Fixes https://github.com/storj/storj/issues/4483

Change-Id: Ica37cb57e8edae301cdc483fbda4f3ddba5d2702
2022-03-09 12:31:33 +01:00
Erik van Velzen
56d1b8bbca satellite/metabasetest: improve test copy creation
Return the newly created segments separately so we can make better
assertion to test deletion.

Change-Id: I33414df4455f87db62d486957425d4984b66c0b2
2022-03-03 13:36:52 +01:00
Erik van Velzen
7a01a2a134 satellite/metabase/metabasetest/create: return segments
Return segments when creating a test object so that it can be checked
that the correct segments are remaining after a delete action.

Change-Id: Ifc245948935ba278806e887672c03abc5f2c2654
2022-02-28 13:40:46 +00:00
Michał Niewrzał
c0297bae78 satellite/metabase: return object metadata with FinishCopyObject
Updates metadata and metainfo to return object metadata with
FinishCopyObject request.

https://github.com/storj/storj/issues/4474

Change-Id: I32cba5c20a943272e9b5964df1b3d6463ad212dc
2022-02-25 12:46:30 +00:00
Fadila Khadar
332e6afed8 satellite/metabasetest: order RawSegments by streamID and position
Change-Id: I669c0c2361c2c7989c8e946eb1fbf2f52074ce0e
2022-02-25 11:19:11 +00:00
Fadila Khadar
5c7c4fedde satellite/metabasetest: include copies in raw state.
It can be useful to compare object copies created during unit tests.
They appear in the segment_copies table as couple (stream_id, ancestor_stream_id).

Change-Id: Id335c3ff7084fe30346456d27e670aff329154ea
2022-02-18 12:48:42 +00:00
Fadila Khadar
0358a08d6b satellite/metabase: BeginCopyObject and FinishCopyObject
Part of server-side copy implementation.

Creates the methods BeginCopyObject and FinishCopyObject in satellite/metabase/copy_object.go.

This commit makes it possible to copy objects and their segments in metabase.

https://github.com/storj/team-metainfo/issues/79

Change-Id: Icc64afce16e5e0e15b83c43cff36797bb9bd39bc
2022-02-17 13:37:39 +01:00