We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.
Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
We are preparing to use object versions internally and to do
that we need to prepare different parts of the system to handle
object versions different than '1'. This change adjust code
responsible for server-side move and copy.
What was done:
* begin methods for move and copy are now using GetObjectLastCommitted
to find object
* results from begin move and copy operation contains now version to
be able to map object correctly with finish operation
* begin methods are putting version into satellite stream id and
finish methods are using this version as parameter instead hardcoded
value
Fixes https://github.com/storj/storj/issues/4867
Change-Id: I1380911279c21e10a3fff0342793efd2e73eafad
Metadata validation for CommitObject request was placed in a wrong
place. There is a case (old uplink) where encrypted key is bundled
inside encrypted metadata bytes and we need to extract it before we
can validate it. This change moves metadata validation to a place where
we are sure we have encrypted metadata and encrypted metadata encrypted
key ready to be checked.
"Run Versions Test" is covering this case and it was failing without
this change.
Change-Id: Ib709ad901fbb3fa4865a393195b7b3f4c0d87e7a
For object Version in different places we are using different types.
Satellite StreamID is using int32 but metabase accepts int64. Metabase
type is correct one and we should align other places with it.
As a small addition this change is also passing version correctly
between requests instead of using hardcoded value.
Change-Id: I63634d73c0a48c009e4db5f203ff18b7f3218b02
Updated metabase.UpdateObjectMetadata method to update set metdata always for last committed object
Closes https://github.com/storj/storj/issues/4870
Change-Id: I060683e31efcaf3e2531fea143cf0567e5ff5f73
Restored GetObjectLatestVersion and renamed it to GetObjectLastCommitted
Add test cases to cover server-side copy
Closes https://github.com/storj/storj/issues/4866
Change-Id: I343b339a60152b8fb92fda97baf80bd8fe60d631
We log metainfo object operations and it looks that the log's message
convention is `Object {operation}`, however the `Object Download`
operation didn't match with the actual operation and the one that was
representing it had was `Download Object`.
This commit changes the log's message for the download object operation
according to the other object operations log messages format and fixes
the log message for the Get Object operation.
For finding this I executed the following command at the root of the
repository to obtain the list of lines where we log object operations.
$> ag 'log\.Info\(".*Object.*",' --no-color git:(main)
satellite/metainfo/endpoint_object.go
179: endpoint.log.Info("Object Upload", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "put"), zap.String("type", "object"))
336: endpoint.log.Info("Object Download", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "get"), zap.String("type", "object"))
557: endpoint.log.Info("Download Object", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "download"), zap.String("type", "object"))
791: endpoint.log.Info("Object List", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "list"), zap.String("type", "object"))
979: endpoint.log.Info("Object Delete", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "delete"), zap.String("type", "object"))
`ag` is a command-line tool similar to `grep`
Change-Id: I9072c5967eb42c397a2c64761d843675dd4991ec
Previously there was no realtime administration of the storage usage
during copies. Now there is.
Closes https://github.com/storj/storj/issues/4719
Change-Id: I0d536bf551d16208116c3aceac89ed590ec473bf
If project.UserAgent is set, use this for bucket.UserAgent on bucket
creation. Otherwise, set bucket attribution as before (getting UserAgent
from request headers).
Tests were updated to create the bucket with a different user, added as
a project member. Otherwise, the tests do not catch the bug.
Change-Id: I7ecf79a8eac5957eed361cbea94823190f58b776
We can use PieceIDDeriver in all places where we are deriving id from
the same id multiple times. We have serveral such places: gc, segment
deletion, segment validation, order limit creation. Using it should
save some resources.
Change-Id: I24668d516c0f7cea4aec6470614067734149501d
The existing versionCollector metrics can tell us how many times various
metainfo endpoints are called, but they don't tell us how many bytes a
client is transferring. We currently can't collect precise information
on this, but we can collect information on how much planned traffic is
requested via order limits.
The implementation as provided is intended to measure objects sizes
before erasure encoding is taken into account.
Change-Id: I2f1d2a7831630e8439ecf5342e933df259151792
Version collector previously returned errors and logged them in the
calling code. It is cleaner to log inside version collector.
Change-Id: I52cb49a1ef53f3f1f51692ddb26ec095cfd0f100
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.
https://github.com/storj/team-metainfo/issues/105
Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
This change has two purposes. First is to avoid DB call in case
source and destination bucket are the same.
Second is to return bucket not found error in correct order. If
source and destination bucket are different we will first check
source and later destination. Currently we will get first error
about not existing destination bucket.
Because of this change we stop putting bucket placement
into satellite stream id but its not needed as we don't use
this value with finish move/copy object methods.
Change-Id: I0f7b3ba604d53c722e8fa4d7a37843a69d02bebd
So far we assumes that metadata key/nonce cannot be empty at all
but at some point we adjusted code to accept empty metadata/key/nonce
to save DB space.
This change is adjusting how we are processing nonce while
FinishMoveObject/FinishCopyObject. We can use storj.Nonce directly
which makes code cleaner. It's also fixing issue in FinishMoveObject
where we didn't convert nonce correctly to []byte.
Part of change is disabling validation for key and nonce until
uplink will be adjusted. We need change uplink to send always
both key and nonce or non of them. Validation will be restored
as soon as change for uplink will be merged.
https://github.com/storj/storj/issues/4646
Change-Id: Ia1772bc430ae591f54c6a9ae0308a4968aa30bed
For server-side copy we adjusted one method DeleteObjectExactVersion.
Other deletion methods won't be used directly in code at the moment.
We will adjust other methods later or decide if we will need them at
all.
To handle deletion of objects with copies or just copies correctly we
need to use DeleteObjectExactVersion method in two places while:
* removing object before upload
* explicit object deletion
This change is also changing DeleteObjectExactVersion method to
delete pending objects because we need this functionality to
delete object before new upload.
https://github.com/storj/storj/issues/4481
Change-Id: Ieff5cc95732bb70ed8cc0ecdd62e03c929857c02
Copy object functionality should support setting new metadata for
copy. This change is adjusting FinishCopyObject method to set new
metadata when OverrideMetadata field is set to true.
Fixes https://github.com/storj/storj/issues/4483
Change-Id: Ica37cb57e8edae301cdc483fbda4f3ddba5d2702
Updates metadata and metainfo to return object metadata with
FinishCopyObject request.
https://github.com/storj/storj/issues/4474
Change-Id: I32cba5c20a943272e9b5964df1b3d6463ad212dc
We don't have different version of object than 1 so at the moment
this method is not needed. Also using GetObjectExactVersion
should be slightly more performant.
Change-Id: I78235d8ae22594cc1d6345dabcc915f41cd7797b
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.
Tests will be split as a separate set of changes.
Updates https://github.com/storj/team-metainfo/issues/12
Change-Id: I6c691e4d0e192fe3ad7974d2d0ab5ced0d272f3c
