Commit Graph

31 Commits

Author SHA1 Message Date
Michal Niewrzal
4d9c9138ce satellite/metainfo: use multiple object versions internally
With this change we are switching methods to begin object, from
BeginObjectExactVersion to BeginObjectNextVersion. Main implication
is that from now it will be possible to have object with version
different than 1. New object will always get first available version.

Main reason to do this it to avoid deleting existing object during
reuploading object. Now we can create multiple pending objects but
only last committed will be available to the user. Any previous
committed object will be deleted.Because of that we moved logic to
delete existing object from BeginObject to CommitoObject request.

New logic is behind feature flat to be able to test it well first
before enablng on production.

Fixes https://github.com/storj/storj/issues/4871

Change-Id: I2dd9c7364fd93796a05ef607bda9c39a741e6a89
2022-10-06 15:19:02 +00:00
Fadila Khadar
02924d0ded satellite/metainfo: temporary feature flag for listing query testing
Fixes: https://github.com/storj/storj/issues/5144

Change-Id: I7650f4d5dd0378e2246339e79710a695996a845c
2022-09-28 17:46:52 +00:00
Qweder93
c210776a36 satellite/{metabase, metainfo}: merge logic for BeginCopyObject and BeginMoveObject
Closes https://github.com/storj/storj/issues/4514

Change-Id: I4d59fe9e3d30f67ce642d6266b878956e38f5db1
2022-09-22 09:02:20 +00:00
Michal Niewrzal
4e71dd302a satellite/{metainfo,metabase}: prepare feature flag MultipleVersions
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.

Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
2022-09-21 13:07:13 +00:00
Qweder93
98fb4c4f32 satellite/metainfo: added test for object TTL validation while upload
Closes https://github.com/storj/team-metainfo/issues/113

Change-Id: I6ff2a5f2cdab6fd42d94fbfc335084b927d86bea
2022-09-08 11:39:41 +03:00
Qweder93
2a1bcfed81 {satellite/metabase, satellite/metainfo, satellite/inspector} : Use metabase.GetObjectLastCommitted instead metabase.GetObjectExactVersion
adjusted metainfo, metabase and inspector methods to use GetObjectLastCommitted,
added test cases to confirm that this method will work correctly with version higher then 1

Closes https://github.com/storj/storj/issues/4869
Closes https://github.com/storj/storj/issues/4873

Change-Id: I8c338dcd1db82c141383f41339a295d54e2fa039
2022-09-08 07:27:22 +00:00
Michal Niewrzal
fa76dfe577 satellite/metabase: use GetObjectLastCommitted with move/copy
We are preparing to use object versions internally and to do
that we need to prepare different parts of the system to handle
object versions different than '1'. This change adjust code
responsible for server-side move and copy.

What was done:
* begin methods for move and copy are now using GetObjectLastCommitted
to find object
* results from begin move and copy operation contains now version to
be able to map object correctly with finish operation
* begin methods are putting version into satellite stream id and
finish methods are using this version as parameter instead hardcoded
value

Fixes https://github.com/storj/storj/issues/4867

Change-Id: I1380911279c21e10a3fff0342793efd2e73eafad
2022-08-30 06:36:02 +00:00
Sean Harvey
5a5f609022
satellite/metainfo: add copy and move log and metrics
Change-Id: I579c03c389dea7f8e7e92ade5bcac53740ce69c8
2022-08-26 18:55:29 +12:00
Michal Niewrzal
d6054214c3 satellite/metainfo: move metadata validation to a proper place
Metadata validation for CommitObject request was placed in a wrong
place. There is a case (old uplink) where encrypted key is bundled
inside encrypted metadata bytes and we need to extract it before we
can validate it. This change moves metadata validation to a place where
we are sure we have encrypted metadata and encrypted metadata encrypted
key ready to be checked.

"Run Versions Test" is covering this case and it was failing without
this change.

Change-Id: Ib709ad901fbb3fa4865a393195b7b3f4c0d87e7a
2022-08-24 10:50:43 +00:00
Michał Niewrzał
ea38860aab satellite/metainfo: align satellite StreamID protobuf version with metabase
For object Version in different places we are using different types.
Satellite StreamID is using int32 but metabase accepts int64. Metabase
type is correct one and we should align other places with it.

As a small addition this change is also passing version correctly
between requests instead of using hardcoded value.

Change-Id: I63634d73c0a48c009e4db5f203ff18b7f3218b02
2022-08-24 10:17:34 +00:00
Qweder93
2139d3966b satellite/metabase: update UpdateObjectMetadata to work with latest commited object
Updated metabase.UpdateObjectMetadata method to update set metdata always for last committed object

Closes https://github.com/storj/storj/issues/4870

Change-Id: I060683e31efcaf3e2531fea143cf0567e5ff5f73
2022-08-24 10:28:57 +03:00
Michał Niewrzał
b722c29e77 satellite/metainfo: add missing metadata validation
We are missing metadata validation for UpdateObjectMetadata and
FinishCopyOject requests.

Change-Id: Idca6a4d1fe108e1593405fd3913442f5b69d09e7
2022-08-23 16:56:45 +02:00
Qweder93
2abe709b6e satellite/metabase: add metabase.GetObjectLatestVersion back
Restored GetObjectLatestVersion and renamed it to GetObjectLastCommitted
Add test cases to cover server-side copy

Closes https://github.com/storj/storj/issues/4866

Change-Id: I343b339a60152b8fb92fda97baf80bd8fe60d631
2022-08-11 22:14:38 +00:00
Ivan Fraixedes
c5abc0de6f satellite/metainfo: Homogenize object operations logs
We log metainfo object operations and it looks that the log's message
convention is `Object {operation}`, however the `Object Download`
operation didn't match with the actual operation and the one that was
representing it had was `Download Object`.

This commit changes the log's message for the download object operation
according to the other object operations log messages format and fixes
the log message for the Get Object operation.

For finding this I executed the following command at the root of the
repository to obtain the list of lines where we log object operations.

   $> ag 'log\.Info\(".*Object.*",' --no-color                                                                                                                                                                                                                                       git:(main)
   satellite/metainfo/endpoint_object.go
   179:    endpoint.log.Info("Object Upload", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "put"), zap.String("type", "object"))
   336:    endpoint.log.Info("Object Download", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "get"), zap.String("type", "object"))
   557:    endpoint.log.Info("Download Object", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "download"), zap.String("type", "object"))
   791:    endpoint.log.Info("Object List", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "list"), zap.String("type", "object"))
   979:    endpoint.log.Info("Object Delete", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "delete"), zap.String("type", "object"))

`ag` is a command-line tool similar to `grep`

Change-Id: I9072c5967eb42c397a2c64761d843675dd4991ec
2022-08-05 23:11:39 +00:00
Erik van Velzen
b5fc04af89 satellite/metainfo: usage limits for copy
Previously there was no realtime administration of the storage usage
during copies. Now there is.

Closes https://github.com/storj/storj/issues/4719

Change-Id: I0d536bf551d16208116c3aceac89ed590ec473bf
2022-07-25 12:35:27 +02:00
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Moby von Briesen
5b79970575 satellite/metainfo: Use project-level attribution at bucket level
If project.UserAgent is set, use this for bucket.UserAgent on bucket
creation. Otherwise, set bucket attribution as before (getting UserAgent
from request headers).

Tests were updated to create the bucket with a different user, added as
a project member. Otherwise, the tests do not catch the bug.

Change-Id: I7ecf79a8eac5957eed361cbea94823190f58b776
2022-06-07 17:08:50 +00:00
Michał Niewrzał
456aea727e satellite: use PieceIDDeriver for derivation
We can use PieceIDDeriver in all places where we are deriving id from
the same id multiple times. We have serveral such places: gc, segment
deletion, segment validation, order limit creation. Using it should
save some resources.

Change-Id: I24668d516c0f7cea4aec6470614067734149501d
2022-05-19 06:31:42 +00:00
Michał Niewrzał
3ebb05ecca satellite/metainfo: add user-agent bandwidth metric
The existing versionCollector metrics can tell us how many times various
metainfo endpoints are called, but they don't tell us how many bytes a
client is transferring.  We currently can't collect precise information
on this, but we can collect information on how much planned traffic is
requested via order limits.

The implementation as provided is intended to measure objects sizes
before erasure encoding is taken into account.

Change-Id: I2f1d2a7831630e8439ecf5342e933df259151792
2022-05-17 09:57:39 +00:00
Michał Niewrzał
db5d781735 satellite/metainfo: simplify version collector logging
Version collector previously returned errors and logged them in the
calling code.  It is cleaner to log inside version collector.

Change-Id: I52cb49a1ef53f3f1f51692ddb26ec095cfd0f100
2022-04-26 10:16:20 +00:00
Michał Niewrzał
e66beb2429 satellite/metainfo: set metadata with BeginObject
This change makes possible to set metadata at the beginning
of upload.

https://github.com/storj/team-metainfo/issues/105

Change-Id: If5b6b90243d9c9318a8b72c2af3fbde400c5d8e7
2022-04-26 08:04:27 +00:00
Michał Niewrzał
6e5a94698e satellite/metabase: add option to override metadata with CommitObject
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.

https://github.com/storj/team-metainfo/issues/105

Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
2022-04-26 08:03:52 +00:00
Michał Niewrzał
8cdadae124 satellite/metainfo: optimize geo config check for move/copy
This change has two purposes. First is to avoid DB call in case
source and destination bucket are the same.
Second is to return bucket not found error in correct order. If
source and destination bucket are different we will first check
source and later destination. Currently we will get first error
about not existing destination bucket.
Because of this change we stop putting bucket placement
into satellite stream id but its not needed as we don't use
this value with finish move/copy object methods.

Change-Id: I0f7b3ba604d53c722e8fa4d7a37843a69d02bebd
2022-04-01 11:49:44 +00:00
Michał Niewrzał
4da3dc8170 satellite/metabase: improve processing of key/nonce while finish copy/move
So far we assumes that metadata key/nonce cannot be empty at all
but at some point we adjusted code to accept empty metadata/key/nonce
to save DB space.

This change is adjusting how we are processing nonce while
FinishMoveObject/FinishCopyObject. We can use storj.Nonce directly
which makes code cleaner. It's also fixing issue in FinishMoveObject
where we didn't convert nonce correctly to []byte.

Part of change is disabling validation for key and nonce until
uplink will be adjusted. We need change uplink to send always
both key and nonce or non of them. Validation will be restored
as soon as change for uplink will be merged.

https://github.com/storj/storj/issues/4646

Change-Id: Ia1772bc430ae591f54c6a9ae0308a4968aa30bed
2022-03-28 14:34:21 +00:00
Michał Niewrzał
35efb6462c satellite/metainfo: adjust deleting to server side copy
For server-side copy we adjusted one method DeleteObjectExactVersion.
Other deletion methods won't be used directly in code at the moment.
We will adjust other methods later or decide if we will need them at
all.

To handle deletion of objects with copies or just copies correctly we
need to use DeleteObjectExactVersion method in two places while:
* removing object before upload
* explicit object deletion

This change is also changing DeleteObjectExactVersion method to
delete pending objects because we need this functionality to
delete object before new upload.

https://github.com/storj/storj/issues/4481

Change-Id: Ieff5cc95732bb70ed8cc0ecdd62e03c929857c02
2022-03-17 13:57:01 +01:00
Michał Niewrzał
ccd16bbef6 satellite/metabase: handle NewEncryptedMetadata while coping object
Copy object functionality should support setting new metadata for
copy. This change is adjusting FinishCopyObject method to set new
metadata when OverrideMetadata field is set to true.

Fixes https://github.com/storj/storj/issues/4483

Change-Id: Ica37cb57e8edae301cdc483fbda4f3ddba5d2702
2022-03-09 12:31:33 +01:00
Michał Niewrzał
1da9697995 satellite/metainfo: use feature flag for begin/finish object copy metods
We want also to cover with feature flag code for begin/finish
object copy methods.

Change-Id: I97330e7d9b310de98408c47c2c1ed41551631bec
2022-02-28 15:09:40 +00:00
Michał Niewrzał
c0297bae78 satellite/metabase: return object metadata with FinishCopyObject
Updates metadata and metainfo to return object metadata with
FinishCopyObject request.

https://github.com/storj/storj/issues/4474

Change-Id: I32cba5c20a943272e9b5964df1b3d6463ad212dc
2022-02-25 12:46:30 +00:00
Fadila Khadar
2d4760fd09 satellite/metainfo: BeginCopyObject and FinishCopyObject
Metainfo endpoints to use server-side copy.

Fixes https://github.com/storj/storj/issues/4475

Change-Id: Ided06aed9e6187d6d8f030e95dda019ba78fff95
2022-02-24 14:38:04 +01:00
Michał Niewrzał
5041ee0abc satellite/metabase: drop GetObjectLatestVersion method
We don't have different version of object than 1 so at the moment
this method is not needed. Also using GetObjectExactVersion
should be slightly more performant.

Change-Id: I78235d8ae22594cc1d6345dabcc915f41cd7797b
2022-02-02 09:40:53 +00:00
Michał Niewrzał
17f56ab636 satellite/metainfo: split endpoint into smaller files (object/segment)
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Updates https://github.com/storj/team-metainfo/issues/12

Change-Id: I6c691e4d0e192fe3ad7974d2d0ab5ced0d272f3c
2022-01-25 16:14:37 +00:00