We decided that we won't use seprate table for handling pending
objects. We need to remove related code.
https://github.com/storj/storj/issues/6421
Change-Id: I442b0f58da75409f725e08e2cd83d29ed4f91ec6
Credit to the tests in TestListObjectsVersioned(), which (since
ListObjects is logically a pretty thin wrapper around
iterateAllVersionsWithStatus()) were able to be translated to tests in
TestIterateObjectsWithStatus with only minor changes.
Change-Id: I6e68675b24fee614e44baddf596703115554014e
This changes metabase behavior such that the latest object
will be the committed object which may result in a new version
assigned to the objects.
Change-Id: I7a731d7db4696caba75fff65107a248569b6111f
There are several different object types in a versioned table,
which will determine the exact behaviour.
The object type states are:
* Pending - the object is yet to be committed and is being uploaded.
* Committed - the object has been finished and can be read.
* DeleteMarker - indicates that the object should be treated as not
present when is at the top of the version stack.
There are also versioning states:
* Unversioned - only one unversioned object is allowed per object key.
* Versioned - multiple objects with the same key are allowed.
Change-Id: I65dfa781e8da253a4e5d572b799d53c351196eee
With the upcoming versioning changes `BeginObjectExactVersion` makes
only sense for testing. Currently this does not rename the options
struct or move it into `metabasetest`, because it would create a
significant amount of merge/rebase noise.
Change-Id: Iafa2f81a05ae66320bc6a839828217ec94c63e1f
There's only one value that BeginExactObject should return and there's
no point in restating that.
Also, use a clearer value for the random object version.
Change-Id: I06b26ad87d64e1b04b48458f624edd630f7f2f1d
Currently it wasn't quite clear what was a stub version and an actual
version. Use a PendingVersion constant to make this distinction clear.
Also use PendingVersion = NextVersion = 0, that way it's clearer that
the version hasn't been yet determined. DefaultVersion = 1 might imply
that the object will get that version once commited, however that will
entirely depend on whether use-pending-objects is used or versioning is
enabled or not.
Change-Id: I21398141f97035c48c778f23b542266b834c44f1
New metabase method IteratePendingObjects to iterate over entries in
pending_objects table. Implementation and tests are mostly copy of
code that we are using to iterate over objects table. Main difference
is that pending_objects table have StreamID column part of primary
key instead Version. Also structure of pending object is smaller
than the one from object table but it's a detail.
Method will be used to support new table in metainfo.ListObjects
request.
Next step will be to port rest of iterator implementation to support
pending_objects table in metainfo.ListPendingObjectStreams.
Part of https://github.com/storj/storj/issues/6047
Change-Id: Ia578182f88840539f3668d4a242953e061eace02
Change is adjusting BeginSegment to check pending object existence in
`pending_objects` or `objects` table.
Satellite stream id is used to determine if we need to use
`pending_objects` or `objects` table.
General goal is to support both tables until `objects` table will be
free from pending objects. Whenever it will be needed code will be
supporting both tables at once.
Part of https://github.com/storj/storj/issues/6046
Change-Id: I08aaa605c23d82695fde352fdbd0a7fd11f46bb5
..instead of using segment_copies and ancestor_stream_id, etc.
This bypasses reference counting entirely, depending on our mark+sweep+
bloomfilter garbage collection strategy to get rid of pieces once they
are no longer part of a segment.
This is only safe to do after we have stopped passing delete requests on
to storage nodes.
Refs: https://github.com/storj/storj/issues/5889
Change-Id: I37bdcffaa752f84fd85045235d6875b3526b5ecc
metabasetest package utils can be used by both tests and benchmarks
if we will use interface TestingT from require package. This change
adjusts metabasettest.CreateObject method
Change-Id: I3c138e2ef9873b804ab5b3402804efa409397a9f
Previously copying an object to it's ancestor location (copy of copy)
broke the object and all copies.
This fixes this by calling the existing delete method rather than a
custom one when there is an existing object at the copy destination.
The check for existing object at destination has been moved to an
earlier point in FinishCopy.
metabase.DeleteObject exposes a transaction parameter so that it can be
reused within metabase.
Closes https://github.com/storj/storj/issues/4707
Uplink test at https://review.dev.storj.io/c/storj/uplink/+/7557
Change-Id: I418fc3337fa9f30146ccc1db456af168ae41c326
So far we assumes that metadata key/nonce cannot be empty at all
but at some point we adjusted code to accept empty metadata/key/nonce
to save DB space.
This change is adjusting how we are processing nonce while
FinishMoveObject/FinishCopyObject. We can use storj.Nonce directly
which makes code cleaner. It's also fixing issue in FinishMoveObject
where we didn't convert nonce correctly to []byte.
Part of change is disabling validation for key and nonce until
uplink will be adjusted. We need change uplink to send always
both key and nonce or non of them. Validation will be restored
as soon as change for uplink will be merged.
https://github.com/storj/storj/issues/4646
Change-Id: Ia1772bc430ae591f54c6a9ae0308a4968aa30bed
If B is a copy of A, and C is a copy of B, then in the segment_copies table, it should appear that C is a copy of A.
Fixes https://github.com/storj/storj/issues/4538
Change-Id: I7e6b03f7584597cf616cd1e0cd0156386771d207
In the server-side copy initial implementation, we are inserting segments one by one. This PR inserts them all at once.
Fixes https://github.com/storj/storj/issues/4476
Change-Id: I776dba99be38a0eef73366e8e9287cbb794003dc
Copy object functionality should support setting new metadata for
copy. This change is adjusting FinishCopyObject method to set new
metadata when OverrideMetadata field is set to true.
Fixes https://github.com/storj/storj/issues/4483
Change-Id: Ica37cb57e8edae301cdc483fbda4f3ddba5d2702
Return segments when creating a test object so that it can be checked
that the correct segments are remaining after a delete action.
Change-Id: Ifc245948935ba278806e887672c03abc5f2c2654
Updates metadata and metainfo to return object metadata with
FinishCopyObject request.
https://github.com/storj/storj/issues/4474
Change-Id: I32cba5c20a943272e9b5964df1b3d6463ad212dc
Part of server-side copy implementation.
Creates the methods BeginCopyObject and FinishCopyObject in satellite/metabase/copy_object.go.
This commit makes it possible to copy objects and their segments in metabase.
https://github.com/storj/team-metainfo/issues/79
Change-Id: Icc64afce16e5e0e15b83c43cff36797bb9bd39bc
This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.
This reverts commit 1ef06fae99.
Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.
Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
Loops are using custom structs to provide
segments while iterating/looping so we need
to expose new field there too.
Change-Id: I12c8f4a01afeac171bf638d278253999fa90a8cb
We added expires_at column to segments table and now
we need to populate this column while committing segment.
We still need to migrate existing segments with
separate tool.
Change-Id: Ibac8c63d97201dd98cc2cb9db385f4cb73bc3f7e
