Currently it's awkward to use any of the existing statuses for the
objects in non-recursive listing. Hence, let's add a new one.
Change-Id: I8485e0f858e69998b097e757091991538ca697fa
Protobuf definition is ready to support deleting specific version of
object so we just need to wire requested version into metainfo
BeginDeleteObject endpoint.
Dependencies bumped to get latest metainfo protobuf definition.
https://github.com/storj/storj/issues/6221
Change-Id: Ifc3cc0b49d9acdf4f7e57e7184b0f2ae045f9113
A randomly generated UUID can hit something that starts with `00`.
Also fix a tiny mistake in the comment.
Change-Id: I25a8b21e0f9523bc486e5a38b0c3cc9c36515231
Use `(a, b, c) = ($1, $2, $3)` for the object stream location
arguments rather than combining multiple AND queries together.
Using a tuple comparison is shorter and also easier to see which
object is selected.
Change-Id: Iba84b89630d57255023c30e309eb6afaee9ab944
This will allow further progress towards S3-compatible object
versioning.
Refs: https://github.com/storj/storj/issues/6352
Change-Id: I0b3aa93fcacd1f9d91a667d619d6cb41fba602a9
While the index shouldn't be necessary as long as our implementation is
correct, it still provides some additional checks for mistakes in the
implementation.
Change-Id: I7ed71ac99a979e375d7f94c8898e6f83ac623cb6
By using a separate function for deleting the latest object and
fetching the latest version we can simplify some of the code.
However, there can be more performant approaches, such as using
ON CONFLICT for updating the existing object or using select and delete
in the same query in databases that support it.
Change-Id: I52bc3f9fa025f44d05ee010723ffb81f5bd2a2d7
There are several different object types in a versioned table,
which will determine the exact behaviour.
The object type states are:
* Pending - the object is yet to be committed and is being uploaded.
* Committed - the object has been finished and can be read.
* DeleteMarker - indicates that the object should be treated as not
present when is at the top of the version stack.
There are also versioning states:
* Unversioned - only one unversioned object is allowed per object key.
* Versioned - multiple objects with the same key are allowed.
Change-Id: I65dfa781e8da253a4e5d572b799d53c351196eee
I believe that this change in semantics won't break anything, because
ListVerifySegments is only used by cmd/tools/segment-verify (which only
needs to operate on non-expired segments) and various tests, none of
which expect ListVerifySegments to include expired segments.
Change-Id: I037f43b16bc5750ed914bc32949418e001df1a8c
With the upcoming versioning changes `BeginObjectExactVersion` makes
only sense for testing. Currently this does not rename the options
struct or move it into `metabasetest`, because it would create a
significant amount of merge/rebase noise.
Change-Id: Iafa2f81a05ae66320bc6a839828217ec94c63e1f
When we do `satellite run api --placement '...'`, the placement rules are not parsed well.
The problem is based on `viper.AllSettings()`, and the main logic is sg. like this (from a new unit test):
```
r := ConfigurablePlacementRule{}
err := r.Set(p)
require.NoError(t, err)
serialized := r.String()
r2 := ConfigurablePlacementRule{}
err = r2.Set(serialized)
require.NoError(t, err)
require.Equal(t, p, r2.String())
```
All settings evaluates the placement rules in `ConfigurablePlacementRules` and stores the string representation.
The problem is that we don't have proper `String()` implementation (it prints out the structs instead of the original definition.
There are two main solutions for this problem:
1. We can fix the `String()`. When we parse a placement rule, the `String()` method should print out the original definition
2. We can switch to use pure string as configuration parameter, and parse the rules only when required.
I feel that 1 is error prone, we can do it (and in this patch I added a lot of `String()` implementations, but it's hard to be sure that our `String()` logic is inline with the parsing logic.
Therefore I decided to make the configuration value of the placements a string (or a wrapper around string).
That's the main reason why this patch seems to be big, as I updated all the usages.
But the main part is in beginning of the `placement.go` (configuration parsing is not a pflag.Value implementation any more, but a separated step).
And `filter.go`, (a few more String implementation for filters.
https://github.com/storj/storj/issues/6248
Change-Id: I47c762d3514342b76a2e85683b1c891502a0756a
Currently each testplanet test is running ranged loop no matter if
it's used or not. This is small change with some benefits like:
* saves some cpu cycles
* less log entries
* ranged loop won't interfere with other systems
Change have no big impact on tests execration but I believe it's nice to
have.
Change-Id: I731846bf625cac47ed4f3ca3bc1d1a4659bdcce8
There's only one value that BeginExactObject should return and there's
no point in restating that.
Also, use a clearer value for the random object version.
Change-Id: I06b26ad87d64e1b04b48458f624edd630f7f2f1d
We stoped returning lots of errors as is to avoid leaking our internals
but some errors were meanigful for client. Example of such error is
"exceeded maximum number of parts". With this change we are wrapping
some important commit object errors with new ErrFailedPrecondition
error to be able to return it easily to uplink.
Change-Id: Id834b78362ed1920f0c3f6f1c7d9587bfd27e36a
With pending_objects table support enabled we missed passing correctly
expiration time from pending object to committed object. This change
updates commit query to take into account expiration time.
Change-Id: I67146d5b2f7f0bda02925d16275fbc59acb705bd
Currently it wasn't quite clear what was a stub version and an actual
version. Use a PendingVersion constant to make this distinction clear.
Also use PendingVersion = NextVersion = 0, that way it's clearer that
the version hasn't been yet determined. DefaultVersion = 1 might imply
that the object will get that version once commited, however that will
entirely depend on whether use-pending-objects is used or versioning is
enabled or not.
Change-Id: I21398141f97035c48c778f23b542266b834c44f1
While adding support for pending_objects table one case was missed.
When we are uploading object to location where old objects exists
we are not removing old object segments at all. Old object is
overwritten with new object metadata but segments remains without
corresponding object. This fix removes all existing committed objects
(with it's segments) before committing new object.
https://github.com/storj/storj/issues/6255
Change-Id: Id657840edf763fd6aec8191788d819191b074fb7
With zombie deletion chore we are removing inactive pending objects from
objects table but new we need also to do this for pending_objects table.
https://github.com/storj/storj/issues/6050
Change-Id: Ia29116c103673a1d9e10c2f16654022572210a8a
Extends metabase.BucketEmpty logic to check also pending_objects
table for any entry.
https://github.com/storj/storj/issues/6057
Change-Id: Ia26c272de24a983b308a0b692e6bd5800487eb98
While deleting bucket we need also to delete pending objects from
pending_objects table.
Part of https://github.com/storj/storj/issues/6048
Change-Id: Icc83eaecf8388704e0b6329c397e8028debcf672
New metabase method IteratePendingObjectsByKeyNew to iterate
over entries in pending_objects table with the same object key.
Implementation and tests are mostly copy of code for
IteratePendingObjectsByKey. Main difference is that pending_objects
table have StreamID column part of primary key instead Version.
Method will be used to support new table in
metainfo.ListPendingObjectStreams request.
After full transition to pending_objects table we should remove 'New'
suffix from methods names.
Part of https://github.com/storj/storj/issues/6047
Change-Id: Ifc1ecbc534f8510fbd70c4ec676cf2bf8abb94cb
New metabase method IteratePendingObjects to iterate over entries in
pending_objects table. Implementation and tests are mostly copy of
code that we are using to iterate over objects table. Main difference
is that pending_objects table have StreamID column part of primary
key instead Version. Also structure of pending object is smaller
than the one from object table but it's a detail.
Method will be used to support new table in metainfo.ListObjects
request.
Next step will be to port rest of iterator implementation to support
pending_objects table in metainfo.ListPendingObjectStreams.
Part of https://github.com/storj/storj/issues/6047
Change-Id: Ia578182f88840539f3668d4a242953e061eace02
We are deleting pending objects while aborting multipart upload. We are
using metainfo BeginDeleteObject to do that. This change starts using
DeletePendingObjectNew to delete entry from pending_objects table when
request indicates that object is in this table.
Part of https://github.com/storj/storj/issues/6048
Change-Id: I4478a9c13c8e3db48dc5de3087ef03d1b4c47a5c
With this change we are removing code responsible for deleting objects
and supporting server side copies created with references. In practice
we are restoring delete queries that we had before server side copy
implementation (with small exception, see bellow).
From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.
https://github.com/storj/storj/issues/5891
Change-Id: Iee4e7a9688cff27a60fb95d60dd233d996f41c85
We don't need to support segment copies with references anymore.
We migrated to copies where all metadata are copied from original
segment to copy.
https://github.com/storj/storj/issues/5891
Change-Id: Ic91dc21b0386ddf5c51aea45530024cd463e8ba9
With this change we are removing code responsible to handle deleting
bucket and supporting server side copies created with references. In practice we are restoring delete queries that we had before server side copy implementation (with small exception, see bellow).
From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.
https://github.com/storj/storj/issues/5891
Change-Id: If866d9f3a1b01e9ebd9b49c4740a6425ba06dd43
