This will allow further progress towards S3-compatible object
versioning.
Refs: https://github.com/storj/storj/issues/6352
Change-Id: I0b3aa93fcacd1f9d91a667d619d6cb41fba602a9
While the index shouldn't be necessary as long as our implementation is
correct, it still provides some additional checks for mistakes in the
implementation.
Change-Id: I7ed71ac99a979e375d7f94c8898e6f83ac623cb6
There are several different object types in a versioned table,
which will determine the exact behaviour.
The object type states are:
* Pending - the object is yet to be committed and is being uploaded.
* Committed - the object has been finished and can be read.
* DeleteMarker - indicates that the object should be treated as not
present when is at the top of the version stack.
There are also versioning states:
* Unversioned - only one unversioned object is allowed per object key.
* Versioned - multiple objects with the same key are allowed.
Change-Id: I65dfa781e8da253a4e5d572b799d53c351196eee
This change adds only schema definition of pending_objects table and
small amount of supporting code which will be useful for testing later.
With this table we would like to achieve two major things:
* simplify `objects` table, before we will start working on object
versioning
* gain performance by removing need to filter `objects` results with `status` column, which is not indexed and we would like to avoid that
https://github.com/storj/storj/issues/6045
Change-Id: I6097ce1c644a8a3dad13185915fe01989ad41d90
This flag was in general one time switch to enable versions internally.
New we can remove it as it makes code more complex.
Change-Id: I740b6e8fae80d5fac51d9425793b02678357490e
This patch addresses the following issues:
1. Running full migration in cockroachdb is quite slow. We already have an approach for unit tests to start from the latest snapshot. This patch makes it possible to use it for integrations tests.
2. Migration requires executing a separated command which makes it hard to run application in containerized test environments (like storj-up) or from IDE. This patch introduces a hidden flag to run migration.
3. Test user creation is painful. We do it with calling GraphQL + admin API. Providing an option with testuser makes the integration tests significant more simple (especially as the projectID -> access grant can be predictable)
Change-Id: I61010728727b490ea6aac32620f2da0484966727
When doing server-side copy, deletes the committed version of the target location if it already exists. It does not touch pending versions. The version of the copy is set to the highest already existing version + 1.
Fixes: https://github.com/storj/storj/issues/5071
Change-Id: I1d91ac17054834b1f4f0970a9fa5d58198c58a37
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.
Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.
Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
Currently the metainfo/metabase DB connections are missing the proper
application_name in order to differentiate and filter queries on the DB
side for analytics.
Without it, it is very time-consuming to correlate processes and their load.
This change adds the "check" on DB connection init and passes the fallbacks
in all places to catch connection strings, that do not set it.
Change-Id: Iea5cea8658bc63778ff89038e5c1c352bf482cfd
segment_copies table has (stream_id, ancestor_stream_id) as primary key,
but it should have only stream_id as primary as a stream_id can only have one ancestor_stream_id.
https://github.com/storj/team-metainfo/issues/84
Change-Id: I0218295e86e449c9dfefa5a21319f3083bf80afd
Part of server-side copy implementation.
Adds new table to metabase for keeping references between
copied segments and its ancestors.
Fixes https://github.com/storj/team-metainfo/issues/84
Change-Id: I436d4b533a3d951bcd752e222e3b721cee7f0844
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB
Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
This change adds a NOT NULL constraint to the created_at column in the segment table.
All occurrences of CreatedAt as a pointer are changed to non pointer version (metabase, segment loop, etc)
Change-Id: I3efd476ebd1edd3327b69c9223d9edc800e1cc52
It turns out that some DB methods are not
passing context to to lower levels even if have
context as a parameter.
Change-Id: I82f4fc1a47c362687a91364d85e4468057e53134
Currently the interface is not useful. When we need to vary the
implementation for testing purposes we can introduce a local interface
for the service/chore that needs it, rather than using the large api.
Unfortunately, this requires adding a cleanup callback for tests, there
might be a better solution to this problem.
Change-Id: I079fe4dbe297b0ae08c10081a1cea4dfbc277682
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.
As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.
Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.
Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
The system and database time may drift. We should use database time for
absolute "as of system time" to ensure that it's not newer than the
current database time. When the "as of system time" is in the future,
then the query will fail.
Change-Id: I5423f6aaad966ca03a76b5ff805bfba932e44a51
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.
Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.
metainfo is going to be the "endpoint" logic for handling requests.
Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
