New flag 'MultipleVersions' was not correctly passed from metainfo
configuration to metabase configuration. Because configuration was
set correctly for unit tests we didn't catch it and issue was found
while testing on QA satellite.
This change reduce number of places where new metabase flags needs
to be propagated from metainfo configuration to avoid problems with
setting new flags in the future.
Fixes https://github.com/storj/storj/issues/5274
Change-Id: I74bc122649febefd87f665be2fba628f6bfd9044
BeginCopyObject checks twice for write permission in the destination
bucket. One check should be enough.
Change-Id: I3d5935d34f69cd48eaaf00d0117683edfdcefc05
We had multiple experiment so far to collect high cardinality data (mainly in aggregated form).
1. we have a `/top` endpoint which aggregates events with upper bound
2. we use same api (eventstat) to publish S3 gateway-mt agents to influxdb
This patch starts to replace theses api with jtolio/eventkit. Instead of aggregation all events can be sent to a collector host where we can do aggregation and/or persisting data.
Change-Id: Id6df4882b51d2dbd2be9401ee4199d14f3ff7186
The threshold of piece deletions from the nodes during CommitObject
when overriding an existing object seemed to cause a race condition in
tests.
This change makes the threshold configurable so we can set it to maximum
so CommitObject waits until all pieces are removed from the nodes in the
test.
Change-Id: Idf6b52e71d0082a1cd87ad99a2edded6892d02a8
We have new flow where existing object is deleted not on begin
object but on commit object. Deletion on commit object is still
missing deletion from storage nodes. This change adds this part
to the code.
Fixes https://github.com/storj/storj/issues/5222
Change-Id: Ibfd34665b2a055ec6c0d6e260c1a57e8a4c62b0e
With this change we are switching methods to begin object, from
BeginObjectExactVersion to BeginObjectNextVersion. Main implication
is that from now it will be possible to have object with version
different than 1. New object will always get first available version.
Main reason to do this it to avoid deleting existing object during
reuploading object. Now we can create multiple pending objects but
only last committed will be available to the user. Any previous
committed object will be deleted.Because of that we moved logic to
delete existing object from BeginObject to CommitoObject request.
New logic is behind feature flat to be able to test it well first
before enablng on production.
Fixes https://github.com/storj/storj/issues/4871
Change-Id: I2dd9c7364fd93796a05ef607bda9c39a741e6a89
Add new project db method, GetSalt, to get project salt. If salt
column is empty, return the sha-256 hash of the project ID. This
new method is used in metainfo endpoint ProjectInfo to return the
project salt to the client. This is backwards compatible because
the salt column is not populated yet. The updated endpoint will
do the same thing as the current endpoint.
Change-Id: I7eba376c865e10995a5a916302feca7cd7c7efa2
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.
Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
Implement a new service to read retain filter from a bucket and
send them out to storagenodes.
This allows the retain filters to be generated by a separate command on
a backup of the database.
Paralellism (setting ConcurrentSends) and end-to-end garbage collection
tests will be restored in a subsequent commit.
Solves https://github.com/storj/team-metainfo/issues/121
Change-Id: Iaf8a33fbf6987676cc3cf74a18a8078916fe673d
Main issue with those tests was that for case where all objects were
uploaded at once (case "some nodes down" and "all nodes down").
Because all objects had the same name while upload each new object
was overwriting existing object. Because of that instead had several
objects to delete by test explicitly we had just 1. Test were not
failing because while overwriting existing object we were deleting it
but it was not what this test should do.
Change-Id: I602116f00be66589c7c0e68fe28c25e5c03e6b5d
We plan to replace metabase.BeginObjectExactVersion usage in
metainfo.BeginObject with metabase.BeginObjectNextVersion. To make this
switch as simple a possible would be nice to have the same results for
both methods. This change is extending return value for
BeginObjectNextVersion to whole object struct. Tests were also adjusted
to be more like metabase.BeginObjectExactVersion tests.
Part of https://github.com/storj/storj/issues/4871
Change-Id: I4db99d74af07e5a73757b55233e0bbdc7b99d565
We are preparing to use object versions internally and to do
that we need to prepare different parts of the system to handle
object versions different than '1'. This change adjust code
responsible for server-side move and copy.
What was done:
* begin methods for move and copy are now using GetObjectLastCommitted
to find object
* results from begin move and copy operation contains now version to
be able to map object correctly with finish operation
* begin methods are putting version into satellite stream id and
finish methods are using this version as parameter instead hardcoded
value
Fixes https://github.com/storj/storj/issues/4867
Change-Id: I1380911279c21e10a3fff0342793efd2e73eafad
Metadata validation for CommitObject request was placed in a wrong
place. There is a case (old uplink) where encrypted key is bundled
inside encrypted metadata bytes and we need to extract it before we
can validate it. This change moves metadata validation to a place where
we are sure we have encrypted metadata and encrypted metadata encrypted
key ready to be checked.
"Run Versions Test" is covering this case and it was failing without
this change.
Change-Id: Ib709ad901fbb3fa4865a393195b7b3f4c0d87e7a
For object Version in different places we are using different types.
Satellite StreamID is using int32 but metabase accepts int64. Metabase
type is correct one and we should align other places with it.
As a small addition this change is also passing version correctly
between requests instead of using hardcoded value.
Change-Id: I63634d73c0a48c009e4db5f203ff18b7f3218b02
Updated metabase.UpdateObjectMetadata method to update set metdata always for last committed object
Closes https://github.com/storj/storj/issues/4870
Change-Id: I060683e31efcaf3e2531fea143cf0567e5ff5f73
Context cancellation that aborts a non-essential Redis operation must
not be logged as an error because the operation is intentionally
canceled.
We are actually considering them not to be an error in following
operation because of the same reason and we return a RPC canceled status
code.
On the other hand it doesn't make sense to continue if the context is
canceled because although this is a non-essential operation if this one
is canceled due to the context the next one will be canceled for the
same reason, hence, we return earlier.
Change-Id: Ib3331975adeb06367d1ea0a578263ef50ae3f079
Restored GetObjectLatestVersion and renamed it to GetObjectLastCommitted
Add test cases to cover server-side copy
Closes https://github.com/storj/storj/issues/4866
Change-Id: I343b339a60152b8fb92fda97baf80bd8fe60d631
As a reminder
* This counters are for data with high-cardinality
* We have strong upper bound for memory limits
* They can be accessed from /top monitoring interface
Example:
```
curl 172.20.0.10:11111/top
since ~ 2022-08-09T07:45:58Z
auth_request_count project=9094cff8-104e-4956-a367-97ea134b7e06 11.000000
auth_request_buckets 1.000000
auth_request_discarded 0.000000
auth_request_count partner=00000000-0000-0000-0000-000000000000 11.000000
auth_request_buckets 1.000000
auth_request_discarded 0.000000
```
Note: discarded 0 --> we didn't hit the memory limit.
Change-Id: I8db09b4aa61bade55cb324b84b7fbcb8f068c179
We log metainfo object operations and it looks that the log's message
convention is `Object {operation}`, however the `Object Download`
operation didn't match with the actual operation and the one that was
representing it had was `Download Object`.
This commit changes the log's message for the download object operation
according to the other object operations log messages format and fixes
the log message for the Get Object operation.
For finding this I executed the following command at the root of the
repository to obtain the list of lines where we log object operations.
$> ag 'log\.Info\(".*Object.*",' --no-color git:(main)
satellite/metainfo/endpoint_object.go
179: endpoint.log.Info("Object Upload", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "put"), zap.String("type", "object"))
336: endpoint.log.Info("Object Download", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "get"), zap.String("type", "object"))
557: endpoint.log.Info("Download Object", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "download"), zap.String("type", "object"))
791: endpoint.log.Info("Object List", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "list"), zap.String("type", "object"))
979: endpoint.log.Info("Object Delete", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "delete"), zap.String("type", "object"))
`ag` is a command-line tool similar to `grep`
Change-Id: I9072c5967eb42c397a2c64761d843675dd4991ec
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now
Resolves: https://github.com/storj/storj/issues/4470
Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
Previously there was no realtime administration of the storage usage
during copies. Now there is.
Closes https://github.com/storj/storj/issues/4719
Change-Id: I0d536bf551d16208116c3aceac89ed590ec473bf
Piece deletion service was using KnownReliable method from
overlaycache to get nodes addresses to send delete request.
KnownReliable was always hitting DB because this method was
not using cache. This change is using new DownloadSelectionCache
to avoid direct DB calls.
Change is not perfect because DownloadSelectionCache is not as
precise as KnownReliable method and can select few more nodes
to which we will send delete request but difference should be
small and we can improve it later.
Updates https://github.com/storj/storj/issues/4959
Change-Id: I4c3d91089a18ac35ebcb469a56536c33f76e44ea
We need to provide the ability to see bucket attribution on the gateway side
so customers can validate if bucket is attributed to them. Extendet metainfo.ListBuckets
request with UserAgent.
Fixes https://github.com/storj/storj/issues/4965
Change-Id: I5624874a7faa14cda06183ad44013e9ebb385b63
This just cleanup change to unblock libuplink to reorganize types
which are aliases to storj types.
Change-Id: Id3edf13f1b0aef52d7606d545aa7a6594cf8d13f
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.
Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
If project.UserAgent is set, use this for bucket.UserAgent on bucket
creation. Otherwise, set bucket attribution as before (getting UserAgent
from request headers).
Tests were updated to create the bucket with a different user, added as
a project member. Otherwise, the tests do not catch the bug.
Change-Id: I7ecf79a8eac5957eed361cbea94823190f58b776
- parallel deletion of 50 objects and their 50 copies (one copy per object)
This test is skipped because it's creating deadlocks that are not automatically retried on postgres
- parallel deletion of 1 object and its 50 copies.
Fixes https://github.com/storj/storj/issues/4745
Change-Id: Id7a28251c06bb12b5edcc88721f60bf7a4bc0492
We can use PieceIDDeriver in all places where we are deriving id from
the same id multiple times. We have serveral such places: gc, segment
deletion, segment validation, order limit creation. Using it should
save some resources.
Change-Id: I24668d516c0f7cea4aec6470614067734149501d
The existing versionCollector metrics can tell us how many times various
metainfo endpoints are called, but they don't tell us how many bytes a
client is transferring. We currently can't collect precise information
on this, but we can collect information on how much planned traffic is
requested via order limits.
The implementation as provided is intended to measure objects sizes
before erasure encoding is taken into account.
Change-Id: I2f1d2a7831630e8439ecf5342e933df259151792
Create an error class for the "pending object error" for distinguishing
it from other errors for allowing to return it as a "Not Found" DRPC
status code instead an "Internal" status code.
"Internal" errors are logged in the satellite error so this was
polluting the server logs aside of returning an inappropriate status
code.
Change-Id: I10a81adfc887c030c08a228158adc8815834b23c
Version collector previously returned errors and logged them in the
calling code. It is cleaner to log inside version collector.
Change-Id: I52cb49a1ef53f3f1f51692ddb26ec095cfd0f100
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.
https://github.com/storj/team-metainfo/issues/105
Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
We implemented server-side copy feature and we would like to
confirm that it is not affecting expired deletion service.
Resolves: https://github.com/storj/storj/issues/4698
Change-Id: Ia8ca27a7ab7764a48a0c85dc7be80a58bfc83729
