Commit Graph

733 Commits

Author SHA1 Message Date
Michal Niewrzal
4544eee72b Revert "satellite/metainfo: enable metainfo.multiple-versions flag by default"
This reverts commit f0ce8996c3.

We need to revert it until https://github.com/storj/storj/issues/5373 is
fixed.

Change-Id: Ibb22af100014724d1910d4871d8f4e159fdea391
2022-12-07 19:43:20 +00:00
Michal Niewrzal
47125300dc mod: bump dependencies
* storj/common
* storj/private

Latests common version requires small refactoring for names and types
used by metainfo code.

Change-Id: I224fe93b4751c996ba6e846be0e5677252cf830f
2022-11-29 13:47:34 +00:00
Michal Niewrzal
f0ce8996c3 satellite/metainfo: enable metainfo.multiple-versions flag by default
We tested new upload flow (with multiple versions) to fix inconsistency
while uploading object on QA/EUN1/SLC. Now we would like to enable it
for all satellites by default. Tests required small adjustments.

Fixes https://github.com/storj/storj/issues/5283

Change-Id: I0d53c041abebc0d182ba5a88bb1dac906c29caf0
2022-11-23 17:05:22 +00:00
Michal Niewrzal
1ad91265bc satellite/metainfo: reduce log to warn
We have code that is used only by old uplinks and can fail at some point
but we don't interrupt anything and only log message about failure.
Until now it was logged as error but it's nothing critial so we can
reduce it to warning.

As an addition log entry was extended with more information about client
that is using this backward compatibility code.

Change-Id: Ie21c673ee59eb10de065cc371132f8f9505e2220
2022-11-17 18:20:59 +00:00
Michal Niewrzal
1d23c264f6 satellite/metainfo: add storj-downloader as a known user agent
Change-Id: I9977770c43b85aca235d2bc18a454e400dce6e53
2022-11-17 16:03:54 +00:00
Michal Niewrzal
6273ed035d satellite/metabase: make UploadID stable for different options
Multipart upload requires to have the same UploadID returned from
different requests (BeginUpload, ListUploads). Otherwise client won't
be able to find existing uploads. Main issue was that data needed to
construct UploadID is in System metadata which can be filtered out
by listing option.

This change is fixing how we are setting Status for listed objects and
it's forcing reading System metadata if we are reading pending objects.

Fixes https://github.com/storj/storj/issues/5298

Change-Id: I8dd5fbab4421a64dc3ed95556408ead4c829f276
2022-11-10 17:35:36 +00:00
Michal Niewrzal
e4fab975ad satellite/metainfo: fix types used in tests
Libuplink is using some aliases to storj package which we will
move directly to libuplink and remove from common/storj.
To make code compilable we need to fix places where we
are using aliased types directly to be able to update libuplink.

Change-Id: I7222a927af3b41e214d1c9204917f3ebce4727ce
2022-11-07 14:48:31 +01:00
Kaloyan Raev
e709112eeb satellite/metainfo: allow list permission for GetObject and GetObjectIPs
GetObject and GetObjectIPs are invoked by the Linksharing service to
display the shared object and its map. These two endpoint currently
require read permission.

There is a use case where an object can be shared with an access grant
that has only list permission. In such a case, the expectation is that
the linksharing service would still display the metadata of the shared
object (name, size, map), but the content would be still inaccessible.
See https://github.com/storj/gateway-mt/issues/209 for details.

This change allows GetObject and GetObjectIPs to require either read or
list permission to support the described use case.

Change-Id: I3477edc7bf8990e9848482890da047094c875d09
2022-11-07 10:59:43 +00:00
Michal Niewrzal
d21bbab2b2 satellite: fix metabase configuration wiring
New flag 'MultipleVersions' was not correctly passed from metainfo
configuration to metabase configuration. Because configuration was
set correctly for unit tests we didn't catch it and issue was found
while testing on QA satellite.

This change reduce number of places where new metabase flags needs
to be propagated from metainfo configuration to avoid problems with
setting new flags in the future.

Fixes https://github.com/storj/storj/issues/5274

Change-Id: I74bc122649febefd87f665be2fba628f6bfd9044
2022-11-02 15:17:34 +00:00
Kaloyan Raev
689732188b satellite/metainfo: remove duplicate permission check in BeginCopyObject
BeginCopyObject checks twice for write permission in the destination
bucket. One check should be enough.

Change-Id: I3d5935d34f69cd48eaaf00d0117683edfdcefc05
2022-10-27 10:19:05 +00:00
Márton Elek
11df98a392
satellite: use evenkit instead of evenstat/top endpoint
We had multiple experiment so far to collect high cardinality data (mainly in aggregated form).

 1. we have a `/top` endpoint which aggregates events with upper bound
 2. we use same api (eventstat) to publish S3 gateway-mt agents to influxdb

This patch starts to replace theses api with jtolio/eventkit. Instead of aggregation all events can be sent to a collector host where we can do aggregation and/or persisting data.

Change-Id: Id6df4882b51d2dbd2be9401ee4199d14f3ff7186
2022-10-24 11:45:12 +02:00
Erik van Velzen
2ed18a0ed6 satellite/metainfo/piecedeletion: configurable threshold
The threshold of piece deletions from the nodes during CommitObject
when overriding an existing object seemed to cause a race condition in
tests.

This change makes the threshold configurable so we can set it to maximum
so CommitObject waits until all pieces are removed from the nodes in the
test.

Change-Id: Idf6b52e71d0082a1cd87ad99a2edded6892d02a8
2022-10-20 16:58:05 +02:00
Michal Niewrzal
e5ac8430c3 satellite/metainfo: delete pieces from nodes on object commit
We have new flow where existing object is deleted not on begin
object but on commit object. Deletion on commit object is still
missing deletion from storage nodes. This change adds this part
to the code.

Fixes https://github.com/storj/storj/issues/5222

Change-Id: Ibfd34665b2a055ec6c0d6e260c1a57e8a4c62b0e
2022-10-12 15:02:24 +00:00
Michal Niewrzal
1c9f4d1824 satellite/metainfo: more tests for CopyObject
Change-Id: Ibb29f348a15f4afe5d75f9a81987cff4633ceb56
2022-10-11 15:57:23 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
Michal Niewrzal
4d9c9138ce satellite/metainfo: use multiple object versions internally
With this change we are switching methods to begin object, from
BeginObjectExactVersion to BeginObjectNextVersion. Main implication
is that from now it will be possible to have object with version
different than 1. New object will always get first available version.

Main reason to do this it to avoid deleting existing object during
reuploading object. Now we can create multiple pending objects but
only last committed will be available to the user. Any previous
committed object will be deleted.Because of that we moved logic to
delete existing object from BeginObject to CommitoObject request.

New logic is behind feature flat to be able to test it well first
before enablng on production.

Fixes https://github.com/storj/storj/issues/4871

Change-Id: I2dd9c7364fd93796a05ef607bda9c39a741e6a89
2022-10-06 15:19:02 +00:00
Fadila Khadar
02924d0ded satellite/metainfo: temporary feature flag for listing query testing
Fixes: https://github.com/storj/storj/issues/5144

Change-Id: I7650f4d5dd0378e2246339e79710a695996a845c
2022-09-28 17:46:52 +00:00
Cameron
41c2bca096 satellite/{console,metainfo,satellitedb}: add projectdb method GetSalt
Add new project db method, GetSalt, to get project salt. If salt
column is empty, return the sha-256 hash of the project ID. This
new method is used in metainfo endpoint ProjectInfo to return the
project salt to the client. This is backwards compatible because
the salt column is not populated yet. The updated endpoint will
do the same thing as the current endpoint.

Change-Id: I7eba376c865e10995a5a916302feca7cd7c7efa2
2022-09-26 13:12:21 +00:00
Michal Niewrzal
289daf6787 satellite/metainfo: close project while testing
Project was not closed in test cases.

Change-Id: I700f671121433167fce7f350f84aa0559233c43c
2022-09-23 07:45:38 +00:00
Qweder93
c210776a36 satellite/{metabase, metainfo}: merge logic for BeginCopyObject and BeginMoveObject
Closes https://github.com/storj/storj/issues/4514

Change-Id: I4d59fe9e3d30f67ce642d6266b878956e38f5db1
2022-09-22 09:02:20 +00:00
Michal Niewrzal
4e71dd302a satellite/{metainfo,metabase}: prepare feature flag MultipleVersions
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.

Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
2022-09-21 13:07:13 +00:00
Erik van Velzen
e6b5501f9b satellite/gc/sender: new service to send retain filters
Implement a new service to read retain filter from a bucket and
send them out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Paralellism (setting ConcurrentSends) and end-to-end garbage collection
tests will be restored in a subsequent commit.

Solves https://github.com/storj/team-metainfo/issues/121

Change-Id: Iaf8a33fbf6987676cc3cf74a18a8078916fe673d
2022-09-20 11:49:40 +00:00
Michal Niewrzal
464ab43558 satellite/metainfo: fix deletion tests
Main issue with those tests was that for case where all objects were
uploaded at once (case "some nodes down" and "all nodes down").
Because all objects had the same name while upload each new object
was overwriting existing object. Because of that instead had several
objects to delete by test explicitly we had just 1. Test were not
failing because while overwriting existing object we were deleting it
but it was not what this test should do.

Change-Id: I602116f00be66589c7c0e68fe28c25e5c03e6b5d
2022-09-12 11:23:53 +00:00
Michal Niewrzal
eea3fac0d3 satellite/metabase: BeginObjectNextVersion returns object
We plan to replace metabase.BeginObjectExactVersion usage in
metainfo.BeginObject with metabase.BeginObjectNextVersion. To make this
switch as simple a possible would be nice to have the same results for
both methods. This change is extending return value for
BeginObjectNextVersion to whole object struct. Tests were also adjusted
to be more like metabase.BeginObjectExactVersion tests.

Part of https://github.com/storj/storj/issues/4871

Change-Id: I4db99d74af07e5a73757b55233e0bbdc7b99d565
2022-09-12 07:54:14 +00:00
Qweder93
98fb4c4f32 satellite/metainfo: added test for object TTL validation while upload
Closes https://github.com/storj/team-metainfo/issues/113

Change-Id: I6ff2a5f2cdab6fd42d94fbfc335084b927d86bea
2022-09-08 11:39:41 +03:00
Qweder93
2a1bcfed81 {satellite/metabase, satellite/metainfo, satellite/inspector} : Use metabase.GetObjectLastCommitted instead metabase.GetObjectExactVersion
adjusted metainfo, metabase and inspector methods to use GetObjectLastCommitted,
added test cases to confirm that this method will work correctly with version higher then 1

Closes https://github.com/storj/storj/issues/4869
Closes https://github.com/storj/storj/issues/4873

Change-Id: I8c338dcd1db82c141383f41339a295d54e2fa039
2022-09-08 07:27:22 +00:00
Michal Niewrzal
fa76dfe577 satellite/metabase: use GetObjectLastCommitted with move/copy
We are preparing to use object versions internally and to do
that we need to prepare different parts of the system to handle
object versions different than '1'. This change adjust code
responsible for server-side move and copy.

What was done:
* begin methods for move and copy are now using GetObjectLastCommitted
to find object
* results from begin move and copy operation contains now version to
be able to map object correctly with finish operation
* begin methods are putting version into satellite stream id and
finish methods are using this version as parameter instead hardcoded
value

Fixes https://github.com/storj/storj/issues/4867

Change-Id: I1380911279c21e10a3fff0342793efd2e73eafad
2022-08-30 06:36:02 +00:00
Sean Harvey
5a5f609022
satellite/metainfo: add copy and move log and metrics
Change-Id: I579c03c389dea7f8e7e92ade5bcac53740ce69c8
2022-08-26 18:55:29 +12:00
Michal Niewrzal
d6054214c3 satellite/metainfo: move metadata validation to a proper place
Metadata validation for CommitObject request was placed in a wrong
place. There is a case (old uplink) where encrypted key is bundled
inside encrypted metadata bytes and we need to extract it before we
can validate it. This change moves metadata validation to a place where
we are sure we have encrypted metadata and encrypted metadata encrypted
key ready to be checked.

"Run Versions Test" is covering this case and it was failing without
this change.

Change-Id: Ib709ad901fbb3fa4865a393195b7b3f4c0d87e7a
2022-08-24 10:50:43 +00:00
Michał Niewrzał
ea38860aab satellite/metainfo: align satellite StreamID protobuf version with metabase
For object Version in different places we are using different types.
Satellite StreamID is using int32 but metabase accepts int64. Metabase
type is correct one and we should align other places with it.

As a small addition this change is also passing version correctly
between requests instead of using hardcoded value.

Change-Id: I63634d73c0a48c009e4db5f203ff18b7f3218b02
2022-08-24 10:17:34 +00:00
Qweder93
2139d3966b satellite/metabase: update UpdateObjectMetadata to work with latest commited object
Updated metabase.UpdateObjectMetadata method to update set metdata always for last committed object

Closes https://github.com/storj/storj/issues/4870

Change-Id: I060683e31efcaf3e2531fea143cf0567e5ff5f73
2022-08-24 10:28:57 +03:00
Michał Niewrzał
b722c29e77 satellite/metainfo: add missing metadata validation
We are missing metadata validation for UpdateObjectMetadata and
FinishCopyOject requests.

Change-Id: Idca6a4d1fe108e1593405fd3913442f5b69d09e7
2022-08-23 16:56:45 +02:00
Ivan Fraixedes
aa1f50197a satellite/metainfo: Don't log error due to ctx cancellation
Context cancellation that aborts a non-essential Redis operation must
not be logged as an error because the operation is intentionally
canceled.

We are actually considering them not to be an error in following
operation because of the same reason and we return a RPC canceled status
code.

On the other hand it doesn't make sense to continue if the context is
canceled because although this is a non-essential operation if this one
is canceled due to the context the next one will be canceled for the
same reason, hence, we return earlier.

Change-Id: Ib3331975adeb06367d1ea0a578263ef50ae3f079
2022-08-19 18:41:54 +00:00
Qweder93
2abe709b6e satellite/metabase: add metabase.GetObjectLatestVersion back
Restored GetObjectLatestVersion and renamed it to GetObjectLastCommitted
Add test cases to cover server-side copy

Closes https://github.com/storj/storj/issues/4866

Change-Id: I343b339a60152b8fb92fda97baf80bd8fe60d631
2022-08-11 22:14:38 +00:00
Márton Elek
f507de67f9 satellite: in-memory 'top'-like counter for project/partner
As a reminder
 * This counters are for data with high-cardinality
 * We have strong upper bound for memory limits
 * They can be accessed from /top monitoring interface

Example:

```
curl 172.20.0.10:11111/top
since ~ 2022-08-09T07:45:58Z
auth_request_count project=9094cff8-104e-4956-a367-97ea134b7e06 11.000000
auth_request_buckets  1.000000
auth_request_discarded  0.000000
auth_request_count partner=00000000-0000-0000-0000-000000000000 11.000000
auth_request_buckets  1.000000
auth_request_discarded  0.000000
```

Note: discarded 0 --> we didn't hit the memory limit.

Change-Id: I8db09b4aa61bade55cb324b84b7fbcb8f068c179
2022-08-11 10:21:54 +00:00
Ivan Fraixedes
c5abc0de6f satellite/metainfo: Homogenize object operations logs
We log metainfo object operations and it looks that the log's message
convention is `Object {operation}`, however the `Object Download`
operation didn't match with the actual operation and the one that was
representing it had was `Download Object`.

This commit changes the log's message for the download object operation
according to the other object operations log messages format and fixes
the log message for the Get Object operation.

For finding this I executed the following command at the root of the
repository to obtain the list of lines where we log object operations.

   $> ag 'log\.Info\(".*Object.*",' --no-color                                                                                                                                                                                                                                       git:(main)
   satellite/metainfo/endpoint_object.go
   179:    endpoint.log.Info("Object Upload", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "put"), zap.String("type", "object"))
   336:    endpoint.log.Info("Object Download", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "get"), zap.String("type", "object"))
   557:    endpoint.log.Info("Download Object", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "download"), zap.String("type", "object"))
   791:    endpoint.log.Info("Object List", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "list"), zap.String("type", "object"))
   979:    endpoint.log.Info("Object Delete", zap.Stringer("Project ID", keyInfo.ProjectID), zap.String("operation", "delete"), zap.String("type", "object"))

`ag` is a command-line tool similar to `grep`

Change-Id: I9072c5967eb42c397a2c64761d843675dd4991ec
2022-08-05 23:11:39 +00:00
Qweder93
2b24632610 {satellite/metainfo, satellite/accounting}: remove ValidateSegmentLimit config value and check
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now

Resolves: https://github.com/storj/storj/issues/4470

Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
2022-07-27 15:36:53 +00:00
Erik van Velzen
b5fc04af89 satellite/metainfo: usage limits for copy
Previously there was no realtime administration of the storage usage
during copies. Now there is.

Closes https://github.com/storj/storj/issues/4719

Change-Id: I0d536bf551d16208116c3aceac89ed590ec473bf
2022-07-25 12:35:27 +02:00
Michał Niewrzał
d72f9525d4 satellite/metainfo/piecedeletion: use nodes cache
Piece deletion service was using KnownReliable method from
overlaycache to get nodes addresses to send delete request.
KnownReliable was always hitting DB because this method was
not using cache. This change is using new DownloadSelectionCache
to avoid direct DB calls.

Change is not perfect because DownloadSelectionCache is not as
precise as KnownReliable method and can select few more nodes
to which we will send delete request but difference should be
small and we can improve it later.

Updates https://github.com/storj/storj/issues/4959

Change-Id: I4c3d91089a18ac35ebcb469a56536c33f76e44ea
2022-07-21 00:43:24 +00:00
Qweder93
cf12802a12 satellite/metainfo: ListBuckets extended with value attribution assigned to bucket
We need to provide the ability to see bucket attribution on the gateway side
so customers can validate if bucket is attributed to them. Extendet metainfo.ListBuckets
request with UserAgent.

Fixes https://github.com/storj/storj/issues/4965

Change-Id: I5624874a7faa14cda06183ad44013e9ebb385b63
2022-07-18 18:23:50 +00:00
Michał Niewrzał
43286dbc37 satellite/metainfo: use metaclient type instead of storj
This just cleanup change to unblock libuplink to reorganize types
which are aliases to storj types.

Change-Id: Id3edf13f1b0aef52d7606d545aa7a6594cf8d13f
2022-07-08 14:17:19 +02:00
Qweder93
417df4d73c satellite/metabase: don't return expired objects
Resolves: https://github.com/storj/team-metainfo/issues/112

Change-Id: If48f865826c22764fc1ff83745c34524cd396aac
2022-07-04 21:32:30 +03:00
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Jeremy Wharton
58c5d44f44 satellite/console: integrate sessions into satellite UI
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.

Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
2022-06-13 08:02:02 +00:00
Moby von Briesen
5b79970575 satellite/metainfo: Use project-level attribution at bucket level
If project.UserAgent is set, use this for bucket.UserAgent on bucket
creation. Otherwise, set bucket attribution as before (getting UserAgent
from request headers).

Tests were updated to create the bucket with a different user, added as
a project member. Otherwise, the tests do not catch the bug.

Change-Id: I7ecf79a8eac5957eed361cbea94823190f58b776
2022-06-07 17:08:50 +00:00
Fadila Khadar
4725a3878c satellite/metainfo: parallel deletion tests
- parallel deletion of 50 objects and their 50 copies (one copy per object)
This test is skipped because it's creating deadlocks that are not automatically retried on postgres
- parallel deletion of 1 object and its 50 copies.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Id7a28251c06bb12b5edcc88721f60bf7a4bc0492
2022-05-26 11:00:42 +00:00
Michał Niewrzał
6759ba831c satellite/metainfo: skip TestUserAgentTransferStats
It's hard to make this test stable on CI but still can be
useful for local testing.

Change-Id: Iac5368d1a9179b67ea2409ee99bcf189db862897
2022-05-25 11:15:23 +00:00
Michał Niewrzał
456aea727e satellite: use PieceIDDeriver for derivation
We can use PieceIDDeriver in all places where we are deriving id from
the same id multiple times. We have serveral such places: gc, segment
deletion, segment validation, order limit creation. Using it should
save some resources.

Change-Id: I24668d516c0f7cea4aec6470614067734149501d
2022-05-19 06:31:42 +00:00
Michał Niewrzał
230424eace satellite/metainfo: add IPFS datastore user agent to metrics
Change-Id: I2ac14c97c64542669acef58ca6ded79984d0bd11
2022-05-17 13:09:22 +02:00
Michał Niewrzał
3ebb05ecca satellite/metainfo: add user-agent bandwidth metric
The existing versionCollector metrics can tell us how many times various
metainfo endpoints are called, but they don't tell us how many bytes a
client is transferring.  We currently can't collect precise information
on this, but we can collect information on how much planned traffic is
requested via order limits.

The implementation as provided is intended to measure objects sizes
before erasure encoding is taken into account.

Change-Id: I2f1d2a7831630e8439ecf5342e933df259151792
2022-05-17 09:57:39 +00:00