Commit Graph

20 Commits

Author SHA1 Message Date
dlamarmorgan
cbc82690d7 satellite: implement metainfo get/set versioning endpoint
Change-Id: Ic43ea8419e7e7092dd191c8ed9f6e4eb4bdff20b
2023-11-18 01:26:34 +00:00
Michal Niewrzal
b96e7401c3 satellite/metainfo: add TestEnableBucketVersioning flag
Additional feature flag (onyly for testing) to set versioning enabled
for all new create buckets. We need it until we will have support
for enabling/disabling versioning for bucket on metainfo API.

In addition this change is fixing also two small issues which makes
testing this flag imposible:
* metabase Status list was not aligned with protobuf definition
* object retruned by metainfo API didn't have correct status set in some
cases

Change-Id: I0d63dff6a08efa588c8999af1e17db476943e067
2023-11-10 19:19:30 +00:00
dlamarmorgan
539b32d01e satellite/metainfo: add UseBucketLevelObjectVersioning by project
This small feature flag will give us the ability to test object
versioning for specific projects without enabling it globally.

Change-Id: I78301f071b7b8079dd1bd4a561fce0800ce9f074
2023-11-09 11:21:06 +00:00
Clement Sam
89d682f49f satellite/metainfo: prevent internal DB errors in Public API
Resolves https://github.com/storj/storj/issues/6081

Change-Id: I0e530db39947138dcafc1b6bd1710ff1ca96b8c5
2023-09-12 15:12:44 +00:00
Artur M. Wolff
37d6df23fa satellite: implement metainfo.GetBucketLocation endpoint
Updates storj/storj-private#408
Updates storj/storj-private#409

Change-Id: Idaaca74b4a5c9c7907d095e0a3a5f29e52843ce6
2023-08-28 13:48:07 +02:00
Michal Niewrzal
95a5cfe647 satellite/buckets: handle bucket exists better
In some rare cases when two entities are trying to create the same
bucket at the same time it's possible that we will return internal
error instead of `bucket already exists`. It's because we are not
handling correctly DB error about constraint error. This change checks
if while inserting bucket into DB we got constraint error and propagate
correct error to metainfo API.

Change-Id: Ie6fd2c943b864b4ea7d71e4a162e74dc3510e386
2023-08-23 14:40:31 +00:00
Michal Niewrzal
a5cbec7b3b satellite/metabase: drop bucket deletion code for copy with references
With this change we are removing code responsible to handle deleting
bucket and supporting server side copies created with references. In practice we are restoring delete queries that we had before server side copy implementation (with small exception, see bellow).

From deletion queries we are also removing parts with segment metadata
as result because we are not longer sending explicit delete requests to
storage nodes.

https://github.com/storj/storj/issues/5891

Change-Id: If866d9f3a1b01e9ebd9b49c4740a6425ba06dd43
2023-08-08 09:07:44 +00:00
Michal Niewrzal
5272fd8497 satellite/metainfo: do full bucket validation only on create
We are doing full bucket name validation for many requests but
we should do this only while creating bucket. Other requests will be
covered only by basic name length validation. Less strict validation for
other requests will make bucket usable in case of invalid bucket names
in DB (we have such cases from the past).

https://github.com/storj/storj/issues/6044

Change-Id: I3a41050e3637787f788705ef15b5dc4df4d01fc6
2023-07-17 16:15:33 +02:00
Cameron
e8fcdc10a4 satellite/metainfo: set user_agent in bucket_metainfos on bucket recreation
Before this change, if a user creates a bucket with a user_agent attributed then deletes and recreates it, the row in bucket_metainfos
will not have the user_agent. This is because we skip setting the field
in bucket_metainfos if the bucket already exists in value_attributions.
This can be problematic, as we return the bucket's user agent during the
ListBuckets operation, and the client may be expecting this value to be
populated.

This change ensures the bucket table user_agent is set when (re)creating a bucket. To avoid decreasing BeginObject performance, which also
updates attribution, a flag has been added to determine whether to
make sure the buckets table is updated: `forceBucketUpdate`.

Change-Id: Iada2f233b327b292ad9f98c73ea76a1b0113c926
2023-07-12 21:48:05 +00:00
Michal Niewrzal
6c08d5024e satellite/metainfo: stop sending delete requests to SN
We decided that we will stop sending explicit delete requests to nodes
and we will cleanup deleted with GC instead.

https://github.com/storj/storj/issues/5888

Change-Id: I65a308cca6fb17e97e3ba85eb3212584c96a32cd
2023-06-14 10:46:02 +00:00
Márton Elek
e4b8febda7 satellite/metainfo: use default_placement of project during bucket creation
https://github.com/storj/storj/issues/5879

Change-Id: I100e834b3b625b3e0ff00cdcb7675ec6bb570992
2023-06-08 11:21:42 +00:00
dlamarmorgan
5fc493b276 metainfo: per-project and user-agent eventkit-based usage tracking
Add eventkit events for each satellite RPC type, tagged with user agent and public project id.

Change-Id: I838401683298d4594495de18563c55031e82f881
2023-04-25 15:13:43 +00:00
Egon Elbre
2405bc8f3b satellite/metabase: stop using the common error type
Updates https://github.com/storj/storj/issues/5291

Change-Id: I7b57a4b454d3619cb5d8ae4cd92f818ad2839c8b
2023-04-19 19:18:18 +03:00
Moby von Briesen
3679e29a93 satellite: Remove remaining references to "partner ID"
This field is deprecated in favor of UserAgent; Removing these
references is the final step necessary before dropping the columns from
the database.

https: //github.com/storj/storj/issues/5432
Change-Id: I3a6619170dcf382f82dc8eddb73b6547eaf636f0
2023-04-19 15:37:10 +00:00
Egon Elbre
41931970fd satellite/*: changes to stop using storj.ListDirection
Currently this duplicates the constants, because we cannot yet bump
common nor uplink.

Change-Id: I81818df8a9092e0b358eb946c31f2a42ac9dbf0a
2023-04-19 15:30:42 +03:00
Egon Elbre
eecb055dfd satellite/buckets: move Bucket definition
Move Bucket struct definition.

Updates https://github.com/storj/storj/issues/5291

Change-Id: I6bfc5ce287793ea479f2cb8b17878ba3cf6b63e0
2023-04-13 17:55:40 -04:00
Qweder93
cf12802a12 satellite/metainfo: ListBuckets extended with value attribution assigned to bucket
We need to provide the ability to see bucket attribution on the gateway side
so customers can validate if bucket is attributed to them. Extendet metainfo.ListBuckets
request with UserAgent.

Fixes https://github.com/storj/storj/issues/4965

Change-Id: I5624874a7faa14cda06183ad44013e9ebb385b63
2022-07-18 18:23:50 +00:00
Moby von Briesen
5b79970575 satellite/metainfo: Use project-level attribution at bucket level
If project.UserAgent is set, use this for bucket.UserAgent on bucket
creation. Otherwise, set bucket attribution as before (getting UserAgent
from request headers).

Tests were updated to create the bucket with a different user, added as
a project member. Otherwise, the tests do not catch the bug.

Change-Id: I7ecf79a8eac5957eed361cbea94823190f58b776
2022-06-07 17:08:50 +00:00
Michał Niewrzał
db5d781735 satellite/metainfo: simplify version collector logging
Version collector previously returned errors and logged them in the
calling code.  It is cleaner to log inside version collector.

Change-Id: I52cb49a1ef53f3f1f51692ddb26ec095cfd0f100
2022-04-26 10:16:20 +00:00
Michał Niewrzał
65c38c6cef satellite/metainfo: split metainfo endpoint into smaller files (buckets)
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Change-Id: I9b097dcc8fa889f985b7f4ef5f8f435a1ff0ef95
2022-01-24 13:47:51 +00:00