Commit Graph

47 Commits

Author SHA1 Message Date
Michal Niewrzal
881137539c satellite/metabase: commit object is not respecting expiration time
With pending_objects table support enabled we missed passing correctly
expiration time from pending object to committed object. This change
updates commit query to take into account expiration time.

Change-Id: I67146d5b2f7f0bda02925d16275fbc59acb705bd
2023-09-14 16:25:28 +02:00
Michal Niewrzal
e21978f11a satellite/metainfo: support getting specific object version
Protobuf definition is read to support getting specific version of
object so we just need to wire requested version into metainfo.GetObject
endpoint.

https://github.com/storj/storj/issues/6221

Change-Id: If4568b82119a6c893788a0a86e598b05ff5951cf
2023-09-12 14:27:40 +00:00
Egon Elbre
28ee6f024c satellite: don't use fmt.Print in tests
Change-Id: Ia10450240ad075c9d78614adff9164f292fb1fa0
2023-09-12 15:11:02 +03:00
Michal Niewrzal
0a3ee6ff8a satellite/metabase: remove old object segments on overwrite
While adding support for pending_objects table one case was missed.
When we are uploading object to location where old objects exists
we are not removing old object segments at all. Old object is
overwritten with new object metadata but segments remains without
corresponding object. This fix removes all existing committed objects
(with it's segments) before committing new object.

https://github.com/storj/storj/issues/6255

Change-Id: Id657840edf763fd6aec8191788d819191b074fb7
2023-09-11 14:16:47 +00:00
Kaloyan Raev
4e499fb9bf satellite/metainfo: respect MaxObjectTTL in BeginObject
If MaxObjectTTL is set in the API key, BeginObject will use it for the
object expiration time, unless an explicit ExpireAt is available in the
request.

Context: https://github.com/storj/storj/issues/6249

Change-Id: I2adf57d979a9c68eec3a787f3739d2f1dbec1f7e
2023-09-08 09:30:48 +00:00
Michal Niewrzal
780c0e0b35 satellite/metainfo: adjust ListPendingObjectStreams to pending_objects
table

New method IteratePendingObjectsByKeyNew is used to provide results for
metainfo.ListPendingObjectStreams. This endpoint is used to list
pending objects with the same object key. In this case to support
both tables (objects, pending_objects) we need to do one query per table
and merge results.

Because existing metainfo protobuf API is missing some fields to have
proper listing cursor we are not able to make ListPendingObjectStreams
correct for returning more than single page. We need to fix it
separately.

With this change also turns out that approach to merge results from
listing objects for ListObjects method was wrong and this change is also
fixing this problem.

Handling both tables will be removed at some point and only
pending_objects will be used to look for results.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I8a88a6f885ad529704e6c032f1d97926123c2909
2023-08-30 13:35:54 +00:00
Michal Niewrzal
c010e37374 satellite/metainfo: adjust ListObjects to use pending_objects table
Adjust metainfo.ListObjects method to use IteratePendingObjects to
support new pending_objects table. New method will be used only when
we are listing pending objects.

Because until objects table will be free from pending objects we can
have results in both tables we are merging listing results. This also
means that in some (rare?) cases we may return more results than
specified listing limit. This situation is temporary.

Part of https://github.com/storj/storj/issues/6047

Change-Id: I06389145e5d916c532dfdbd3dcc9ef68ef70e515
2023-08-28 16:22:54 +00:00
Michal Niewrzal
5a8ef89824 satellite/{metainfo,metabase}: delete from pending_objects table
We are deleting pending objects while aborting multipart upload. We are
using metainfo BeginDeleteObject to do that. This change starts using
DeletePendingObjectNew to delete entry from pending_objects table when
request indicates that object is in this table.

Part of https://github.com/storj/storj/issues/6048

Change-Id: I4478a9c13c8e3db48dc5de3087ef03d1b4c47a5c
2023-08-21 08:06:23 +00:00
Márton Elek
de7aabc8c9 satellite/{repair,rangedloop,overlay}: fix node tag placement selection for repair
This patch fixes the node tag based placement of rangedloop/repairchecker + repair process.

The main change is just adding the node tags for Reliable and KnownReliabel database calls + adding new tests to prove, it works.

https://github.com/storj/storj/issues/6126

Change-Id: I245d654a18c1d61b2c72df49afa0718d0de76da1
2023-08-16 15:45:41 +00:00
Egon Elbre
1f261bcc70 go.mod: bump lang to 1.19 and common
Change-Id: I8d91f97d786456da29ebe89a78412c50efbb8ccc
2023-08-10 18:41:15 +03:00
Michal Niewrzal
7be844351d satellite/metainfo: remove ServerSideCopyDuplicateMetadata
https://github.com/storj/storj/issues/5891

Change-Id: Ib5440169107acca6e832c2280e1ad12dfd380f28
2023-08-08 12:15:10 +00:00
Michal Niewrzal
03f8ad323d satellite/metabase: remove segment_copies support from ListSegments
We don't need to support segment copies with references anymore.
We migrated to copies where all metadata are copied from original
segment to copy.

https://github.com/storj/storj/issues/5891

Change-Id: Ic91dc21b0386ddf5c51aea45530024cd463e8ba9
2023-08-08 11:21:08 +00:00
Michal Niewrzal
0303920da7 satellite/metainfo: remove unused method
Change-Id: I08e307e6909cdc46951c5f3112d77a685e67fe2e
2023-07-18 08:45:29 +00:00
Márton Elek
97a89c3476 satellite: switch to use nodefilters instead of old placement.AllowedCountry
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.

The 90% of this patch is just search and replace:

 * we need to use NodeFilters instead of placement.AllowedCountry
 * which means, we need an initialized PlacementRules available everywhere
 * which means we need to configure the placement rules

The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.

Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
2023-07-07 16:55:45 +00:00
Michal Niewrzal
6c08d5024e satellite/metainfo: stop sending delete requests to SN
We decided that we will stop sending explicit delete requests to nodes
and we will cleanup deleted with GC instead.

https://github.com/storj/storj/issues/5888

Change-Id: I65a308cca6fb17e97e3ba85eb3212584c96a32cd
2023-06-14 10:46:02 +00:00
Michal Niewrzal
f7c7851519 satellite/metainfo: filter metainfo.GetObjectIPs by bucket/object placement
For now we will use bucket placement to determine if we should exclude
some node IPs from metainfo.GetObjectIPs results. Bucket placement is
retrieved directly from DB in parallel to metabase
GetStreamPieceCountByNodeID request.

GetObjectIPs is not heavily used so additional request to DB shouldn't
be a problem for now.

https://github.com/storj/storj/issues/5950

Change-Id: Idf58b1cfbcd1afff5f23868ba2f71ce239f42439
2023-06-07 16:52:02 +00:00
Michal Niewrzal
114eda6e87 satellite/metainfo: remove sleep from upload limit test
lrucache is now using time2 package and we can make expiration
test without using time.Sleep.

https://github.com/storj/storj/issues/5788

Change-Id: I48f2693c3db78fcf4e30e618bb3304be3625100c
2023-04-18 09:38:21 +02:00
Egon Elbre
eecb055dfd satellite/buckets: move Bucket definition
Move Bucket struct definition.

Updates https://github.com/storj/storj/issues/5291

Change-Id: I6bfc5ce287793ea479f2cb8b17878ba3cf6b63e0
2023-04-13 17:55:40 -04:00
Michal Niewrzal
31f5e2cb65 satellite/metainfo: limit uploads to the same location
We would like to have ability to limit burst uploads to the single
object (the same location). This change we are limiting such upload to
one per second.

Change-Id: Ib9351df1017cbc07d7fc2f846c2dbdbfcd3a360c
2023-04-06 15:49:11 +00:00
Egon Elbre
f5020de57c storagenode/blobstore: move blob store logic
The blobstore implementation is entirely related to storagenode, so the
rightful place is together with the storagenode implementation.

Fixes https://github.com/storj/storj/issues/5754

Change-Id: Ie6637b0262cf37af6c3e558556c7604d9dc3613d
2023-04-05 18:06:20 +00:00
Michal Niewrzal
54b6e1614a satellite/metainfo: drop MultipleVersions config flag
This flag was in general one time switch to enable versions internally.
New we can remove it as it makes code more complex.

Change-Id: I740b6e8fae80d5fac51d9425793b02678357490e
2023-03-29 13:45:09 +00:00
JT Olio
f905878ff9 go.mod: bump storj.io/uplink
this change brings in:

 * private/piecestore: disable noise
 * uplink: fix noise version report
 * private/piecestore: fix noise
 * private/metaclient: honor context cancellation on retries
 * private/metaclient: fix multi-passphrase ListObjects paging
 * private/metaclient: fix broken ListUploads method

it also adds:

 * private/storage/streams/batchaggregator: introduce Aggregator
 * private/storage/streams/streambatcher: introduce Batcher
 * private/storage/streams: pieceupload package
 * private/storage/streams: segmenttracker package

but these commits provably change no existing behavior, since they
only add unused, new packages and types.

Change-Id: Id9efc65117b7e1ddcb3ffc8cc15d3e665ed11d7d
2023-02-21 16:09:19 +00:00
Michal Niewrzal
0a525292e4 satellite/metainfo: fix duplicates while listing pending objects
While working on fixing listing for committed objects we didn't fix
the same case for pending objects. For case were we have many
pending objects under different locations we need to set cursor
version to highest value to avoid duplicates.

For case where we have many pending objects under the same location
we will need to make a separate fix.

https://github.com/storj/storj/issues/5570

Change-Id: Id5c8eb728868e8e1177fdbcf65a493142be4eaf0
2023-02-15 16:52:53 +00:00
Michal Niewrzal
41bcc6bb62 satellite/metainfo: fix duplicates while listing committed objects
We have an issue where object can appear in two different listing pages.
It's because protobuf listing cursor doesn't have version included and
now we can have internally versions higher than 1. On satellite side
version 1 was always used as a default cursor version.

As a workaround for existing implementation of libuplink library we will
use always maximum version for listing cursor on satellite side.

Fixing protobuf and libuplink implementation will happen later.

https://github.com/storj/storj/issues/5570

Change-Id: Ibd27b174556c9d8b8bd60fab8cff7862fd11e994
2023-02-14 14:47:27 +01:00
Kaloyan Raev
b1d4a159a6 satellite/metainfo: ListUploads returns incorrect UploadID
ListUploads returns incorrect UploadID if Expires was set in
BeginUpload. DB is truncating expiration date to microseconds precision
so we need to do this also in code.

Change-Id: Iee0cf45cb705342f6bb9a2f745acca91cce6ff52
2023-02-09 10:33:08 +00:00
Michal Niewrzal
77afdae741 satellite/metabase: handle target pending/committed objects while move
Before we introduced objects versions internally move operation was
always failing when under target location object exists. But then we
had only single version 1 all the time. With versions different than 1
we need to check all existing objects under target location.

To be backward compatible with our API new logic looks like this:
* if there is no object under target location use source object version
as target version
* if there are only pending objects find first free (highest) version
which could be used to move object there
* if there is committed object under target location reject move
operation

Fixes https://github.com/storj/storj/issues/5403

Change-Id: I717f3e7c42470b406287d6ec335f6f057d3fc3b5
2023-01-04 08:50:51 +00:00
Michal Niewrzal
0759cbdc7f satellite/metabase: handle copies with GetStreamPieceCountByNodeID
We missed proper handling of object copies for method
GetStreamPieceCountByNodeID which is used by metabase.GetObjectIPs.
That caused some lack of IPs returned when queriyng IPs of copy and
broke things like pices map on linksharing.

Fixes https://github.com/storj/storj/issues/5406

Change-Id: I9574776f34880788c2dc9ff78a6ae20d44fe628f
2022-12-13 12:32:56 +01:00
Michal Niewrzal
47125300dc mod: bump dependencies
* storj/common
* storj/private

Latests common version requires small refactoring for names and types
used by metainfo code.

Change-Id: I224fe93b4751c996ba6e846be0e5677252cf830f
2022-11-29 13:47:34 +00:00
Michal Niewrzal
6273ed035d satellite/metabase: make UploadID stable for different options
Multipart upload requires to have the same UploadID returned from
different requests (BeginUpload, ListUploads). Otherwise client won't
be able to find existing uploads. Main issue was that data needed to
construct UploadID is in System metadata which can be filtered out
by listing option.

This change is fixing how we are setting Status for listed objects and
it's forcing reading System metadata if we are reading pending objects.

Fixes https://github.com/storj/storj/issues/5298

Change-Id: I8dd5fbab4421a64dc3ed95556408ead4c829f276
2022-11-10 17:35:36 +00:00
Michal Niewrzal
e4fab975ad satellite/metainfo: fix types used in tests
Libuplink is using some aliases to storj package which we will
move directly to libuplink and remove from common/storj.
To make code compilable we need to fix places where we
are using aliased types directly to be able to update libuplink.

Change-Id: I7222a927af3b41e214d1c9204917f3ebce4727ce
2022-11-07 14:48:31 +01:00
Erik van Velzen
2ed18a0ed6 satellite/metainfo/piecedeletion: configurable threshold
The threshold of piece deletions from the nodes during CommitObject
when overriding an existing object seemed to cause a race condition in
tests.

This change makes the threshold configurable so we can set it to maximum
so CommitObject waits until all pieces are removed from the nodes in the
test.

Change-Id: Idf6b52e71d0082a1cd87ad99a2edded6892d02a8
2022-10-20 16:58:05 +02:00
Michal Niewrzal
e5ac8430c3 satellite/metainfo: delete pieces from nodes on object commit
We have new flow where existing object is deleted not on begin
object but on commit object. Deletion on commit object is still
missing deletion from storage nodes. This change adds this part
to the code.

Fixes https://github.com/storj/storj/issues/5222

Change-Id: Ibfd34665b2a055ec6c0d6e260c1a57e8a4c62b0e
2022-10-12 15:02:24 +00:00
Michal Niewrzal
1c9f4d1824 satellite/metainfo: more tests for CopyObject
Change-Id: Ibb29f348a15f4afe5d75f9a81987cff4633ceb56
2022-10-11 15:57:23 +00:00
Michal Niewrzal
4d9c9138ce satellite/metainfo: use multiple object versions internally
With this change we are switching methods to begin object, from
BeginObjectExactVersion to BeginObjectNextVersion. Main implication
is that from now it will be possible to have object with version
different than 1. New object will always get first available version.

Main reason to do this it to avoid deleting existing object during
reuploading object. Now we can create multiple pending objects but
only last committed will be available to the user. Any previous
committed object will be deleted.Because of that we moved logic to
delete existing object from BeginObject to CommitoObject request.

New logic is behind feature flat to be able to test it well first
before enablng on production.

Fixes https://github.com/storj/storj/issues/4871

Change-Id: I2dd9c7364fd93796a05ef607bda9c39a741e6a89
2022-10-06 15:19:02 +00:00
Fadila Khadar
02924d0ded satellite/metainfo: temporary feature flag for listing query testing
Fixes: https://github.com/storj/storj/issues/5144

Change-Id: I7650f4d5dd0378e2246339e79710a695996a845c
2022-09-28 17:46:52 +00:00
Michal Niewrzal
289daf6787 satellite/metainfo: close project while testing
Project was not closed in test cases.

Change-Id: I700f671121433167fce7f350f84aa0559233c43c
2022-09-23 07:45:38 +00:00
Michal Niewrzal
4e71dd302a satellite/{metainfo,metabase}: prepare feature flag MultipleVersions
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.

Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
2022-09-21 13:07:13 +00:00
Michal Niewrzal
464ab43558 satellite/metainfo: fix deletion tests
Main issue with those tests was that for case where all objects were
uploaded at once (case "some nodes down" and "all nodes down").
Because all objects had the same name while upload each new object
was overwriting existing object. Because of that instead had several
objects to delete by test explicitly we had just 1. Test were not
failing because while overwriting existing object we were deleting it
but it was not what this test should do.

Change-Id: I602116f00be66589c7c0e68fe28c25e5c03e6b5d
2022-09-12 11:23:53 +00:00
Michal Niewrzal
eea3fac0d3 satellite/metabase: BeginObjectNextVersion returns object
We plan to replace metabase.BeginObjectExactVersion usage in
metainfo.BeginObject with metabase.BeginObjectNextVersion. To make this
switch as simple a possible would be nice to have the same results for
both methods. This change is extending return value for
BeginObjectNextVersion to whole object struct. Tests were also adjusted
to be more like metabase.BeginObjectExactVersion tests.

Part of https://github.com/storj/storj/issues/4871

Change-Id: I4db99d74af07e5a73757b55233e0bbdc7b99d565
2022-09-12 07:54:14 +00:00
Qweder93
98fb4c4f32 satellite/metainfo: added test for object TTL validation while upload
Closes https://github.com/storj/team-metainfo/issues/113

Change-Id: I6ff2a5f2cdab6fd42d94fbfc335084b927d86bea
2022-09-08 11:39:41 +03:00
Qweder93
2a1bcfed81 {satellite/metabase, satellite/metainfo, satellite/inspector} : Use metabase.GetObjectLastCommitted instead metabase.GetObjectExactVersion
adjusted metainfo, metabase and inspector methods to use GetObjectLastCommitted,
added test cases to confirm that this method will work correctly with version higher then 1

Closes https://github.com/storj/storj/issues/4869
Closes https://github.com/storj/storj/issues/4873

Change-Id: I8c338dcd1db82c141383f41339a295d54e2fa039
2022-09-08 07:27:22 +00:00
Michał Niewrzał
b722c29e77 satellite/metainfo: add missing metadata validation
We are missing metadata validation for UpdateObjectMetadata and
FinishCopyOject requests.

Change-Id: Idca6a4d1fe108e1593405fd3913442f5b69d09e7
2022-08-23 16:56:45 +02:00
Erik van Velzen
b5fc04af89 satellite/metainfo: usage limits for copy
Previously there was no realtime administration of the storage usage
during copies. Now there is.

Closes https://github.com/storj/storj/issues/4719

Change-Id: I0d536bf551d16208116c3aceac89ed590ec473bf
2022-07-25 12:35:27 +02:00
Fadila Khadar
4725a3878c satellite/metainfo: parallel deletion tests
- parallel deletion of 50 objects and their 50 copies (one copy per object)
This test is skipped because it's creating deadlocks that are not automatically retried on postgres
- parallel deletion of 1 object and its 50 copies.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Id7a28251c06bb12b5edcc88721f60bf7a4bc0492
2022-05-26 11:00:42 +00:00
Fadila Khadar
2d4760fd09 satellite/metainfo: BeginCopyObject and FinishCopyObject
Metainfo endpoints to use server-side copy.

Fixes https://github.com/storj/storj/issues/4475

Change-Id: Ided06aed9e6187d6d8f030e95dda019ba78fff95
2022-02-24 14:38:04 +01:00
Michał Niewrzał
b24379aa91 satellite/metainfo: use single testplanet instance in tests
Tests refactoring to reuse testplanet instance between some
of tests. This should decrease resources needed to run those
tests.

Change-Id: I98f3041ec23085d3903b19acd339904973319ec1
2022-02-16 15:27:26 +00:00
Michał Niewrzał
dffa7845f6 satellite/metainfo: split endpoint tests
We already split main code base, now we need to split test
to reflect new files structure (bucket/object/segment/other).

Fixes https://github.com/storj/team-metainfo/issues/12

Change-Id: Ica1054c4fc7df764483b03f204b4beba094df8e1
2022-02-02 07:46:14 +00:00