Commit Graph

34 Commits

Author SHA1 Message Date
Fadila Khadar
35f74b78e0 satellite/metabase: IterateLoopSegments accepts ranges
Fixes: https://github.com/storj/storj/issues/5207

Change-Id: I7872696068320987825de2d381f57ea503736e89
2022-10-13 14:12:40 +00:00
Fadila Khadar
7788170234 satellite/metabase: ListObjects
Current metainfo.ListObjects implementation is using metabase iterator to list objects.
In the non-recursive case, it used to retrieve all the corresponding rows and then discarded the entries that did not fit the listing request.
This can lead in some edge cases (each prefix contains more than batchsize objects/sub-prefixes) to make unecessary calls to the db.

This change defines the metabase.ListObjects and aims at retrieving only prefixes (but not objects under it) and objects by modifying the SQL query.

In this version, it is not optimized on the database side. Cockroach will still have to go through all rows under a prefix, so there is still room for improvement.

metainfo.ListObjects is not currently using this method as we would like to assess its performance on the QA satellite first.

Fixes https://github.com/storj/storj/issues/5088

Change-Id: Ied3a9210871871d9d4a3096888d3e40c2dceed61
2022-09-20 11:15:44 +00:00
Egon Elbre
546aa315fa satellite/metabase: add ListVerifySegments
We need a method for getting a list of segments from the metabase,
without converting the aliases and omitting all inline segments.

Change-Id: I26d919c675fc285ab03a35b327edd9b5c8bbe4b0
2022-09-12 17:32:29 +03:00
Michal Niewrzal
eea3fac0d3 satellite/metabase: BeginObjectNextVersion returns object
We plan to replace metabase.BeginObjectExactVersion usage in
metainfo.BeginObject with metabase.BeginObjectNextVersion. To make this
switch as simple a possible would be nice to have the same results for
both methods. This change is extending return value for
BeginObjectNextVersion to whole object struct. Tests were also adjusted
to be more like metabase.BeginObjectExactVersion tests.

Part of https://github.com/storj/storj/issues/4871

Change-Id: I4db99d74af07e5a73757b55233e0bbdc7b99d565
2022-09-12 07:54:14 +00:00
Qweder93
df5b0d0044 satellite/metabase: added method DeleteObjectLastCommitted
method similar to metabase.DeleteObjectExactVersion which will delete last committed object

Closes 	https://github.com/storj/storj/issues/4872

Change-Id: Ia9f8c227dc59575bf8ed297886b35536097028b4
2022-08-31 12:26:56 +00:00
Qweder93
2abe709b6e satellite/metabase: add metabase.GetObjectLatestVersion back
Restored GetObjectLatestVersion and renamed it to GetObjectLastCommitted
Add test cases to cover server-side copy

Closes https://github.com/storj/storj/issues/4866

Change-Id: I343b339a60152b8fb92fda97baf80bd8fe60d631
2022-08-11 22:14:38 +00:00
Michal Niewrzal
5458cbdad8 satellite/metabase/metabasetest: use interface for tests and benchmarks
metabasetest package utils can be used by both tests and benchmarks
if we will use interface TestingT from require package. This change
adjusts metabasettest.CreateObject method

Change-Id: I3c138e2ef9873b804ab5b3402804efa409397a9f
2022-08-03 13:08:06 +00:00
Michał Niewrzał
c934f45bfc satellite/metabase: drop IterateObjectsAllVersions
We are not using this method and most probably we
won't need to list objects with all statuses at once.
Removing for now.

Change-Id: I7aa0468c5f635ee2fb1fe51db382595c6343dd9c
2022-05-26 14:12:20 +00:00
Márton Elek
31560c94f8 satellite/metabase/test: extend default time diff for comparison
Tests are intermittently fail with similar error:

```
    --- FAIL: TestDeletePendingObject/Cockroach (15.85s)

    test.go:380:
        	Error Trace:	test.go:380
        	           				delete_test.go:221
        	Error:      	Should be zero, but was   metabase.DeleteObjectResult{
        	           	 	Objects: []metabase.Object{
        	           	 		{
        	           	 			ObjectStream: {ProjectID: {0x0f, 0x40, 0x70, 0x41, ...}, BucketName: "txxywyg4", ObjectKey: "\xbb+$\x17\x80\xc6\xcaC\xa3\xdb\xc3z*\xa8\xbe\xaf", Version: 1, ...},
        	           	- 			CreatedAt:    s"2022-05-20 14:40:15.995376773 +0200 CEST",
        	           	+ 			CreatedAt:    s"2022-05-20 14:40:21.04949 +0200 CEST",
        	           	 			ExpiresAt:    nil,
        	           	 			Status:       1,
        	           	 			... // 9 identical fields
        	           	 		},
        	           	 	},
        	           	 	Segments: {{RootPieceID: {0x01, 0x00, 0x00, 0x00, ...}, Pieces: {{...}}}, {RootPieceID: {0x01, 0x00, 0x00, 0x00, ...}, Pieces: {{...}}}},
        	           	 }
        	Test:       	TestDeletePendingObject/Cockroach/with_segments
        --- FAIL: TestDeletePendingObject/Cockroach/with_segments (0.68s)
```

Looks like we shouldn't have an assumption that all tests can be finished in 5 seconds, especially not in highly parallel environment.

These tests use `time.Now` at the beginning and compare the time saved in the database (usually filled by the database).

The difference shouldn't be higher than 20 seconds (before this commit: 5 seconds) which assumes that the records are saved in this timeframe...

Change-Id: Ia6f52897d13f88c6857c05d728bf8e72ab863c9b
2022-05-23 10:24:58 +00:00
Qweder93
50b826e582 satellite/metabase: drop GetProjectSegmentCount and related code
Resolves: https://github.com/storj/storj/issues/4744

Change-Id: I66b12e17de461242745e785dedd397fbe3b386fa
2022-05-17 19:01:51 +00:00
Michał Niewrzał
25e92fe443 satellite/metabase: deletion query as explicit transaction
Latest CRDB version did't work with our server-side copy deletion
query and we had to rewrite it.

Fixes https://github.com/storj/storj/issues/4655

Change-Id: I66d5c4abfc3c3f53dea8bc01ae11cd2b020e4289
2022-03-31 11:40:36 +00:00
Michał Niewrzał
18ab60dc92 satellite/metabase: drop GetSegmentByOffset
Methods was never used in production and it's not sure that
it will be used at all. Let's drop it and restore if will be needed.

Fixes https://github.com/storj/storj/issues/4480

Change-Id: Ifd780d0096b67be7e72dff84bdcf1d957e0b48b5
2022-03-30 09:03:48 +00:00
Fadila Khadar
93f8c6a159 satellite/metabase: require non-zero newStreamID for FinishCopyObject
We were not checking if we were provided an empty StreamID.
Furthermore, this changes returns the object copy with the correct createdAt field.

Change-Id: Iefc563c34ae9d8c1e233895155c1718bf905df91
2022-03-16 15:47:52 +00:00
Michał Niewrzał
1ef66bf872 satellite/metabase: adjust ListSegments to return copied segments correctly
Fixes https://github.com/storj/storj/issues/4479

Change-Id: I5a5f0378d14b81c819b5e76bf3b6e3540a41e861
2022-03-15 09:52:22 +00:00
Michał Niewrzał
c0297bae78 satellite/metabase: return object metadata with FinishCopyObject
Updates metadata and metainfo to return object metadata with
FinishCopyObject request.

https://github.com/storj/storj/issues/4474

Change-Id: I32cba5c20a943272e9b5964df1b3d6463ad212dc
2022-02-25 12:46:30 +00:00
Fadila Khadar
0358a08d6b satellite/metabase: BeginCopyObject and FinishCopyObject
Part of server-side copy implementation.

Creates the methods BeginCopyObject and FinishCopyObject in satellite/metabase/copy_object.go.

This commit makes it possible to copy objects and their segments in metabase.

https://github.com/storj/team-metainfo/issues/79

Change-Id: Icc64afce16e5e0e15b83c43cff36797bb9bd39bc
2022-02-17 13:37:39 +01:00
Michał Niewrzał
bc161794fc satellite/metabase: drop DeleteObjectLatestVersion method
This method was never used, except tests.

Change-Id: Idc1e69b2e2971995b5c4e6cf78a2b5fc69f39ad2
2022-02-02 14:33:48 +00:00
Michał Niewrzał
5041ee0abc satellite/metabase: drop GetObjectLatestVersion method
We don't have different version of object than 1 so at the moment
this method is not needed. Also using GetObjectExactVersion
should be slightly more performant.

Change-Id: I78235d8ae22594cc1d6345dabcc915f41cd7797b
2022-02-02 09:40:53 +00:00
Michał Niewrzał
60d03b9e42 satellite/metainfo: drop GetSegmentByLocation
Method is not used anymore. We need it during transition
from metainfo loop into segment loop.

Change-Id: Id32590ff9b02e06025efce1c32869e2cf9cd0bc5
2021-12-16 16:32:45 +01:00
Michał Niewrzał
de8464e848 satellite/metabase: add GetProjectSegmentCount method
This new method will be used with mechanism to limit number
of segments per project, similar to limiting buckets or bandwidth.
This is only one of multiple changes we will do to implement this
limitation.

Change-Id: Ia516c2a006ad1d7b4431d780679be9d809848f4b
2021-11-24 13:20:02 +00:00
Michał Niewrzał
f5f2398431 satellite/metabase: remove code related to part deletion
It is not used currently.

Change-Id: I32e9fa357814a3159ee8659a607cd6986f361fbf
2021-10-19 10:11:49 +00:00
crawter
5e4b196b11 satellite/metainfo: finish move object
Second method needed to perform server-side move. It updates
metadata key and nonce and all segments key and nonces.

Change-Id: Ia43b26622a13048269f0ae9e1524b345db112adb
2021-09-20 16:21:01 +03:00
crawter
07311cdd6f satellite/metainfo: begin move object added
First from two methods needed to perform server-side move. It gets
metadata key and nonce and all segments key and nonces and returns
all of that to uplink.

Change-Id: Ied2c79559e77d3f63091c4d61948f2d6a2147d67
2021-09-14 13:18:21 +00:00
Michał Niewrzał
67a33e4c06 satellite/metabase: drop unused IterateLoopStreams method
We were using this method for metainfo loop but now
segment loop is not using it so we can drop it.

Change-Id: I60c5b4f86a619259906d8c2ba76e665b8715be75
2021-09-09 14:12:44 +00:00
Kaloyan Raev
a767aed591
satellite/{metabase,metainfo}: require StreamID for UpdateObjectMetadata
This avoids corrupting objects if reuploads and metadata updates happen
concurrently.
2021-07-08 17:50:37 +03:00
Kaloyan Raev
a6086d421f satellite/metabase: implement SetObjectMetadataLatestVersion
It replaces UpdateMetadataObject that hasn't been used anywhere yet.

Change-Id: I1b2131acd75924c487a1c3ca099674251c5c9b56
2021-07-06 01:06:16 +00:00
Egon Elbre
0df4a27bf7 satellite/metabase: add method to get table statistics
In loop we need to start verifying it's correctness. This allows to
gather these stats.

Change-Id: I146fb50e2b3658b6f3c2682cdc1983e6abd73c29
2021-06-16 18:34:55 +03:00
JT Olio
6949dc0bac satellite/metaloop: missing monitoring on observers
Change-Id: I630fbb0448c8d08b426486b3e49abfbca03332a6
2021-06-15 13:39:13 +00:00
Michał Niewrzał
3af91e7a90 satellite/metabase: add iteration over segments
Method IterateLoopSegments can be used to iterate over all segments in metabase without touching objects.

Change-Id: I3cc0e783884b603b47ef3f8233e357aa8a391250
2021-05-26 09:59:33 +00:00
Michał Niewrzał
547a6e9930 satellite/metabase: add DeletePart method
Method will delete all segments for given part.

Change-Id: I11bdfdf30176749aee03cea0d1d2fa467f7cebea
2021-05-11 22:45:43 +02:00
Michał Niewrzał
033006403f satellite/metainfo: fix setting object ZombieDeletionDeadline
So far we were setting ZombieDeletionDeadline alwasy as nil and because of that DB default was never set. This change adds separate query for inserting object if deadline is not set.

Change-Id: I3d6a16570e7c74b5304e13edad8c7adcd021340c
2021-05-11 07:26:42 +00:00
Egon Elbre
6161436d8b satellite/metabase/metabasetest: sort results
Deletion results are not guaranteed to be sorted, hence we need to sort
them before comparing.

Change-Id: I6b571dd812d4256327c6c64fff74c780c62b2948
2021-05-07 12:36:02 +03:00
Michał Niewrzał
e8ef689929 satellite/metabase: add DeleteZombieObjects method
We will use this method to delete objects where zombie_deletion_deadline passed.

Change-Id: Id24322e40dad5b8bceb0f84080922c746c0b802d
2021-05-04 07:08:16 +00:00
Egon Elbre
948529a600 satellite/metabase/metabasetest: move test utils to a separate package
Change-Id: Iaa607996478497d5c97c6a6973d099b315b3abf0
2021-04-26 13:12:58 +03:00