JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,813 Commits 5 Branches 0 Tags 94 MiB
57f374af24
Commit Graph

410 Commits

Author SHA1 Message Date
Kaloyan Raev
fc85179a19 satellite/metainfo: refactor SegmentLocation.Index to SegmentPosition 
Change-Id: Ic9403c8126712693326dd83d6ba4f3b84be3e0c7
 2020-12-14 13:35:53 +02:00
Kaloyan Raev
7d8f19e94d satellite/metainfo: metainfo loop should yield StreamID for segments 
Change-Id: If6c86add75ce79ffcfe95353225719c7d4b5a459
 2020-12-14 13:04:38 +02:00
Michal Niewrzal
e7e6985ae9 satellite/metainfo/metabase: add UpdateSegmentPieces method 
We need to be able to update just remote_pieces column in DB. This is
needed at least for repair process.

Change-Id: I20dcc9b06babfefbbf102f32b1d14946379f26c2
 2020-12-14 10:25:09 +00:00
Kaloyan Raev
2bb010e7c5 cmd: remove segment reaper 
It was designed to detect and remove zombie segments in the PointerDB.
This tool should be not relevant with the MetabaseDB anymore.

Change-Id: I112552203b1329a5a659f69a0043eb1f8dadb551
 2020-12-14 09:36:37 +00:00
Michal Niewrzal
7e6e0d3e2e satellite/metainfo: metainfo loop implementation with metabase 
Change-Id: Iadac469519de605a88e624df23265289771b2006
 2020-12-11 16:15:57 +01:00
Michal Niewrzal
b3acc1101a Merge 'master' branch 
Change-Id: Iee99400c7095770e61cde94b3b2c8eb0ddec463d
 2020-12-10 15:42:52 +01:00
Michal Niewrzal
c2a97aeb14 satellite/satellitedb: add ListAllBuckets method 
We need to be able to list all buckets in DB without knowing project ID.
This method will be used to list buckets for metainfo loop
implementation based on metabase.

Change-Id: Iac75af0eee4f31e80a15577575a8249cbca787b2
 2020-12-10 14:19:27 +00:00
Fadila Khadar
15add3c752 satellite/metainfo: enable deletion of a pending object. 
Change-Id: I1ac24ad340d411c52b92ef18f51a684d9214ed47
 2020-12-10 13:56:14 +01:00
Egon Elbre
365410d10b satellite/metainfo/metabase: add commit with specifying a list of segments 
Change-Id: Ibb9999545691d150f36e20ce70ac00a3802ad7b2
 2020-12-09 15:16:38 +02:00
Michal Niewrzal
f077564bb7 satellite/metainfo: setup access to metabase in metainfo loop 
This is initial change to replace metainfo loop implementation with
metabase.

Change-Id: Ib23cb047bbbe6cff146305935a16ba833d40351a
 2020-12-09 09:50:04 +01:00
Kaloyan Raev
3f5e07fb1f satellite/metainfo: more tests fixed 
- TestBucketNameValidation
- TestBatch
- TestCommitObjectMetadataSize
- TestIDs

TestOverwriteZombieSegments is removed as not relevant to metabase.

Change-Id: I13cf5abe342089960628f185061303fd4f9d09a4
 2020-12-08 14:00:24 +02:00
Kaloyan Raev
d0d0a192cd satellite/metainfo: fix TestEndpoint_DeleteObjectPieces tests 
This also removes the
TestEndpoint_DeleteObjectPieces_ObjectWithoutLastSegment test case as it
does not seem relevant to metabase.

Change-Id: I06a0ecaa8232c10c15e433517a7ba056933bf858
 2020-12-08 10:22:05 +00:00
Michal Niewrzal
fb5244e8f1 satellite/metainfo: migrate GetObjectIP endpoint method to metabase 
Change-Id: I0da346fc2ce84787a09d2bb7c4ffc02a5d8a3b2f
 2020-12-08 08:50:15 +00:00
Michal Niewrzal
218bbeaffa Merge 'master' branch 
Change-Id: Ica5c25607a951076dd9f77e35e308062f71ce3f0
 2020-12-07 15:05:52 +01:00
Stefan Benten
494bd5db81
all: golangci-lint v1.33.0 fixes (#3985) 2020-12-05 17:01:42 +01:00
Ethan Adams
f90ea10a4a
Allow for DB application names per process. (#3983) 2020-12-04 11:24:39 +01:00
Kaloyan Raev
a7685f50c9 satellite/metainfo/metabase: set maxParts to MaxListLimit if greater 
We should set the client requested maxParts to MaxListLimit if it is
greater than that value instead of returning an error.

MinIO default value for maxParts is 10,000 while the satellite's
MaxListLimit is 1,000. If we return an error, the ListParts with default
maxParts will throw an error.

Change-Id: I06739e1d8d8f96803eba491585395da0443aec04
 2020-12-03 11:32:07 +00:00
Fadila Khadar
fd7cc20a7e satellite/metainfo/metabase: add DeletePendingObject 
Change-Id: I7ba7d3d7f179712940e91b05501310279bbb97a4
 2020-12-03 10:21:04 +00:00
Kaloyan Raev
bc25cc8ddd satellite/metainfo: add encryption info in metadata of list items 
This is needed to resolve 'encryption type 0 is not supported' errors in
gateway.

Change-Id: I33c486968d5a1a1310b82e9778a8ce01429f7043
 2020-12-02 13:34:41 +02:00
Michal Niewrzal
83e7cd2a40 satellite/metainfo/metabase: replace SQL substring with go code 
We have some issues with SUBSTRING function on cockroachdb so for now we
are removing it from SQL query and replacing with go code.

Change-Id: I5be921211067d42e7d1a4997076bcfdbed9617a1
 2020-12-02 10:47:18 +01:00
Kaloyan Raev
e8378a87bf satellite/metainfo: include redundancy in satStreamID of ListObjects 
This fixes the issue with using UploadID from the ListMultipartUploads
response.

Change-Id: If077a5804d03cc139f75febf472d5fb73f0f7929
 2020-12-01 15:11:34 +00:00
Michal Niewrzal
d5c0264163 satellite/metainfo: implement deleteBucketObjects with metabase objects 
iterator

This method replaces `deleteByPrefix` as at the moment only function of
this method was to delete objects in a bucket.

Change-Id: I5266103672003fbd64f3847f53760b1ba0016fe2
 2020-12-01 14:54:23 +00:00
Michal Niewrzal
f08e34f15e satellite/metainfo: fix listing objects when cursor is specified 
We need to combine prefix + cursor to iterate objects correctly.

Change-Id: I2f4a29ba3ee65ce52eb1e8b88a3b09cf93e0b2bb
 2020-12-01 15:01:44 +01:00
Michal Niewrzal
e4c4ab06ba satellite/metainfo: add testing methods for getting objects and segments 
Change-Id: I4229d80090609df27d998a90bdb3137469dba042
 2020-12-01 12:44:08 +00:00
Kaloyan Raev
46839b992a satellite/metainfo: expired deletion service to use Metabase 
It also sets the Expires time correctly in the Metabase when uploading
objects.

Change-Id: Iec1b3ec8d11346a91dfc2ba23a1b08edec4a84d3
 2020-12-01 09:12:14 +00:00
Michal Niewrzal
5a7bc9657d Merge 'master' branch 
Change-Id: If583132a821274dc4b78cf5f72b853ba8460c619
 2020-11-30 12:57:22 +01:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test 
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
 2020-11-30 10:49:23 +01:00
Michal Niewrzal
cfb45a785c satellite/metainfo/metabase: add TotalPlainSize to objects table 
We need to know how large is object in general and we shouldn't query
segments for that.

Change-Id: Ie316e1ec38bfc39065e6046b79b55d58f3ce9d53
 2020-11-25 14:35:28 +01:00
Michal Niewrzal
8ceef9f357 satellite/metainfo: temporary disable one assertion in test 
This is need to merge https://review.dev.storj.io/c/storj/uplink/+/3208
, after that this code will be back.

Change-Id: If9f2f1db95c7a1bba64a41c45a39bd3096a519e7
 2020-11-25 13:21:41 +00:00
Michal Niewrzal
983b1737c2 satellite/metainfo: implement IsBucketEmpty with metabase 
Use objects iterator to check if bucket is empty.

Change-Id: I590bb7258a47dd816f079039a1151df809dd2637
 2020-11-25 12:20:30 +00:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to 
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
 2020-11-24 17:50:33 +01:00
Egon Elbre
dad36179c6 satellite/metainfo/metabase: fix segment listing with cursor.Part 
Change-Id: I910888d435b11d2d4687ea0042998a1078cd6443
 2020-11-24 15:31:57 +02:00
Kaloyan Raev
67c210a64c satellite/metainfo: set list cursor to version 1 
Otherwise, if left to default version 0, the iterator will include the
cursor item in the result, which fails some tests.

Change-Id: I85103a36852477f371ec46c673a82c2e129978b7
 2020-11-24 10:17:42 +00:00
Michal Niewrzal
5be4d413b0 satellite/metainfo: set PlainSize for inline segment 
Plain size is required by metabase implementation.

Change-Id: I6739744bff4ce485e0066f2a49acdf38560d31fb
 2020-11-20 13:54:14 +01:00
Kaloyan Raev
05fe497a6f satellite/metainfo: add stream id when listing pending objects 
Change-Id: Idcdffd70b82c0fe4f79635fc193604b66101ef76
 2020-11-19 14:22:09 +02:00
Michal Niewrzal
1465539220 satellite/metainfo: ignore empty inline segments for backward 
compatibility

Change-Id: I2567d77e172391945dee24cea8e110fa1963d663
 2020-11-18 14:12:03 +00:00
Michal Niewrzal
70b1c7aa17 satellite/metainfo: check if EncryptedMetadataNonce was set 
It possible that old uplinks won't set it.

Change-Id: Ifd3c4d4a45c5636cfeb70b0a37bcf1305a097727
 2020-11-18 14:04:31 +01:00
Kaloyan Raev
a30f5d7ec0 satellite/metainfo: use Object.Status from ObjectListRequest 
Change-Id: Ie2e48d2eedf69bd0f04b7c9d6e7e217fec6b0267
 2020-11-18 13:31:52 +02:00
Michal Niewrzal
9bdc8ebf75 satellite/metainfo: adjust GetObject to handle 
EncryptedMetadataEncryptedKey

Change-Id: If159fdcc9f37e9449a7d8459a277366d6d6b5802
 2020-11-17 16:38:10 +00:00
Egon Elbre
72fed3b3fb satellite/metainfo/metabase: add slow non-recursive listing 
Change-Id: Ide60640400caea6474465af5224010fde60826c7
 2020-11-17 16:47:29 +02:00
Egon Elbre
5e7d47b9cd satellite/metainfo: code cleaning 
Rename iterator tests to iterator_test.go.
Add Delimiter to metabase package.

Change-Id: Icea6542b54176d716daba4afad901713f1dcac90
 2020-11-17 15:50:40 +02:00
Egon Elbre
2e426ef0a8 satellite/metainfo/metabase: iterator compatibility 
Old iterator returns object keys without prefixes, this helps to reduce
the bandwidth from the database. The endpoint also doesn't send the
prefixes.

Change-Id: I77d85dae671ee3a16abe75db14e19674e80abaf4
 2020-11-17 15:50:39 +02:00
Fadila Khadar
c9bbd83f96 satellite/metainfo/metabase: align object status to those already in use 
Change-Id: I33b2ce7282838e1172b7cb92d4e76169c96fddcb
 2020-11-17 14:07:42 +01:00
Michal Niewrzal
b7df41a0a1 satellite/metainfo/metabase: add EncryptedMetainfoEncryptedKey parameter 
to metabase

* EncryptedMetainfoEncryptedKey added to CommitSegment and
UpdateMetadata request
* EncryptedMetainfoEncryptedKey returned with GetObject response and all
delete responses
* EncryptedMetainfoEncryptedKey returned with object iterator results

Change-Id: I917541ab5f3e1863bc8f238d17a15fbf72a23025
 2020-11-17 10:19:12 +00:00
Egon Elbre
402cfcb7c6 satellite/metainfo/metabase: add prefix to iterator 
Change-Id: I2dd7b7917aed9def0be3f131ca60ab12f2077d83
 2020-11-16 17:38:59 +02:00
Egon Elbre
7aba265db0 satellite/metainfo/metabase: add status option to iteration 
Change-Id: Ie9d1a3704d8d08e3421ba2244a1f3be57d15ed8f
 2020-11-16 15:41:20 +02:00
Kaloyan Raev
a015f41927 satellite/metainfo: ListObjects to use Metabase API 
Change-Id: If75d04b9a44f08515be717a85f397b66f8a5c56f
 2020-11-16 13:27:17 +02:00
Michal Niewrzal
cd7d911b26 satellite/metainfo: implement ListSegments with metabase 
Change-Id: I78fb2400fed0bb01c5e05bd5bbcddef5e4a3bdf0
 2020-11-16 08:33:35 +00:00
Kaloyan Raev
ff32268310 satellite/metainfo/metabase: DeleteExpiredObjects 
Change-Id: I16293ce1a33f714aa005022daf3d40670cc66b04
 2020-11-13 15:31:31 +02:00
Fadila Khadar
a749ac9f47 satellite/metainfo/metabase: iterateObjectsAllVersions 
Iterate through all objects in a bucket
recursive, no prefix support

Change-Id: Ieaad9fbd0b5c1593554db9d543f4ee04851bac28
 2020-11-13 13:50:08 +01:00