JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,951 Commits 5 Branches 0 Tags 94 MiB
d54ae9f10f
Commit Graph

63 Commits

Author SHA1 Message Date
Egon Elbre
8e47adfad5 satellite/metainfo/metabase: optimize a query 
Change-Id: I6395225f68cd8fbc8e3078c80ac6cb079c3c6a4f
 2021-01-19 15:37:03 +02:00
Michał Niewrzał
38beecc7ad satellite/metainfo: Override pending object on upload 
On upload we need to override pending and committed object. This change is adjusting DeleteObjectAllVersions to delete both.

Change-Id: Ib66c2af207c618119f7bf0de7fa9d3e5145d8641
 2021-01-14 15:16:02 +00:00
Kaloyan Raev
6dff40f5c5 Merge remote-tracking branch 'origin/main' into multipart-upload 
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/metainfo.go

Change-Id: Ib5c49f3c911c58319855a171f9ce73657da976d9
 2021-01-14 14:33:59 +02:00
Egon Elbre
0df0e980a0 satellite/metainfo/metabase: fix FixedSegmentSize calculation 
Commiting multipart object with same sized segments should always
end with FixedSegmentSize = -1.

Change-Id: Iedeafb763b62b2513de0a753dce20ddad5b32d90
 2021-01-11 14:47:49 +02:00
Egon Elbre
24833465e6 satellite/metainfo/metabase: avoid magic constant 
Change-Id: I4f01e38f67e18ae9cb9845a8e75a987acba66427
 2021-01-11 10:22:21 +00:00
Michał Niewrzał
ec88d21a3c Merge 'main' branch. 
Change-Id: I6e8162d1a6caf75e89c9f9c9f9522730aebf83ae
 2021-01-11 10:26:58 +01:00
Michał Niewrzał
71ca18134b satellite/metainfo: reject downloading a segment from multipart object 
Non-multipart uplink implementation is always trying to download object
by downloading last segment first (PartNumber=0, Index=-1) but this
approach won't work with multipart object. We need to reject such old
style request with reasonable message.

Change-Id: I9221e019933565a8d25136bdfef3e054320bac3d
 2021-01-11 08:04:33 +00:00
Egon Elbre
51731db121 satellite/orders: use smaller encrypted metadata 
Avoid using project uuid string representation, because
it uses more bandwidth.

This reduces the encrypted metadata size from 118 -> 97 bytes.

Change-Id: Ic53a81b83acc065f24f28cd404f9c0b1fe592594
 2021-01-08 16:40:31 +00:00
Michał Niewrzał
da316c27aa satellite/metainfo/metabase: disable PlainSize validation 
We need to accept segments with PlainSize == 0 to be backward compatible
for old uplinks.

Change-Id: I926abe751e9e0ad7e06e42a5301ab78a3f23aafa
 2021-01-07 14:14:20 +00:00
Kaloyan Raev
4fc61f7afa satellite/metainfo/metabase: allow commmitting objects > 2 GiB 
The total_plain_size and total_encrypted_size columns in the objects
table were set as INT4, which limits the size of committed objects to
just 2 GiB.

This patch migrates the DB to change the type of these fields to INT8.

Change-Id: Iad7e7b44a652e6c5b8e17b80588637bb48390fe6
 2021-01-06 18:23:03 +02:00
Michal Niewrzal
66d4d5eb48 satellite/metainfo/metabase: implement IterateObjectsAllVersions for 
pending/committed objects

Change-Id: Ibf390821b6a23919078de4f18c2653e308320904
 2020-12-22 10:27:09 +01:00
Michal Niewrzal
dad8360b39 satellite/metainfo/metabase: rename IterateObjectsAllVersions to 
IterateObjectsAllVersionsWithStatus

We need different implementation for IterateObjectsAllVersions because
we want to iterate over all object without specifying object status.
Existing method will have new name but implementation details are not
changed.

Change-Id: I01b987996772fa7f8fd73da9910d52db2d1aa0d7
 2020-12-21 16:47:32 +00:00
Kaloyan Raev
bafc6af992 ci: remove workaround for failing tests 
Change-Id: I3eb673fae6c81bee17d7437cb870d5f5ba6978d5
 2020-12-21 18:07:40 +02:00
Michal Niewrzal
f3ef8088e7 satellite/metainfo/metabase: add Verify method for Pieces 
This change adds Verify method for pieces to do some basic checks.

Change-Id: I0ff4313b594d2cb3aad7da545f940e10ee654b77
 2020-12-21 12:49:03 +00:00
Egon Elbre
1728e45e55 satellite/metainfo/metabase: optimize DeleteBucket 
Change-Id: If34cdeae0f688cb96717905fc8287c66ea3034be
 2020-12-16 14:39:33 +00:00
Egon Elbre
4706f01876 satellite/metainfo: add TestingAll{Segments,Objects} 
Change-Id: Ia758c119d5ebd7bbb21216a463c99c2e9afcdeb0
 2020-12-16 10:21:36 +00:00
Kaloyan Raev
4fba9921f6 satellite/metainfo/metabase: define ErrSegmentNotFound error class 
This makes it easier to callers of GetSegmentByPosition to determine if
the segment is missing.

Change-Id: I2d8546dddf07dcf790d2f7c08d308ed589b34f2f
 2020-12-15 15:48:41 +02:00
Kaloyan Raev
fc85179a19 satellite/metainfo: refactor SegmentLocation.Index to SegmentPosition 
Change-Id: Ic9403c8126712693326dd83d6ba4f3b84be3e0c7
 2020-12-14 13:35:53 +02:00
Michal Niewrzal
e7e6985ae9 satellite/metainfo/metabase: add UpdateSegmentPieces method 
We need to be able to update just remote_pieces column in DB. This is
needed at least for repair process.

Change-Id: I20dcc9b06babfefbbf102f32b1d14946379f26c2
 2020-12-14 10:25:09 +00:00
Michal Niewrzal
7e6e0d3e2e satellite/metainfo: metainfo loop implementation with metabase 
Change-Id: Iadac469519de605a88e624df23265289771b2006
 2020-12-11 16:15:57 +01:00
Fadila Khadar
15add3c752 satellite/metainfo: enable deletion of a pending object. 
Change-Id: I1ac24ad340d411c52b92ef18f51a684d9214ed47
 2020-12-10 13:56:14 +01:00
Egon Elbre
365410d10b satellite/metainfo/metabase: add commit with specifying a list of segments 
Change-Id: Ibb9999545691d150f36e20ce70ac00a3802ad7b2
 2020-12-09 15:16:38 +02:00
Michal Niewrzal
218bbeaffa Merge 'master' branch 
Change-Id: Ica5c25607a951076dd9f77e35e308062f71ce3f0
 2020-12-07 15:05:52 +01:00
Kaloyan Raev
a7685f50c9 satellite/metainfo/metabase: set maxParts to MaxListLimit if greater 
We should set the client requested maxParts to MaxListLimit if it is
greater than that value instead of returning an error.

MinIO default value for maxParts is 10,000 while the satellite's
MaxListLimit is 1,000. If we return an error, the ListParts with default
maxParts will throw an error.

Change-Id: I06739e1d8d8f96803eba491585395da0443aec04
 2020-12-03 11:32:07 +00:00
Fadila Khadar
fd7cc20a7e satellite/metainfo/metabase: add DeletePendingObject 
Change-Id: I7ba7d3d7f179712940e91b05501310279bbb97a4
 2020-12-03 10:21:04 +00:00
Michal Niewrzal
83e7cd2a40 satellite/metainfo/metabase: replace SQL substring with go code 
We have some issues with SUBSTRING function on cockroachdb so for now we
are removing it from SQL query and replacing with go code.

Change-Id: I5be921211067d42e7d1a4997076bcfdbed9617a1
 2020-12-02 10:47:18 +01:00
Michal Niewrzal
e4c4ab06ba satellite/metainfo: add testing methods for getting objects and segments 
Change-Id: I4229d80090609df27d998a90bdb3137469dba042
 2020-12-01 12:44:08 +00:00
Michal Niewrzal
cfb45a785c satellite/metainfo/metabase: add TotalPlainSize to objects table 
We need to know how large is object in general and we shouldn't query
segments for that.

Change-Id: Ie316e1ec38bfc39065e6046b79b55d58f3ce9d53
 2020-11-25 14:35:28 +01:00
Michal Niewrzal
983b1737c2 satellite/metainfo: implement IsBucketEmpty with metabase 
Use objects iterator to check if bucket is empty.

Change-Id: I590bb7258a47dd816f079039a1151df809dd2637
 2020-11-25 12:20:30 +00:00
Egon Elbre
dad36179c6 satellite/metainfo/metabase: fix segment listing with cursor.Part 
Change-Id: I910888d435b11d2d4687ea0042998a1078cd6443
 2020-11-24 15:31:57 +02:00
Egon Elbre
72fed3b3fb satellite/metainfo/metabase: add slow non-recursive listing 
Change-Id: Ide60640400caea6474465af5224010fde60826c7
 2020-11-17 16:47:29 +02:00
Egon Elbre
5e7d47b9cd satellite/metainfo: code cleaning 
Rename iterator tests to iterator_test.go.
Add Delimiter to metabase package.

Change-Id: Icea6542b54176d716daba4afad901713f1dcac90
 2020-11-17 15:50:40 +02:00
Egon Elbre
2e426ef0a8 satellite/metainfo/metabase: iterator compatibility 
Old iterator returns object keys without prefixes, this helps to reduce
the bandwidth from the database. The endpoint also doesn't send the
prefixes.

Change-Id: I77d85dae671ee3a16abe75db14e19674e80abaf4
 2020-11-17 15:50:39 +02:00
Fadila Khadar
c9bbd83f96 satellite/metainfo/metabase: align object status to those already in use 
Change-Id: I33b2ce7282838e1172b7cb92d4e76169c96fddcb
 2020-11-17 14:07:42 +01:00
Michal Niewrzal
b7df41a0a1 satellite/metainfo/metabase: add EncryptedMetainfoEncryptedKey parameter 
to metabase

* EncryptedMetainfoEncryptedKey added to CommitSegment and
UpdateMetadata request
* EncryptedMetainfoEncryptedKey returned with GetObject response and all
delete responses
* EncryptedMetainfoEncryptedKey returned with object iterator results

Change-Id: I917541ab5f3e1863bc8f238d17a15fbf72a23025
 2020-11-17 10:19:12 +00:00
Egon Elbre
402cfcb7c6 satellite/metainfo/metabase: add prefix to iterator 
Change-Id: I2dd7b7917aed9def0be3f131ca60ab12f2077d83
 2020-11-16 17:38:59 +02:00
Egon Elbre
7aba265db0 satellite/metainfo/metabase: add status option to iteration 
Change-Id: Ie9d1a3704d8d08e3421ba2244a1f3be57d15ed8f
 2020-11-16 15:41:20 +02:00
Kaloyan Raev
a015f41927 satellite/metainfo: ListObjects to use Metabase API 
Change-Id: If75d04b9a44f08515be717a85f397b66f8a5c56f
 2020-11-16 13:27:17 +02:00
Kaloyan Raev
ff32268310 satellite/metainfo/metabase: DeleteExpiredObjects 
Change-Id: I16293ce1a33f714aa005022daf3d40670cc66b04
 2020-11-13 15:31:31 +02:00
Fadila Khadar
a749ac9f47 satellite/metainfo/metabase: iterateObjectsAllVersions 
Iterate through all objects in a bucket
recursive, no prefix support

Change-Id: Ieaad9fbd0b5c1593554db9d543f4ee04851bac28
 2020-11-13 13:50:08 +01:00
Egon Elbre
8182d8a726 satellite/metainfo/metabase: use txutil.WithTx in delete 
Change-Id: I75965c4dcf57479ace2367f5d3069d785628e86a
 2020-11-12 14:38:36 +02:00
Egon Elbre
bc460cd62d satellite/metainfo/metabase: use txutil.WithTx in commit 
Change-Id: I84347291179c7042e22bda703a1c9bd80a1513a0
 2020-11-12 14:38:33 +02:00
Michal Niewrzal
9d25b3a7d3 satellite/metainfo/metabase: add ListSegments request 
Change-Id: Ie777a0f791685beab3973e76b10a9ce76398e7d7
 2020-11-12 11:46:17 +01:00
Kaloyan Raev
9dcd0936d7 satellite/metainfo: MakeInlineSegment to use metabase 
Change-Id: Ib32c1d9e91e3f13b4e44abefb2b5d0352b64bd79
 2020-11-11 15:24:21 +02:00
Michal Niewrzal
3fe16f4003 satellite/metainfo: upload/download with metabase 
This change is adjusting metainfo endpoint to use metabase for uploading
and downloading remote objects. Inline segments will be added later.

Change-Id: I109d45bf644cd48096c47361043ebd8dfeaea0f3
 2020-11-11 12:13:52 +00:00
Fadila Khadar
2bd239bb75 satellite/metainfo/metabase: tests commit inline segment 
Change-Id: Ifad461041c7b9d8c78fc9f998a4e708853087b02
 2020-11-10 15:00:35 +00:00
Michal Niewrzal
92f9251074 satellite/metainfo/metabase: add GetSegmentByOffset request 
Change-Id: Iea2ce439ec1f7285e447d590c1297294204edc2e
 2020-11-10 11:33:34 +00:00
Egon Elbre
184e1ffa9d satellite/metainfo/metabase: commit object without proofs 
Change-Id: I788111b9e5bdc961be9e731f8203e52ca9da9dca
 2020-11-09 15:18:07 +00:00
Egon Elbre
0a77deec81 satellite/metainfo/metabase: basic migrate 
Change-Id: Ia34f38e763af9ba88f75038b8ecfc9eb6550eae0
 2020-11-05 16:31:02 +02:00
Kaloyan Raev
56e6bc884c metabase: add DeleteObjectsAllVersions method 
Change-Id: I3252dd0aac595b233f4861c3c1375622326d955d
 2020-11-05 12:17:54 +02:00