JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,740 Commits 5 Branches 0 Tags 94 MiB
76199db3c7
Commit Graph

385 Commits

Author SHA1 Message Date
Michal Niewrzal
5a7bc9657d Merge 'master' branch 
Change-Id: If583132a821274dc4b78cf5f72b853ba8460c619
 2020-11-30 12:57:22 +01:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test 
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
 2020-11-30 10:49:23 +01:00
Michal Niewrzal
cfb45a785c satellite/metainfo/metabase: add TotalPlainSize to objects table 
We need to know how large is object in general and we shouldn't query
segments for that.

Change-Id: Ie316e1ec38bfc39065e6046b79b55d58f3ce9d53
 2020-11-25 14:35:28 +01:00
Michal Niewrzal
8ceef9f357 satellite/metainfo: temporary disable one assertion in test 
This is need to merge https://review.dev.storj.io/c/storj/uplink/+/3208
, after that this code will be back.

Change-Id: If9f2f1db95c7a1bba64a41c45a39bd3096a519e7
 2020-11-25 13:21:41 +00:00
Michal Niewrzal
983b1737c2 satellite/metainfo: implement IsBucketEmpty with metabase 
Use objects iterator to check if bucket is empty.

Change-Id: I590bb7258a47dd816f079039a1151df809dd2637
 2020-11-25 12:20:30 +00:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to 
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
 2020-11-24 17:50:33 +01:00
Egon Elbre
dad36179c6 satellite/metainfo/metabase: fix segment listing with cursor.Part 
Change-Id: I910888d435b11d2d4687ea0042998a1078cd6443
 2020-11-24 15:31:57 +02:00
Kaloyan Raev
67c210a64c satellite/metainfo: set list cursor to version 1 
Otherwise, if left to default version 0, the iterator will include the
cursor item in the result, which fails some tests.

Change-Id: I85103a36852477f371ec46c673a82c2e129978b7
 2020-11-24 10:17:42 +00:00
Michal Niewrzal
5be4d413b0 satellite/metainfo: set PlainSize for inline segment 
Plain size is required by metabase implementation.

Change-Id: I6739744bff4ce485e0066f2a49acdf38560d31fb
 2020-11-20 13:54:14 +01:00
Kaloyan Raev
05fe497a6f satellite/metainfo: add stream id when listing pending objects 
Change-Id: Idcdffd70b82c0fe4f79635fc193604b66101ef76
 2020-11-19 14:22:09 +02:00
Michal Niewrzal
1465539220 satellite/metainfo: ignore empty inline segments for backward 
compatibility

Change-Id: I2567d77e172391945dee24cea8e110fa1963d663
 2020-11-18 14:12:03 +00:00
Michal Niewrzal
70b1c7aa17 satellite/metainfo: check if EncryptedMetadataNonce was set 
It possible that old uplinks won't set it.

Change-Id: Ifd3c4d4a45c5636cfeb70b0a37bcf1305a097727
 2020-11-18 14:04:31 +01:00
Kaloyan Raev
a30f5d7ec0 satellite/metainfo: use Object.Status from ObjectListRequest 
Change-Id: Ie2e48d2eedf69bd0f04b7c9d6e7e217fec6b0267
 2020-11-18 13:31:52 +02:00
Michal Niewrzal
9bdc8ebf75 satellite/metainfo: adjust GetObject to handle 
EncryptedMetadataEncryptedKey

Change-Id: If159fdcc9f37e9449a7d8459a277366d6d6b5802
 2020-11-17 16:38:10 +00:00
Egon Elbre
72fed3b3fb satellite/metainfo/metabase: add slow non-recursive listing 
Change-Id: Ide60640400caea6474465af5224010fde60826c7
 2020-11-17 16:47:29 +02:00
Egon Elbre
5e7d47b9cd satellite/metainfo: code cleaning 
Rename iterator tests to iterator_test.go.
Add Delimiter to metabase package.

Change-Id: Icea6542b54176d716daba4afad901713f1dcac90
 2020-11-17 15:50:40 +02:00
Egon Elbre
2e426ef0a8 satellite/metainfo/metabase: iterator compatibility 
Old iterator returns object keys without prefixes, this helps to reduce
the bandwidth from the database. The endpoint also doesn't send the
prefixes.

Change-Id: I77d85dae671ee3a16abe75db14e19674e80abaf4
 2020-11-17 15:50:39 +02:00
Fadila Khadar
c9bbd83f96 satellite/metainfo/metabase: align object status to those already in use 
Change-Id: I33b2ce7282838e1172b7cb92d4e76169c96fddcb
 2020-11-17 14:07:42 +01:00
Michal Niewrzal
b7df41a0a1 satellite/metainfo/metabase: add EncryptedMetainfoEncryptedKey parameter 
to metabase

* EncryptedMetainfoEncryptedKey added to CommitSegment and
UpdateMetadata request
* EncryptedMetainfoEncryptedKey returned with GetObject response and all
delete responses
* EncryptedMetainfoEncryptedKey returned with object iterator results

Change-Id: I917541ab5f3e1863bc8f238d17a15fbf72a23025
 2020-11-17 10:19:12 +00:00
Egon Elbre
402cfcb7c6 satellite/metainfo/metabase: add prefix to iterator 
Change-Id: I2dd7b7917aed9def0be3f131ca60ab12f2077d83
 2020-11-16 17:38:59 +02:00
Egon Elbre
7aba265db0 satellite/metainfo/metabase: add status option to iteration 
Change-Id: Ie9d1a3704d8d08e3421ba2244a1f3be57d15ed8f
 2020-11-16 15:41:20 +02:00
Kaloyan Raev
a015f41927 satellite/metainfo: ListObjects to use Metabase API 
Change-Id: If75d04b9a44f08515be717a85f397b66f8a5c56f
 2020-11-16 13:27:17 +02:00
Michal Niewrzal
cd7d911b26 satellite/metainfo: implement ListSegments with metabase 
Change-Id: I78fb2400fed0bb01c5e05bd5bbcddef5e4a3bdf0
 2020-11-16 08:33:35 +00:00
Kaloyan Raev
ff32268310 satellite/metainfo/metabase: DeleteExpiredObjects 
Change-Id: I16293ce1a33f714aa005022daf3d40670cc66b04
 2020-11-13 15:31:31 +02:00
Fadila Khadar
a749ac9f47 satellite/metainfo/metabase: iterateObjectsAllVersions 
Iterate through all objects in a bucket
recursive, no prefix support

Change-Id: Ieaad9fbd0b5c1593554db9d543f4ee04851bac28
 2020-11-13 13:50:08 +01:00
Egon Elbre
8182d8a726 satellite/metainfo/metabase: use txutil.WithTx in delete 
Change-Id: I75965c4dcf57479ace2367f5d3069d785628e86a
 2020-11-12 14:38:36 +02:00
Egon Elbre
bc460cd62d satellite/metainfo/metabase: use txutil.WithTx in commit 
Change-Id: I84347291179c7042e22bda703a1c9bd80a1513a0
 2020-11-12 14:38:33 +02:00
Michal Niewrzal
9d25b3a7d3 satellite/metainfo/metabase: add ListSegments request 
Change-Id: Ie777a0f791685beab3973e76b10a9ce76398e7d7
 2020-11-12 11:46:17 +01:00
Michal Niewrzal
fd707392e7 satellite/metainfo: set RedundancyScheme for begin segment 
response

Multipart upload needs it for uploading parts.

Change-Id: I34886a9dabd17ecb7d55d4c6e19914a1c821f638
 2020-11-11 17:25:52 +00:00
Kaloyan Raev
9dcd0936d7 satellite/metainfo: MakeInlineSegment to use metabase 
Change-Id: Ib32c1d9e91e3f13b4e44abefb2b5d0352b64bd79
 2020-11-11 15:24:21 +02:00
Michal Niewrzal
3fe16f4003 satellite/metainfo: upload/download with metabase 
This change is adjusting metainfo endpoint to use metabase for uploading
and downloading remote objects. Inline segments will be added later.

Change-Id: I109d45bf644cd48096c47361043ebd8dfeaea0f3
 2020-11-11 12:13:52 +00:00
Fadila Khadar
2bd239bb75 satellite/metainfo/metabase: tests commit inline segment 
Change-Id: Ifad461041c7b9d8c78fc9f998a4e708853087b02
 2020-11-10 15:00:35 +00:00
Michal Niewrzal
48f1f7132e satellite/metainfo: set RS defaults 
While resolving conflicts with `master` I missed this change which is
needed e.g. to run storj-sim.

Change-Id: I56a548ed92b978510526c26c81af03051acfde2f
 2020-11-10 13:57:27 +01:00
Michal Niewrzal
7dde184cb5 Merge 'master' branch 
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
 2020-11-10 11:58:59 +00:00
Michal Niewrzal
92f9251074 satellite/metainfo/metabase: add GetSegmentByOffset request 
Change-Id: Iea2ce439ec1f7285e447d590c1297294204edc2e
 2020-11-10 11:33:34 +00:00
Kaloyan Raev
3ed4183e52 satellite/metainfo: delete object to use metabase 
Change-Id: I2ab63a719fdbc1f8a7fbb4ad73d51a2d2dcfadc6
 2020-11-10 09:55:23 +00:00
Moby von Briesen
db6bc6503d satellite/metainfo: Update metainfo RS config to more easily support multiple RS schemes. 
Make metainfo.RSConfig a valid pflag config value. This allows us to
configure the RSConfig as a string like k/m/o/n-shareSize, which makes
having multiple supported RS schemes easier in the future.

RS-related config values that are no longer needed have been removed
(MinTotalThreshold, MaxTotalThreshold, MaxBufferMem, Verify).

Change-Id: I0178ae467dcf4375c504e7202f31443d627c15e1
 2020-11-09 22:16:13 +00:00
Egon Elbre
184e1ffa9d satellite/metainfo/metabase: commit object without proofs 
Change-Id: I788111b9e5bdc961be9e731f8203e52ca9da9dca
 2020-11-09 15:18:07 +00:00
Egon Elbre
0a77deec81 satellite/metainfo/metabase: basic migrate 
Change-Id: Ia34f38e763af9ba88f75038b8ecfc9eb6550eae0
 2020-11-05 16:31:02 +02:00
Kaloyan Raev
56e6bc884c metabase: add DeleteObjectsAllVersions method 
Change-Id: I3252dd0aac595b233f4861c3c1375622326d955d
 2020-11-05 12:17:54 +02:00
Michal Niewrzal
2398afe985 satellite/metainfo/metabase: add GetLatestObjectLastSegment request 
Change-Id: I50c452e379f78864b38921f31e53a0554642ab02
 2020-11-05 08:20:34 +00:00
Kaloyan Raev
f986fdfe72 metabase: optimize delete segments 
Segments are deleted in a single DB query instead of a separate one for
each segment.

Change-Id: Icb9ae92b4b8405030c9d9ef57ba30bee6a35515c
 2020-11-04 11:59:56 +02:00
Michal Niewrzal
4134100d80 satellite/metainfo/metabase: add UpdateObjectMetadata request 
Change-Id: Iec73000e58f373294f37ef3e4470412d1c5f1340
 2020-11-03 11:08:41 +00:00
Kaloyan Raev
89cefa9045 metabase: add Objects info to DeleteObjectResult 
Change-Id: If7ed2ed0bbba08ad971c64852d51ff1a066c4f1e
 2020-11-03 12:45:49 +02:00
Kaloyan Raev
f7aa5b4e39 metabase: CommitObject return metabase.Object 
An immediate purpose is to be used in test cases for deleting objects

Change-Id: I481d3be5800e55da231ecb628ef7eb3ea011d196
 2020-11-03 09:57:44 +00:00
Kaloyan Raev
7e9ac2ec0a metabase: use storj.ErrObjectNotFound instead of custom error 
Change-Id: Ie90e438f7d7b58e6ea986359849dd8319433ae9f
 2020-11-03 09:35:22 +00:00
Michal Niewrzal
ae17ae73d7 satellite/metainfo/metabase: add metadata and nonce to CommitObject 
request

Change-Id: I584775800963fc2167c749f3da20475036fa5325
 2020-11-02 16:49:18 +01:00
Egon Elbre
7183dca6cb all: fix defers in loop 
defer should not be called in a loop.

Change-Id: Ifa5a25a56402814b974bcdfb0c2fce56df8e7e59
 2020-11-02 15:06:38 +02:00
Egon Elbre
716068a1e0 Merge branch 'master'. 
Change-Id: Ic14325edc291573582dce0cea3e04991a820b48b
 2020-11-02 13:02:01 +02:00
Michal Niewrzal
0205f0d807 satellite/metainfo: fix usage of types from internalpb 
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I34e121fce47371ee4cfd92cce03809520b68859f
 2020-10-30 16:03:45 +02:00