JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,712 Commits 5 Branches 0 Tags 94 MiB
dad36179c6
Commit Graph

377 Commits

Author SHA1 Message Date
Egon Elbre
dad36179c6 satellite/metainfo/metabase: fix segment listing with cursor.Part 
Change-Id: I910888d435b11d2d4687ea0042998a1078cd6443
 2020-11-24 15:31:57 +02:00
Kaloyan Raev
67c210a64c satellite/metainfo: set list cursor to version 1 
Otherwise, if left to default version 0, the iterator will include the
cursor item in the result, which fails some tests.

Change-Id: I85103a36852477f371ec46c673a82c2e129978b7
 2020-11-24 10:17:42 +00:00
Michal Niewrzal
5be4d413b0 satellite/metainfo: set PlainSize for inline segment 
Plain size is required by metabase implementation.

Change-Id: I6739744bff4ce485e0066f2a49acdf38560d31fb
 2020-11-20 13:54:14 +01:00
Kaloyan Raev
05fe497a6f satellite/metainfo: add stream id when listing pending objects 
Change-Id: Idcdffd70b82c0fe4f79635fc193604b66101ef76
 2020-11-19 14:22:09 +02:00
Michal Niewrzal
1465539220 satellite/metainfo: ignore empty inline segments for backward 
compatibility

Change-Id: I2567d77e172391945dee24cea8e110fa1963d663
 2020-11-18 14:12:03 +00:00
Michal Niewrzal
70b1c7aa17 satellite/metainfo: check if EncryptedMetadataNonce was set 
It possible that old uplinks won't set it.

Change-Id: Ifd3c4d4a45c5636cfeb70b0a37bcf1305a097727
 2020-11-18 14:04:31 +01:00
Kaloyan Raev
a30f5d7ec0 satellite/metainfo: use Object.Status from ObjectListRequest 
Change-Id: Ie2e48d2eedf69bd0f04b7c9d6e7e217fec6b0267
 2020-11-18 13:31:52 +02:00
Michal Niewrzal
9bdc8ebf75 satellite/metainfo: adjust GetObject to handle 
EncryptedMetadataEncryptedKey

Change-Id: If159fdcc9f37e9449a7d8459a277366d6d6b5802
 2020-11-17 16:38:10 +00:00
Egon Elbre
72fed3b3fb satellite/metainfo/metabase: add slow non-recursive listing 
Change-Id: Ide60640400caea6474465af5224010fde60826c7
 2020-11-17 16:47:29 +02:00
Egon Elbre
5e7d47b9cd satellite/metainfo: code cleaning 
Rename iterator tests to iterator_test.go.
Add Delimiter to metabase package.

Change-Id: Icea6542b54176d716daba4afad901713f1dcac90
 2020-11-17 15:50:40 +02:00
Egon Elbre
2e426ef0a8 satellite/metainfo/metabase: iterator compatibility 
Old iterator returns object keys without prefixes, this helps to reduce
the bandwidth from the database. The endpoint also doesn't send the
prefixes.

Change-Id: I77d85dae671ee3a16abe75db14e19674e80abaf4
 2020-11-17 15:50:39 +02:00
Fadila Khadar
c9bbd83f96 satellite/metainfo/metabase: align object status to those already in use 
Change-Id: I33b2ce7282838e1172b7cb92d4e76169c96fddcb
 2020-11-17 14:07:42 +01:00
Michal Niewrzal
b7df41a0a1 satellite/metainfo/metabase: add EncryptedMetainfoEncryptedKey parameter 
to metabase

* EncryptedMetainfoEncryptedKey added to CommitSegment and
UpdateMetadata request
* EncryptedMetainfoEncryptedKey returned with GetObject response and all
delete responses
* EncryptedMetainfoEncryptedKey returned with object iterator results

Change-Id: I917541ab5f3e1863bc8f238d17a15fbf72a23025
 2020-11-17 10:19:12 +00:00
Egon Elbre
402cfcb7c6 satellite/metainfo/metabase: add prefix to iterator 
Change-Id: I2dd7b7917aed9def0be3f131ca60ab12f2077d83
 2020-11-16 17:38:59 +02:00
Egon Elbre
7aba265db0 satellite/metainfo/metabase: add status option to iteration 
Change-Id: Ie9d1a3704d8d08e3421ba2244a1f3be57d15ed8f
 2020-11-16 15:41:20 +02:00
Kaloyan Raev
a015f41927 satellite/metainfo: ListObjects to use Metabase API 
Change-Id: If75d04b9a44f08515be717a85f397b66f8a5c56f
 2020-11-16 13:27:17 +02:00
Michal Niewrzal
cd7d911b26 satellite/metainfo: implement ListSegments with metabase 
Change-Id: I78fb2400fed0bb01c5e05bd5bbcddef5e4a3bdf0
 2020-11-16 08:33:35 +00:00
Kaloyan Raev
ff32268310 satellite/metainfo/metabase: DeleteExpiredObjects 
Change-Id: I16293ce1a33f714aa005022daf3d40670cc66b04
 2020-11-13 15:31:31 +02:00
Fadila Khadar
a749ac9f47 satellite/metainfo/metabase: iterateObjectsAllVersions 
Iterate through all objects in a bucket
recursive, no prefix support

Change-Id: Ieaad9fbd0b5c1593554db9d543f4ee04851bac28
 2020-11-13 13:50:08 +01:00
Egon Elbre
8182d8a726 satellite/metainfo/metabase: use txutil.WithTx in delete 
Change-Id: I75965c4dcf57479ace2367f5d3069d785628e86a
 2020-11-12 14:38:36 +02:00
Egon Elbre
bc460cd62d satellite/metainfo/metabase: use txutil.WithTx in commit 
Change-Id: I84347291179c7042e22bda703a1c9bd80a1513a0
 2020-11-12 14:38:33 +02:00
Michal Niewrzal
9d25b3a7d3 satellite/metainfo/metabase: add ListSegments request 
Change-Id: Ie777a0f791685beab3973e76b10a9ce76398e7d7
 2020-11-12 11:46:17 +01:00
Michal Niewrzal
fd707392e7 satellite/metainfo: set RedundancyScheme for begin segment 
response

Multipart upload needs it for uploading parts.

Change-Id: I34886a9dabd17ecb7d55d4c6e19914a1c821f638
 2020-11-11 17:25:52 +00:00
Kaloyan Raev
9dcd0936d7 satellite/metainfo: MakeInlineSegment to use metabase 
Change-Id: Ib32c1d9e91e3f13b4e44abefb2b5d0352b64bd79
 2020-11-11 15:24:21 +02:00
Michal Niewrzal
3fe16f4003 satellite/metainfo: upload/download with metabase 
This change is adjusting metainfo endpoint to use metabase for uploading
and downloading remote objects. Inline segments will be added later.

Change-Id: I109d45bf644cd48096c47361043ebd8dfeaea0f3
 2020-11-11 12:13:52 +00:00
Fadila Khadar
2bd239bb75 satellite/metainfo/metabase: tests commit inline segment 
Change-Id: Ifad461041c7b9d8c78fc9f998a4e708853087b02
 2020-11-10 15:00:35 +00:00
Michal Niewrzal
48f1f7132e satellite/metainfo: set RS defaults 
While resolving conflicts with `master` I missed this change which is
needed e.g. to run storj-sim.

Change-Id: I56a548ed92b978510526c26c81af03051acfde2f
 2020-11-10 13:57:27 +01:00
Michal Niewrzal
7dde184cb5 Merge 'master' branch 
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
 2020-11-10 11:58:59 +00:00
Michal Niewrzal
92f9251074 satellite/metainfo/metabase: add GetSegmentByOffset request 
Change-Id: Iea2ce439ec1f7285e447d590c1297294204edc2e
 2020-11-10 11:33:34 +00:00
Kaloyan Raev
3ed4183e52 satellite/metainfo: delete object to use metabase 
Change-Id: I2ab63a719fdbc1f8a7fbb4ad73d51a2d2dcfadc6
 2020-11-10 09:55:23 +00:00
Egon Elbre
184e1ffa9d satellite/metainfo/metabase: commit object without proofs 
Change-Id: I788111b9e5bdc961be9e731f8203e52ca9da9dca
 2020-11-09 15:18:07 +00:00
Egon Elbre
0a77deec81 satellite/metainfo/metabase: basic migrate 
Change-Id: Ia34f38e763af9ba88f75038b8ecfc9eb6550eae0
 2020-11-05 16:31:02 +02:00
Kaloyan Raev
56e6bc884c metabase: add DeleteObjectsAllVersions method 
Change-Id: I3252dd0aac595b233f4861c3c1375622326d955d
 2020-11-05 12:17:54 +02:00
Michal Niewrzal
2398afe985 satellite/metainfo/metabase: add GetLatestObjectLastSegment request 
Change-Id: I50c452e379f78864b38921f31e53a0554642ab02
 2020-11-05 08:20:34 +00:00
Kaloyan Raev
f986fdfe72 metabase: optimize delete segments 
Segments are deleted in a single DB query instead of a separate one for
each segment.

Change-Id: Icb9ae92b4b8405030c9d9ef57ba30bee6a35515c
 2020-11-04 11:59:56 +02:00
Michal Niewrzal
4134100d80 satellite/metainfo/metabase: add UpdateObjectMetadata request 
Change-Id: Iec73000e58f373294f37ef3e4470412d1c5f1340
 2020-11-03 11:08:41 +00:00
Kaloyan Raev
89cefa9045 metabase: add Objects info to DeleteObjectResult 
Change-Id: If7ed2ed0bbba08ad971c64852d51ff1a066c4f1e
 2020-11-03 12:45:49 +02:00
Kaloyan Raev
f7aa5b4e39 metabase: CommitObject return metabase.Object 
An immediate purpose is to be used in test cases for deleting objects

Change-Id: I481d3be5800e55da231ecb628ef7eb3ea011d196
 2020-11-03 09:57:44 +00:00
Kaloyan Raev
7e9ac2ec0a metabase: use storj.ErrObjectNotFound instead of custom error 
Change-Id: Ie90e438f7d7b58e6ea986359849dd8319433ae9f
 2020-11-03 09:35:22 +00:00
Michal Niewrzal
ae17ae73d7 satellite/metainfo/metabase: add metadata and nonce to CommitObject 
request

Change-Id: I584775800963fc2167c749f3da20475036fa5325
 2020-11-02 16:49:18 +01:00
Egon Elbre
716068a1e0 Merge branch 'master'. 
Change-Id: Ic14325edc291573582dce0cea3e04991a820b48b
 2020-11-02 13:02:01 +02:00
Michal Niewrzal
0205f0d807 satellite/metainfo: fix usage of types from internalpb 
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I34e121fce47371ee4cfd92cce03809520b68859f
 2020-10-30 16:03:45 +02:00
Michal Niewrzal
517212bfa8 satellite/metainfo: fix usage of types from internalpb 
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I3344623dc7acfa73db6c20cd3212301e74335857
 2020-10-30 13:20:58 +00:00
Kaloyan Raev
b8c6fb764c satellite/metainfo: add metabase to metainfo service 
Change-Id: Ie3ff238b138d8a57d99e32b13f7a71aa624d53e3
 2020-10-30 12:49:47 +02:00
Egon Elbre
995900e02f Merge branch 'master' into multipart-upload 
Change-Id: I8717f9f7cc54f3934402a98f21129e16841b25bd
 2020-10-30 11:48:05 +02:00
Michal Niewrzal
8f26f66da0 internalpb: move satellite specific protobuf types storj/storj 
We have some types that are only valid for satellite usage. Such types
are SatStreamID and SatSegmentID. This change moves those types to
storj/storj and adds basic infrastructure for generating code.

Change-Id: I1e643844f947ce06b13e51ff16b7e671267cea64
 2020-10-30 08:49:16 +00:00
Egon Elbre
53b396b901 satellite/metainfo/metabase: move Encryption to Begin 
Change-Id: Ic1fb935ce7668e901f6cbe14a67c071a1f16a9ad
 2020-10-29 16:04:21 +02:00
Egon Elbre
e3985799a1 storage/{cockroachkv,postgreskv}: add ctx to opening 
Database opening usually dial and hence we should pass ctx to them.

Change-Id: Iecf41241aaa94d54506cbc80b0e53449848d8819
 2020-10-29 10:49:08 +00:00
Michal Niewrzal
809eb14ac2 satellite/metainfo/metabase: move metainfo PoC into storj repo 
Change-Id: I39356d8bc7305b4a8ea0c1fb5603010ad72a68b9
 2020-10-29 09:44:55 +01:00
Michal Niewrzal
cb1fea87f8 satellite/metainfo: mark unused methods as 'not implemented' 
Some of metainfo endpoint methods are not used but we still have
implementation there. This change removes unused code and returns
unimplemented error for those methods.

Change-Id: I74e75e0caff76a4f5d119ee989b687b4e9d6e6f9
 2020-10-28 12:42:47 +00:00