Commit Graph

4610 Commits

Author SHA1 Message Date
Michal Niewrzal
b7df41a0a1 satellite/metainfo/metabase: add EncryptedMetainfoEncryptedKey parameter
to metabase

* EncryptedMetainfoEncryptedKey added to CommitSegment and
UpdateMetadata request
* EncryptedMetainfoEncryptedKey returned with GetObject response and all
delete responses
* EncryptedMetainfoEncryptedKey returned with object iterator results

Change-Id: I917541ab5f3e1863bc8f238d17a15fbf72a23025
2020-11-17 10:19:12 +00:00
Egon Elbre
402cfcb7c6 satellite/metainfo/metabase: add prefix to iterator
Change-Id: I2dd7b7917aed9def0be3f131ca60ab12f2077d83
2020-11-16 17:38:59 +02:00
Egon Elbre
7aba265db0 satellite/metainfo/metabase: add status option to iteration
Change-Id: Ie9d1a3704d8d08e3421ba2244a1f3be57d15ed8f
2020-11-16 15:41:20 +02:00
Kaloyan Raev
a015f41927 satellite/metainfo: ListObjects to use Metabase API
Change-Id: If75d04b9a44f08515be717a85f397b66f8a5c56f
2020-11-16 13:27:17 +02:00
Michal Niewrzal
7c384c8293 Merge 'master' branch
Change-Id: I1eefd5a56449e577820977d61fa4a22bdd4fc230
2020-11-16 10:02:54 +01:00
Michal Niewrzal
cd7d911b26 satellite/metainfo: implement ListSegments with metabase
Change-Id: I78fb2400fed0bb01c5e05bd5bbcddef5e4a3bdf0
2020-11-16 08:33:35 +00:00
Kaloyan Raev
ff32268310 satellite/metainfo/metabase: DeleteExpiredObjects
Change-Id: I16293ce1a33f714aa005022daf3d40670cc66b04
2020-11-13 15:31:31 +02:00
Fadila Khadar
a749ac9f47 satellite/metainfo/metabase: iterateObjectsAllVersions
Iterate through all objects in a bucket
recursive, no prefix support

Change-Id: Ieaad9fbd0b5c1593554db9d543f4ee04851bac28
2020-11-13 13:50:08 +01:00
Egon Elbre
8182d8a726 satellite/metainfo/metabase: use txutil.WithTx in delete
Change-Id: I75965c4dcf57479ace2367f5d3069d785628e86a
2020-11-12 14:38:36 +02:00
Egon Elbre
bc460cd62d satellite/metainfo/metabase: use txutil.WithTx in commit
Change-Id: I84347291179c7042e22bda703a1c9bd80a1513a0
2020-11-12 14:38:33 +02:00
Michal Niewrzal
9d25b3a7d3 satellite/metainfo/metabase: add ListSegments request
Change-Id: Ie777a0f791685beab3973e76b10a9ce76398e7d7
2020-11-12 11:46:17 +01:00
Michal Niewrzal
fd707392e7 satellite/metainfo: set RedundancyScheme for begin segment
response

Multipart upload needs it for uploading parts.

Change-Id: I34886a9dabd17ecb7d55d4c6e19914a1c821f638
2020-11-11 17:25:52 +00:00
Kaloyan Raev
9dcd0936d7 satellite/metainfo: MakeInlineSegment to use metabase
Change-Id: Ib32c1d9e91e3f13b4e44abefb2b5d0352b64bd79
2020-11-11 15:24:21 +02:00
Michal Niewrzal
3fe16f4003 satellite/metainfo: upload/download with metabase
This change is adjusting metainfo endpoint to use metabase for uploading
and downloading remote objects. Inline segments will be added later.

Change-Id: I109d45bf644cd48096c47361043ebd8dfeaea0f3
2020-11-11 12:13:52 +00:00
Fadila Khadar
2bd239bb75 satellite/metainfo/metabase: tests commit inline segment
Change-Id: Ifad461041c7b9d8c78fc9f998a4e708853087b02
2020-11-10 15:00:35 +00:00
Michal Niewrzal
48f1f7132e satellite/metainfo: set RS defaults
While resolving conflicts with `master` I missed this change which is
needed e.g. to run storj-sim.

Change-Id: I56a548ed92b978510526c26c81af03051acfde2f
2020-11-10 13:57:27 +01:00
Michal Niewrzal
7dde184cb5 Merge 'master' branch
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
2020-11-10 11:58:59 +00:00
Michal Niewrzal
92f9251074 satellite/metainfo/metabase: add GetSegmentByOffset request
Change-Id: Iea2ce439ec1f7285e447d590c1297294204edc2e
2020-11-10 11:33:34 +00:00
Kaloyan Raev
3ed4183e52 satellite/metainfo: delete object to use metabase
Change-Id: I2ab63a719fdbc1f8a7fbb4ad73d51a2d2dcfadc6
2020-11-10 09:55:23 +00:00
Michal Niewrzal
dbf9f121ed ci: mark failures as unstable
Change-Id: I3cfa07a35cb789f409d95d4778c07ca3355c0f3c
2020-11-09 16:48:38 +01:00
Egon Elbre
184e1ffa9d satellite/metainfo/metabase: commit object without proofs
Change-Id: I788111b9e5bdc961be9e731f8203e52ca9da9dca
2020-11-09 15:18:07 +00:00
Egon Elbre
754838f5d0 ci: convert test failures to unstable
Change-Id: Ib0252c666c086379bd6e2a3a47011479fe3988ed
2020-11-09 11:18:12 +00:00
Egon Elbre
0a77deec81 satellite/metainfo/metabase: basic migrate
Change-Id: Ia34f38e763af9ba88f75038b8ecfc9eb6550eae0
2020-11-05 16:31:02 +02:00
Kaloyan Raev
56e6bc884c metabase: add DeleteObjectsAllVersions method
Change-Id: I3252dd0aac595b233f4861c3c1375622326d955d
2020-11-05 12:17:54 +02:00
Michal Niewrzal
2398afe985 satellite/metainfo/metabase: add GetLatestObjectLastSegment request
Change-Id: I50c452e379f78864b38921f31e53a0554642ab02
2020-11-05 08:20:34 +00:00
Kaloyan Raev
f986fdfe72 metabase: optimize delete segments
Segments are deleted in a single DB query instead of a separate one for
each segment.

Change-Id: Icb9ae92b4b8405030c9d9ef57ba30bee6a35515c
2020-11-04 11:59:56 +02:00
Michal Niewrzal
4134100d80 satellite/metainfo/metabase: add UpdateObjectMetadata request
Change-Id: Iec73000e58f373294f37ef3e4470412d1c5f1340
2020-11-03 11:08:41 +00:00
Kaloyan Raev
89cefa9045 metabase: add Objects info to DeleteObjectResult
Change-Id: If7ed2ed0bbba08ad971c64852d51ff1a066c4f1e
2020-11-03 12:45:49 +02:00
Kaloyan Raev
f7aa5b4e39 metabase: CommitObject return metabase.Object
An immediate purpose is to be used in test cases for deleting objects

Change-Id: I481d3be5800e55da231ecb628ef7eb3ea011d196
2020-11-03 09:57:44 +00:00
Kaloyan Raev
7e9ac2ec0a metabase: use storj.ErrObjectNotFound instead of custom error
Change-Id: Ie90e438f7d7b58e6ea986359849dd8319433ae9f
2020-11-03 09:35:22 +00:00
Michal Niewrzal
ae17ae73d7 satellite/metainfo/metabase: add metadata and nonce to CommitObject
request

Change-Id: I584775800963fc2167c749f3da20475036fa5325
2020-11-02 16:49:18 +01:00
Egon Elbre
716068a1e0 Merge branch 'master'.
Change-Id: Ic14325edc291573582dce0cea3e04991a820b48b
2020-11-02 13:02:01 +02:00
Egon Elbre
fd8e697ab2 {satellite,storagenode}/internalpb: use specific package name
Ensure we don't register types with the same name into protobuf.

Change-Id: I53d025863fff8c91a067ca5819befa87eb5e35bb
2020-10-30 17:31:08 +02:00
Egon Elbre
9674ce40e6 mod: bump dependencies
Change-Id: I61beaf9793609f7614478cfa604b2721d9010b52
2020-10-30 16:17:55 +02:00
Michal Niewrzal
0205f0d807 satellite/metainfo: fix usage of types from internalpb
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I34e121fce47371ee4cfd92cce03809520b68859f
2020-10-30 16:03:45 +02:00
Egon Elbre
77c4f99fa0 satellite/internalpb: move delegated_repair.proto
Change-Id: If4f37c52b151e09cf35d2145b463ef1e9ab529ae
2020-10-30 15:31:32 +02:00
Egon Elbre
11338e9beb satellite/internalpb: move audithistory.pb
Change-Id: I8eee84d49ed90459168ddaf04ae57f790c2a22c4
2020-10-30 15:30:11 +02:00
Egon Elbre
1903b15474 storagenode/internalpb: move gracefulexit.proto
Change-Id: Ia3614846ed49a39c8f39331516d16d45a695240b
2020-10-30 15:24:56 +02:00
Michal Niewrzal
517212bfa8 satellite/metainfo: fix usage of types from internalpb
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I3344623dc7acfa73db6c20cd3212301e74335857
2020-10-30 13:20:58 +00:00
Egon Elbre
cda67a659a storagenode/internalpb: move inspector.proto
Change-Id: I951379c3b2ff00d1bc09d6a49c026a7e723432d6
2020-10-30 14:51:26 +02:00
Egon Elbre
7ce372c686 satellite/internalpb: add inspectors
Change-Id: Ib688e43d05135c0c31ae95df533f1e4535ea396a
2020-10-30 13:28:17 +02:00
Egon Elbre
004e610d0f satellite/internalpb: move datarepair.pb to internal
Change-Id: If901d9ff4e5ee6715b963eeeb46513a602a44b3d
2020-10-30 13:28:14 +02:00
Kaloyan Raev
b8c6fb764c satellite/metainfo: add metabase to metainfo service
Change-Id: Ie3ff238b138d8a57d99e32b13f7a71aa624d53e3
2020-10-30 12:49:47 +02:00
Egon Elbre
995900e02f Merge branch 'master' into multipart-upload
Change-Id: I8717f9f7cc54f3934402a98f21129e16841b25bd
2020-10-30 11:48:05 +02:00
Michal Niewrzal
8f26f66da0 internalpb: move satellite specific protobuf types storj/storj
We have some types that are only valid for satellite usage. Such types
are SatStreamID and SatSegmentID. This change moves those types to
storj/storj and adds basic infrastructure for generating code.

Change-Id: I1e643844f947ce06b13e51ff16b7e671267cea64
2020-10-30 08:49:16 +00:00
Qweder93
f5ba8b8009 storagenode/suspensions: added offline-suspension notificatio chore + tests
Change-Id: I2521cd2e7d08a1dd379e717a554a026c7508c18f
2020-10-29 19:44:22 +02:00
Egon Elbre
e0dca4042d all: add pprof labels for debugger
By using pprof.Labels debugger is able to show service/peer names in
goroutine names.

Change-Id: I5f55253470f7cc7e556f8e8b87f746394e41675f
2020-10-29 15:10:07 +00:00
Egon Elbre
53b396b901 satellite/metainfo/metabase: move Encryption to Begin
Change-Id: Ic1fb935ce7668e901f6cbe14a67c071a1f16a9ad
2020-10-29 16:04:21 +02:00
Qweder93
624255e8ba storagenode/secret: db tests added, small renaming fixes added
Change-Id: I7eae1a9a64c20a39c97e81fa741cfc9b9e1e615a
2020-10-29 14:23:04 +02:00
Egon Elbre
caefde6b32 private/{dbutil,tagsql}: pass ctx to database opening
Database opening usually dial and hence we should pass ctx to them.

Change-Id: Iaa2875981570d83e65be3710f841cf30349f807b
2020-10-29 10:51:29 +00:00