Commit Graph

149 Commits

Author SHA1 Message Date
Qweder93
332b673a05 satellite/metainfo: comment tests for rename
comment tests that contains fields needs
to rename on uplink side without breaking compatibility.
after rename tests will be moved back from comments.

Change-Id: I3bc4aff6ae7f6711ade956ac389f0d7e1a1ab91a
2022-01-10 17:46:32 +02:00
Qweder93
15c2b2cc10 satellite/metainfo: comment tests for rename
comment tests that contains fields needs
to rename on uplink side without breaking compatibility.
after rename tests will be moved back from comments.

Change-Id: I439783c62678c32805a85aa52bef1d2b767543a1
2022-01-10 14:44:43 +00:00
Michał Niewrzał
ff653f6f22 satellite/metainfo: stop using in tests unused method
We would like to remove it from uplink but to do that
we need to stop using it in storj tests.

Change-Id: I8f6af74d6b8b8e447ab8ed3a00517b40840dad58
2021-12-22 15:05:53 +01:00
Kaloyan Raev
7d84482381 satellite/metainfo: prevent moving objects between geofenced buckets
Initially, we want to avoid moving objects between buckets with
different geofencing config.

Change-Id: I0cc497873eb5732623ef2d9bc5f78ba1cc48c6b8
2021-11-24 08:34:22 +00:00
Márton Elek
76c2228fbd satellite/metainfo: propagate geofencing between buckets and stream id
Github: https://github.com/storj/storj/issues/4245

Change-Id: I83d34367aab1f3c0d46a044f54980b2d50174b19
2021-11-24 08:05:05 +00:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
Michał Niewrzał
12d4dc16a1 satellite/{metainfo,metabase}: make metadata optional for CommitObject
To resolve problem with lack of ability to set metadata while MPU on
gateway we are adding setting metadata with BeginObject. This
change makes also metadata optional while CommitObject. We need
this functionality to not override metadata set with BeginObject in
case when metadata is not set with CommitObject.

Another reson is that we would like to not set metadata at all if user
didn't specify metadata. At the moment we always setting some bytes
for metadata fields e.g. empty EncryptedMetadata field can have key
and nonce set.

Change-Id: Ifee25b7718eb1f919119db9b698b29d8b5ebe2ec
2021-11-10 08:08:46 +00:00
Erik van Velzen
91158fd3bf satellite: omit system metadata fields if not requested
To reduce database load, add option to omit as many object properties
as possible when listing objects.

Change-Id: I817633801b00629a4042d1d1bd2389ee581953de
2021-11-08 09:19:44 +01:00
Qweder93
b314559a13 satellite/metainfo: TestBatch params updated
Removed PathCipher and DefaultSegmentSize from CreateBucketParams
since they are unused anymore and breaks Integration on uplink side

Change-Id: I1393a7f1f436940731aa59edd693043336383290
2021-11-05 14:53:55 +02:00
Michał Niewrzał
1fdb0eaa5b Revert "satellite/metabase: use storj.Nonce instead []byte"
This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.

This reverts commit 1ef06fae99.

Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
2021-10-12 15:39:54 +02:00
Michał Niewrzał
1ef06fae99 satellite/metabase: use storj.Nonce instead []byte
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.

Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
2021-10-11 16:13:34 +00:00
Yingrong Zhao
c911360eb5 satellite/metainfo: separate burst limit from rate limit config
This PR utilize the new burst limit column from projects table to allow
control on the limit for request per seconds and token bucket size
When no burst limit is explicitly set, rate limit is applied to both so
we don't limit how quickly request can be made in a second.

Change-Id: I883235c60c5d6416aeadd1c80ed2ebd193aa4d9f
2021-09-28 22:47:41 +00:00
crawter
e1a4195a35 satellite/metainfo/endpoint: MaxObjectKey length validation for BeginObject and CommitObject added
Change-Id: I6a766415a84b1e3c44ce65052ffa8ed4511d00dd
2021-09-13 15:07:16 +03:00
Michał Niewrzał
495e530933 satellite/metainfo: drop metainfo.Service
Drop a service that in fact don't make sense and
its just a wrapper for direct DB methods.

Change-Id: I1cb76f3ecc2d8765964d919c88541179957645c1
2021-09-09 17:30:10 +02:00
Michał Niewrzał
c258f4bbac private/testplanet: move Metabase outside Metainfo for satellite
At some point we moved metabase package outside Metainfo
but we didn't do that for satellite structure. This change
refactors only tests.
When uplink will be adjusted we can remove old entries in
Metainfo struct.

Change-Id: I2b66ed29f539b0ec0f490cad42c72840e0351bcb
2021-09-09 07:15:51 +00:00
Bill Thorp
549e799bbe satellite: wrap notfound on failed deletes as DRPC errors
We're seeing BeginDeleteObject in metaclient returning object not found:
metabase: no rows deleted in the Gateway-MT mint tests.  There's a
client check for rpcStatus.NotFound, but the metabase endpoint isn't
wrapping the db error as a DRPC error.

Here's the chain:

  gateway.AbortMultipartUpload()
  project.AbortUpload()
  metainfoClient.BeginDeleteObject() <- understands DRPC errors
  endpoint.DeletePendingObject() <- where this code is
  db.DeletePendingObject() <- returns error

Change-Id: I93991de76487426df0a807b0d1e69fc975196a1a
2021-08-16 09:02:35 +00:00
Michał Niewrzał
bc79f01aaa satellite/metabase: set expires_at while committing segment
We added expires_at column to segments table and now
we need to populate this column while committing segment.
We still need to migrate existing segments with
separate tool.

Change-Id: Ibac8c63d97201dd98cc2cb9db385f4cb73bc3f7e
2021-06-10 16:32:28 +00:00
Egon Elbre
10a0216af5 satellite/metainfo: use range for specifying download limit
Previously the object range was not used for calculating order limit.
This meant that even if you were downloading only a small range it would
account bandwidth based on the full segment.

This doesn't fully address the accounting since the lazy segment
downloads do not send their requested range nor requested limit.

Change-Id: Ic811e570c889be87bac4293547d6537a255078da
2021-06-01 09:36:55 +00:00
Egon Elbre
a8533042a3 mod: bump uplink
Renamed a private package in uplink and some of the tests need to be
updated.

Change-Id: I8735fc16c2066ecba669618f0c1dadf68273704d
2021-04-30 16:53:54 +03:00
Egon Elbre
2c657f594e satellite/metainfo: don't rely on exact error name
Change-Id: I7975a00c32891a43b4f1e6dc4f5847201844f2ec
2021-04-29 18:35:56 +00:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00
Kaloyan Raev
035c393da0 satellite: update tests to pass etag.Reader to multipart.PutObjectPart
Change-Id: Ibe99357945ae7a91f5b5d4f87b83d425c9fa84a5
2021-03-29 13:18:11 +00:00
Kaloyan Raev
e0f15130a2 satellite/metainfo: ensure list prefix is prepended in satStreamID
When listing pending objects with prefix, the prefix should be prepended
to the EncryptedPath in satStreamID. Otherwise, listing multipart
uploads may display different UploadID than expected.

Change-Id: I27e9f9af9348783e053ad123121b6ddd051739e4
2021-02-19 16:32:55 +02:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Caleb Case
f011a5e8cc
Use WithNonce when creating caveats
This ensures the caveats are unique even when they contain the same
permissions and will result in unique macaroons. This is important to
ensure revocation doesn't impact more macaroons than intended.

Change-Id: I6354edd0119f2d85eaf580f2d1926a3de9151b88
2021-02-04 13:46:32 -05:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Michał Niewrzał
50bea3ab1a satellite/metainfo: adjust tests to changes in uplink
One of uplink method changed its signature and we need to fix test on satellite side.

Change-Id: Ib89ea6aa25c57bac11bc3e0e9c2c89a4b9debd7c
2021-01-26 08:54:35 +00:00
Kaloyan Raev
ea48322dd3 satellite/metainfo: use deterministic signing for satStreamID
So we can have stable UploadID for multipart uploads.

Change-Id: Iac6780394c8cc0f96c0b9c4b850b92ed3627a9b0
2021-01-19 15:56:29 +00:00
Michał Niewrzał
d54ae9f10f satellite/metainfo: close project in TestObjectOverrideOnUpload
Change-Id: If19245474f54cea1657f704e155e09e1511eaeec
2021-01-19 16:33:30 +01:00
Michał Niewrzał
38beecc7ad satellite/metainfo: Override pending object on upload
On upload we need to override pending and committed object. This change is adjusting DeleteObjectAllVersions to delete both.

Change-Id: Ib66c2af207c618119f7bf0de7fa9d3e5145d8641
2021-01-14 15:16:02 +00:00
Michał Niewrzał
71ca18134b satellite/metainfo: reject downloading a segment from multipart object
Non-multipart uplink implementation is always trying to download object
by downloading last segment first (PartNumber=0, Index=-1) but this
approach won't work with multipart object. We need to reject such old
style request with reasonable message.

Change-Id: I9221e019933565a8d25136bdfef3e054320bac3d
2021-01-11 08:04:33 +00:00
Kaloyan Raev
bafc6af992 ci: remove workaround for failing tests
Change-Id: I3eb673fae6c81bee17d7437cb870d5f5ba6978d5
2020-12-21 18:07:40 +02:00
Kaloyan Raev
6206aa88e4 {satellite/metainfo,private/testplanet} use TestingAllSegments in tests
Change-Id: I8c641a24fabf3ea537312978a42501eab8d6a339
2020-12-17 09:58:31 +00:00
Kaloyan Raev
fc85179a19 satellite/metainfo: refactor SegmentLocation.Index to SegmentPosition
Change-Id: Ic9403c8126712693326dd83d6ba4f3b84be3e0c7
2020-12-14 13:35:53 +02:00
Kaloyan Raev
3f5e07fb1f satellite/metainfo: more tests fixed
- TestBucketNameValidation
- TestBatch
- TestCommitObjectMetadataSize
- TestIDs

TestOverwriteZombieSegments is removed as not relevant to metabase.

Change-Id: I13cf5abe342089960628f185061303fd4f9d09a4
2020-12-08 14:00:24 +02:00
Michal Niewrzal
e4c4ab06ba satellite/metainfo: add testing methods for getting objects and segments
Change-Id: I4229d80090609df27d998a90bdb3137469dba042
2020-12-01 12:44:08 +00:00
Michal Niewrzal
5a7bc9657d Merge 'master' branch
Change-Id: If583132a821274dc4b78cf5f72b853ba8460c619
2020-11-30 12:57:22 +01:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
2020-11-30 10:49:23 +01:00
Michal Niewrzal
8ceef9f357 satellite/metainfo: temporary disable one assertion in test
This is need to merge https://review.dev.storj.io/c/storj/uplink/+/3208
, after that this code will be back.

Change-Id: If9f2f1db95c7a1bba64a41c45a39bd3096a519e7
2020-11-25 13:21:41 +00:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
2020-11-24 17:50:33 +01:00
Michal Niewrzal
3fe16f4003 satellite/metainfo: upload/download with metabase
This change is adjusting metainfo endpoint to use metabase for uploading
and downloading remote objects. Inline segments will be added later.

Change-Id: I109d45bf644cd48096c47361043ebd8dfeaea0f3
2020-11-11 12:13:52 +00:00
Michal Niewrzal
7dde184cb5 Merge 'master' branch
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
2020-11-10 11:58:59 +00:00
Egon Elbre
7183dca6cb all: fix defers in loop
defer should not be called in a loop.

Change-Id: Ifa5a25a56402814b974bcdfb0c2fce56df8e7e59
2020-11-02 15:06:38 +02:00
Michal Niewrzal
0205f0d807 satellite/metainfo: fix usage of types from internalpb
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I34e121fce47371ee4cfd92cce03809520b68859f
2020-10-30 16:03:45 +02:00
Michal Niewrzal
cb1fea87f8 satellite/metainfo: mark unused methods as 'not implemented'
Some of metainfo endpoint methods are not used but we still have
implementation there. This change removes unused code and returns
unimplemented error for those methods.

Change-Id: I74e75e0caff76a4f5d119ee989b687b4e9d6e6f9
2020-10-28 12:42:47 +00:00
Michal Niewrzal
aa47e70f03 satellite/metainfo: use metabase.SegmentKey with metainfo.Service
Instead of using string or []byte we will be using dedicated type
SegmentKey.

Change-Id: I6ca8039f0741f6f9837c69a6d070228ed10f2220
2020-09-03 15:11:32 +00:00
Isaac Hess
2d54447088 satellite/metainfo: Fix GetObjectIPs loop and test
This change forces the test of GetObjectIPs to use multiple remote
segments (earlier versions of the test were accidentally using inline
segments). This change also revealed a small bug in the for loop code,
which is fixed.

Change-Id: Ic486b079d221952ba13553acf0ca41a8873f3f21
2020-08-31 15:56:12 -06:00
Isaac Hess
ba1a113e2e satellite/metainfo: Add test for GetObjectIPs
This PR adds the integration test for GetObjectIPs, testing all repos.

Change-Id: Id4d4f19c427a2e320b2a82efa68150fa7f5f86fb
2020-08-31 10:27:56 -06:00
Egon Elbre
c86c732fc0 satellite: simplify tests
satellite.DB.Console().Projects().GetAll database query
can be replaced with planet.Uplinks[0].Projects[0].ID

Change-Id: I73b82b91afb2dde7b690917345b798f9d81f6831
2020-08-28 22:28:04 +00:00