Commit Graph

134 Commits

Author SHA1 Message Date
Bill Thorp
549e799bbe satellite: wrap notfound on failed deletes as DRPC errors
We're seeing BeginDeleteObject in metaclient returning object not found:
metabase: no rows deleted in the Gateway-MT mint tests.  There's a
client check for rpcStatus.NotFound, but the metabase endpoint isn't
wrapping the db error as a DRPC error.

Here's the chain:

  gateway.AbortMultipartUpload()
  project.AbortUpload()
  metainfoClient.BeginDeleteObject() <- understands DRPC errors
  endpoint.DeletePendingObject() <- where this code is
  db.DeletePendingObject() <- returns error

Change-Id: I93991de76487426df0a807b0d1e69fc975196a1a
2021-08-16 09:02:35 +00:00
Michał Niewrzał
bc79f01aaa satellite/metabase: set expires_at while committing segment
We added expires_at column to segments table and now
we need to populate this column while committing segment.
We still need to migrate existing segments with
separate tool.

Change-Id: Ibac8c63d97201dd98cc2cb9db385f4cb73bc3f7e
2021-06-10 16:32:28 +00:00
Egon Elbre
10a0216af5 satellite/metainfo: use range for specifying download limit
Previously the object range was not used for calculating order limit.
This meant that even if you were downloading only a small range it would
account bandwidth based on the full segment.

This doesn't fully address the accounting since the lazy segment
downloads do not send their requested range nor requested limit.

Change-Id: Ic811e570c889be87bac4293547d6537a255078da
2021-06-01 09:36:55 +00:00
Egon Elbre
a8533042a3 mod: bump uplink
Renamed a private package in uplink and some of the tests need to be
updated.

Change-Id: I8735fc16c2066ecba669618f0c1dadf68273704d
2021-04-30 16:53:54 +03:00
Egon Elbre
2c657f594e satellite/metainfo: don't rely on exact error name
Change-Id: I7975a00c32891a43b4f1e6dc4f5847201844f2ec
2021-04-29 18:35:56 +00:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00
Kaloyan Raev
035c393da0 satellite: update tests to pass etag.Reader to multipart.PutObjectPart
Change-Id: Ibe99357945ae7a91f5b5d4f87b83d425c9fa84a5
2021-03-29 13:18:11 +00:00
Kaloyan Raev
e0f15130a2 satellite/metainfo: ensure list prefix is prepended in satStreamID
When listing pending objects with prefix, the prefix should be prepended
to the EncryptedPath in satStreamID. Otherwise, listing multipart
uploads may display different UploadID than expected.

Change-Id: I27e9f9af9348783e053ad123121b6ddd051739e4
2021-02-19 16:32:55 +02:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Caleb Case
f011a5e8cc
Use WithNonce when creating caveats
This ensures the caveats are unique even when they contain the same
permissions and will result in unique macaroons. This is important to
ensure revocation doesn't impact more macaroons than intended.

Change-Id: I6354edd0119f2d85eaf580f2d1926a3de9151b88
2021-02-04 13:46:32 -05:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Michał Niewrzał
50bea3ab1a satellite/metainfo: adjust tests to changes in uplink
One of uplink method changed its signature and we need to fix test on satellite side.

Change-Id: Ib89ea6aa25c57bac11bc3e0e9c2c89a4b9debd7c
2021-01-26 08:54:35 +00:00
Kaloyan Raev
ea48322dd3 satellite/metainfo: use deterministic signing for satStreamID
So we can have stable UploadID for multipart uploads.

Change-Id: Iac6780394c8cc0f96c0b9c4b850b92ed3627a9b0
2021-01-19 15:56:29 +00:00
Michał Niewrzał
d54ae9f10f satellite/metainfo: close project in TestObjectOverrideOnUpload
Change-Id: If19245474f54cea1657f704e155e09e1511eaeec
2021-01-19 16:33:30 +01:00
Michał Niewrzał
38beecc7ad satellite/metainfo: Override pending object on upload
On upload we need to override pending and committed object. This change is adjusting DeleteObjectAllVersions to delete both.

Change-Id: Ib66c2af207c618119f7bf0de7fa9d3e5145d8641
2021-01-14 15:16:02 +00:00
Michał Niewrzał
71ca18134b satellite/metainfo: reject downloading a segment from multipart object
Non-multipart uplink implementation is always trying to download object
by downloading last segment first (PartNumber=0, Index=-1) but this
approach won't work with multipart object. We need to reject such old
style request with reasonable message.

Change-Id: I9221e019933565a8d25136bdfef3e054320bac3d
2021-01-11 08:04:33 +00:00
Kaloyan Raev
bafc6af992 ci: remove workaround for failing tests
Change-Id: I3eb673fae6c81bee17d7437cb870d5f5ba6978d5
2020-12-21 18:07:40 +02:00
Kaloyan Raev
6206aa88e4 {satellite/metainfo,private/testplanet} use TestingAllSegments in tests
Change-Id: I8c641a24fabf3ea537312978a42501eab8d6a339
2020-12-17 09:58:31 +00:00
Kaloyan Raev
fc85179a19 satellite/metainfo: refactor SegmentLocation.Index to SegmentPosition
Change-Id: Ic9403c8126712693326dd83d6ba4f3b84be3e0c7
2020-12-14 13:35:53 +02:00
Kaloyan Raev
3f5e07fb1f satellite/metainfo: more tests fixed
- TestBucketNameValidation
- TestBatch
- TestCommitObjectMetadataSize
- TestIDs

TestOverwriteZombieSegments is removed as not relevant to metabase.

Change-Id: I13cf5abe342089960628f185061303fd4f9d09a4
2020-12-08 14:00:24 +02:00
Michal Niewrzal
e4c4ab06ba satellite/metainfo: add testing methods for getting objects and segments
Change-Id: I4229d80090609df27d998a90bdb3137469dba042
2020-12-01 12:44:08 +00:00
Michal Niewrzal
5a7bc9657d Merge 'master' branch
Change-Id: If583132a821274dc4b78cf5f72b853ba8460c619
2020-11-30 12:57:22 +01:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
2020-11-30 10:49:23 +01:00
Michal Niewrzal
8ceef9f357 satellite/metainfo: temporary disable one assertion in test
This is need to merge https://review.dev.storj.io/c/storj/uplink/+/3208
, after that this code will be back.

Change-Id: If9f2f1db95c7a1bba64a41c45a39bd3096a519e7
2020-11-25 13:21:41 +00:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
2020-11-24 17:50:33 +01:00
Michal Niewrzal
3fe16f4003 satellite/metainfo: upload/download with metabase
This change is adjusting metainfo endpoint to use metabase for uploading
and downloading remote objects. Inline segments will be added later.

Change-Id: I109d45bf644cd48096c47361043ebd8dfeaea0f3
2020-11-11 12:13:52 +00:00
Michal Niewrzal
7dde184cb5 Merge 'master' branch
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
2020-11-10 11:58:59 +00:00
Egon Elbre
7183dca6cb all: fix defers in loop
defer should not be called in a loop.

Change-Id: Ifa5a25a56402814b974bcdfb0c2fce56df8e7e59
2020-11-02 15:06:38 +02:00
Michal Niewrzal
0205f0d807 satellite/metainfo: fix usage of types from internalpb
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I34e121fce47371ee4cfd92cce03809520b68859f
2020-10-30 16:03:45 +02:00
Michal Niewrzal
cb1fea87f8 satellite/metainfo: mark unused methods as 'not implemented'
Some of metainfo endpoint methods are not used but we still have
implementation there. This change removes unused code and returns
unimplemented error for those methods.

Change-Id: I74e75e0caff76a4f5d119ee989b687b4e9d6e6f9
2020-10-28 12:42:47 +00:00
Michal Niewrzal
aa47e70f03 satellite/metainfo: use metabase.SegmentKey with metainfo.Service
Instead of using string or []byte we will be using dedicated type
SegmentKey.

Change-Id: I6ca8039f0741f6f9837c69a6d070228ed10f2220
2020-09-03 15:11:32 +00:00
Isaac Hess
2d54447088 satellite/metainfo: Fix GetObjectIPs loop and test
This change forces the test of GetObjectIPs to use multiple remote
segments (earlier versions of the test were accidentally using inline
segments). This change also revealed a small bug in the for loop code,
which is fixed.

Change-Id: Ic486b079d221952ba13553acf0ca41a8873f3f21
2020-08-31 15:56:12 -06:00
Isaac Hess
ba1a113e2e satellite/metainfo: Add test for GetObjectIPs
This PR adds the integration test for GetObjectIPs, testing all repos.

Change-Id: Id4d4f19c427a2e320b2a82efa68150fa7f5f86fb
2020-08-31 10:27:56 -06:00
Egon Elbre
c86c732fc0 satellite: simplify tests
satellite.DB.Console().Projects().GetAll database query
can be replaced with planet.Uplinks[0].Projects[0].ID

Change-Id: I73b82b91afb2dde7b690917345b798f9d81f6831
2020-08-28 22:28:04 +00:00
Bill Thorp
dbb53151f0 private/testplanet: Decrease metainfo MaxBuckets test value to speed testing.
TestMaxOutBuckets is one of our slower tests (50-90s).
This change seems to make it 2-12s.

It reduces the number of buckets that need to be created.
It also removes unnecessary storage nodes.

Change-Id: I1012fc6e9258b2f7674b16da4e8b418741c93eea
2020-08-26 17:31:31 +00:00
Michal Niewrzal
ceb2eee494 satellite/metainfo: make overwrites impossible without delete permission
https://storjlabs.atlassian.net/browse/USR-1007

Change-Id: Ic7fa600a00a53b80a37acee4ac994be6ea4cb279
2020-08-06 18:01:44 +00:00
Michal Niewrzal
20184d3604 satellite/metainfo: move TestAttributionReport to attribution tests
Additionally test was simplified by adding ability to set user agent for
testplanet uplink.

Change-Id: I82942c2280562b5118a42aa8e1e0f53092f8dbe1
2020-07-30 19:18:15 +00:00
Michal Niewrzal
18b3495248 satellite/metainfo: restore tests disabled to remove StreamID from
BeginDeleteObject method on uplink side

Change-Id: Id10cdeb367759f9949dc557b291726fd331d38ba
2020-07-22 13:54:39 +00:00
Michal Niewrzal
e69cec0b75 satellite/metainfo: stop generating StreamID for BeginDeleteObject
request

We are no longer using `BeginDeleteSegment` or `ListSegments` so we can
avoid generating StreamID as a result of `BeginDeleteObject`.

StreamID from `BeginDeleteObject` is also not used on Uplink side.

Change-Id: I3b068deab17068459849b5cf05811cad4b8a9034
2020-07-17 17:51:06 +00:00
Jennifer Johnson
784a156eea satellite: prevents uplink from creating a bucket once it exceeds the max bucket allocation.
Change-Id: I4b3822ed723c03dbbc0df136b2201027e19ba0cd
2020-07-15 17:27:05 +00:00
Isaac Hess
34b019d16a satellite/metainfo: Handle revocation request
Logic to handle a request to revoke a macaroon.

Change-Id: I5d5c93fcc2e026b0aaf82bfdfacc822185b10f9f
2020-06-22 13:50:07 -06:00
Isaac Hess
2d727bb14e satellite: Check macaroon revocation
When a request comes in on the satellite api and we validate the
macaroon, we now also check if any of the macaroon's tails have been
revoked.

Change-Id: I80ce4312602baf431cfa1b1285f79bed88bb4497
2020-06-22 13:50:07 -06:00
Egon Elbre
19d431ff38 satellite/metainfo: disable old endpoints
First step in removing endpoints that were used by pre 1.0 clients.

Change-Id: Id861c73ffac97139aa9540f4973ea849e572f906
2020-06-16 11:22:44 +00:00
Moby von Briesen
b82d04e618 satellite/metainfo: limit size of uplink-provided metadata to 2KiB
Change-Id: Id44a46046ddb4a12102525531f4502fcff2b6252
2020-06-01 16:51:29 -04:00
Michal Niewrzal
84892631c8 private/testplanet: remove old libuplink from testplanet
Change-Id: Ib1553f84d0b3ae12a5b00382f0f53357b6a273e2
2020-05-28 13:50:23 +00:00
igor gaidaienko
75ddecdf54 satellite/metainfo: Add test to verify payment report for attributed bucket
For this automated test I want to make sure the data in payment report is correct

Change-Id: I48a3b0e1dc9b2324435b763c87c363f1e310ff4e
2020-05-13 17:22:04 +03:00
Egon Elbre
588826157b satellite/metainfo: reduce RateLimit test flakyness
Delay of 100ms could happen due to other things happening on the test
server. Increase the time to 1s.

Change-Id: I2c7c21f966101771633d73e84cf9850d28089e71
2020-04-29 14:20:02 +03:00
Michal Niewrzal
c021b35879 private/testplanet: migrate testplanet to new libuplink
Replace most of old libuplink usages in testplanet. 100% migration will
be possible when we will be able to implement UploadWithClientConfig
with new libuplink.

Change-Id: I432d7d4917c7b67d46a058abd0a2a6a13f565ac4
2020-04-20 12:43:34 +00:00
Egon Elbre
9052085f70 private/testplanet: simplify uplink usage
Change-Id: I3e488dc296f1094ce95e6d6597ca6d3f8da90a76
2020-04-16 16:45:55 +00:00