Commit Graph

178 Commits

Author SHA1 Message Date
Michal Niewrzal
aa47e70f03 satellite/metainfo: use metabase.SegmentKey with metainfo.Service
Instead of using string or []byte we will be using dedicated type
SegmentKey.

Change-Id: I6ca8039f0741f6f9837c69a6d070228ed10f2220
2020-09-03 15:11:32 +00:00
Yingrong Zhao
af773ec8a6 cmd/uplink: use DeleteBucketWithObjects for bucket force deletion
This PR updates `uplink rb --force` command to use the new libuplink API
`DeleteBucketWithObjects`.
It also updates `DeleteBucket` endpoint to return a specific error
message when a given bucket has concurrent writes while being deleted.

Change-Id: Ic9593d55b0c27b26cd8966dd1bc8cd1e02a6666e
2020-09-02 16:39:20 +00:00
Isaac Hess
2d54447088 satellite/metainfo: Fix GetObjectIPs loop and test
This change forces the test of GetObjectIPs to use multiple remote
segments (earlier versions of the test were accidentally using inline
segments). This change also revealed a small bug in the for loop code,
which is fixed.

Change-Id: Ic486b079d221952ba13553acf0ca41a8873f3f21
2020-08-31 15:56:12 -06:00
Isaac Hess
351aa70eb7 satellite/metainfo: Implement GetObjectIPs
Change-Id: Ibabbe7c555b790498d28a6ac4c95fcf2f7376978
2020-08-31 10:27:56 -06:00
Egon Elbre
9225fc5aef satellite/accounting: simplify ExceedsBandwidthUsage call
Change-Id: I5376da2329e44da8f060226d2a76432df0acdaa0
2020-08-28 18:10:02 +03:00
Egon Elbre
3ca405aa97 satellite/orders: use metabase types as arguments
Change-Id: I7ddaad207c20572a5ea762667531770a56fd54ef
2020-08-28 15:52:37 +03:00
Yingrong Zhao
bd5213f68b satellite/metainfo: implement batch delete for DeleteBucket
This PR changes DeleteBucket to be able to delete all objects within a
bucket if `DeleteAll` is set in `BucketDeleteRequest`.
It also changes `DeleteBucket` API to treat `ErrBucketNotFound` as a
successful delete operation instead of returning an error back to the
client.

Change-Id: I3a22c16224c7894f2d0c2a40ba1ae8717fa1005f
2020-08-24 13:28:09 +00:00
Yingrong Zhao
0518b16370 satellite/piecedeletion: move node info retrieval into the service
This change will require less work for the user of peiecedeletion
service by moving overlay database call into the package.

Change-Id: I14a150ab71fe885780e7a7a74db006a779507ae5
2020-08-13 16:07:54 +00:00
Yingrong Zhao
b71da59f8a satellite/metainfo: fix client context cancelation
We should still ignore client cancelation.
Also remove logger dependency on `Report`.

Change-Id: I59987c6b1aaa93202b07e5ef6e207d5191542b52
2020-08-11 20:58:23 +00:00
Isaac Hess
a14887e20b satellite/metainfo: Add GetObjectIPs method
This adds the unimplemented GetObjectIPs method to metainfo endpoint so
we can import new common protobuf definitions.

Change-Id: I154f26baccb6bb3c66de3eb25611930545c9754b
2020-08-11 11:39:15 -06:00
Egon Elbre
94a09ce20b all: add missing dots
Change-Id: I93b86c9fb3398c5d3c9121b8859dad1c615fa23a
2020-08-11 17:50:01 +03:00
Yingrong Zhao
2b2cb468bf satellite/metainfo: implement DeleteObjectPieces with objectdeletion package
Use new objectdeletion package for deleting pointers.

In the best case scenario, it will make on database call to fetch
information about the number of segments. And another request to delete
and fetch information about other segments.

This PR also changes our object deletion API to return no error when an
object is not found but instead consider such operation as success. This
behavior is asligned with S3 API and makes the code less complex.

Change-Id: I280c56e8b5d815a8c4dafe8227689467e899775a
2020-08-06 16:10:13 -04:00
Michal Niewrzal
ceb2eee494 satellite/metainfo: make overwrites impossible without delete permission
https://storjlabs.atlassian.net/browse/USR-1007

Change-Id: Ic7fa600a00a53b80a37acee4ac994be6ea4cb279
2020-08-06 18:01:44 +00:00
Michal Niewrzal
88dcc93f3c satellite/metainfo: use user PartnerID for bucket attribution
Change-Id: I20f1bd432333f9b37ca8fb457c349eff94ffb392
2020-08-06 13:14:07 +00:00
Rafael Gomes
935f44ddb7 satellite/metainfo: Add Delete Service config
Change-Id: I0a6e3ce1adfe1488eb23da9dda92877af1834599
2020-08-03 14:28:02 +00:00
Michal Niewrzal
e69cec0b75 satellite/metainfo: stop generating StreamID for BeginDeleteObject
request

We are no longer using `BeginDeleteSegment` or `ListSegments` so we can
avoid generating StreamID as a result of `BeginDeleteObject`.

StreamID from `BeginDeleteObject` is also not used on Uplink side.

Change-Id: I3b068deab17068459849b5cf05811cad4b8a9034
2020-07-17 17:51:06 +00:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
Jennifer Johnson
784a156eea satellite: prevents uplink from creating a bucket once it exceeds the max bucket allocation.
Change-Id: I4b3822ed723c03dbbc0df136b2201027e19ba0cd
2020-07-15 17:27:05 +00:00
Isaac Hess
34b019d16a satellite/metainfo: Handle revocation request
Logic to handle a request to revoke a macaroon.

Change-Id: I5d5c93fcc2e026b0aaf82bfdfacc822185b10f9f
2020-06-22 13:50:07 -06:00
Isaac Hess
2d727bb14e satellite: Check macaroon revocation
When a request comes in on the satellite api and we validate the
macaroon, we now also check if any of the macaroon's tails have been
revoked.

Change-Id: I80ce4312602baf431cfa1b1285f79bed88bb4497
2020-06-22 13:50:07 -06:00
Egon Elbre
6cc7fd5f31 satellite/metainfo: remove old endpoints
Change-Id: Ic85c27caeea08879714e8303b285b6136f9a7fd1
2020-06-16 16:12:14 +03:00
Isaac Hess
05a981bc13 satellite/metainfo: Add RevokeAPIKey stub endpoint
Change-Id: I07fb997dda48ea204aab11f656a1a37762684339
2020-06-16 07:03:02 -06:00
Egon Elbre
19d431ff38 satellite/metainfo: disable old endpoints
First step in removing endpoints that were used by pre 1.0 clients.

Change-Id: Id861c73ffac97139aa9540f4973ea849e572f906
2020-06-16 11:22:44 +00:00
Rafael Gomes
958ea1b9df satellite/accounting: add download limit cache
Change-Id: I722930cab8bd5d240f4878dc6997e9bc7637311f
2020-06-12 16:33:46 -03:00
Michal Niewrzal
e8c4010311 satellite/metainfo: add missing error check
Change-Id: I4043633505a3eaee8304f71f5ddea17de2b3d10f
2020-06-02 09:38:55 +00:00
Moby von Briesen
b82d04e618 satellite/metainfo: limit size of uplink-provided metadata to 2KiB
Change-Id: Id44a46046ddb4a12102525531f4502fcff2b6252
2020-06-01 16:51:29 -04:00
Egon Elbre
d6c90b7ab5 pkg/macaroon: remove aliased package
Change-Id: I8b06e9a794d00752948f190623006e1afffdb031
2020-05-29 13:34:20 +00:00
Egon Elbre
7f323754a4 metainfo/piecedeletion: use NodeURL-s
Change-Id: I247dbfe03e7864e940e4cd1d0f343f38e84099e0
2020-05-21 08:37:13 +03:00
littleskunk
1b06e2a928
satellite/metainfo: remove deprecated error validation (#3876) 2020-05-18 19:58:42 +02:00
Egon Elbre
4e94da3fda satellite/overlay: add feature flag for node selection cache
Also distinguish the purpose for selecting nodes to avoid potential
confusion, what should allow caching and what shouldn't.

Change-Id: Iee2451c1f10d0f1c81feb1641507400d89918d61
2020-05-06 16:13:47 +03:00
Egon Elbre
7e0e74c65c satellite/metainfo: attribution based on useragent
Automatically attach attribution information to bucket during
BeginObject or CreateBucket when the UserAgent is set.

Change-Id: I405cb26c5a2f7394b30e3f2cf5d2214c8781eb8b
2020-04-17 10:47:14 +03:00
Kaloyan Raev
a2ce836761 remove sugar logging
Change-Id: I6b6ca9704837cb3f5f5449ba7f55661487814d9f
2020-04-15 12:37:47 +00:00
Egon Elbre
c97131ae78 satellite/metainfo: organize attribution methods
Change-Id: I4f35599c3f923861b5b05b128bf904480679f5f9
2020-04-14 11:27:43 +03:00
Michal Niewrzal
f36e8548f1 satellite/metainfo: adjust max inline segment size validation to
potential encryption overhead.

This is the same approach we have for validating remote segment size.

https://storjlabs.atlassian.net/browse/USR-619

Change-Id: I2597ee734313a3068fd986001680bbedbf1bed2a
2020-04-09 12:34:10 +00:00
Egon Elbre
11a44cdd88 all: don't depend on gogo/proto directly
Change-Id: I8822dea0d1b7b99e0b828e0373a0308a42dde2be
2020-04-08 17:32:15 +00:00
Michal Niewrzal
2d1a6968b0 satellite/metainfo: revert returning segment size and inline segment
size with BeginObject

Such solution will add one round trip to satellite during upload so for
now we are reverting this until we will have solution for this.

Change-Id: Ic2d826448ab7b0318cd6922df05deee9167cf2f0
2020-04-06 13:36:34 +02:00
Matt Robinson
4b01a8dd18
Descellate Usage Error to Debug (#3780) 2020-04-03 13:20:20 +02:00
Egon Elbre
6492b13d81 all: remove old uuid
Change-Id: I3a137f73456f010c37d3933dbe12cbbb840b809f
2020-04-02 19:30:36 +03:00
Michal Niewrzal
c178a08cb8 satellite/metainfo: add max segment size and max inline size to
BeginObject response

We want to control inline segment size and segment size on satellite
side. We need to return such information to uplink like with redundancy
scheme.

Change-Id: If04b0a45a2757a01c0cc046432c115f475e9323c
2020-04-02 12:41:28 +00:00
Michal Niewrzal
4a79b609e9 satellite/metainfo: fix panic when we batch BeginObjectDelete without
all permissions

Without read and list permissions BeginObjectDelete won't return error
if occurs. This was breaking Batch processing because there was
assumption that without error response will be always not nil.

https://storjlabs.atlassian.net/browse/SM-590

Change-Id: I0fc9539e429110a660eb28725b266d5e4771d198
2020-04-02 12:20:19 +00:00
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
2020-03-31 19:16:41 +03:00
Egon Elbre
1d79228ed0 satellite/metainfo: support uplink useragent
Adds support for parsing user agent and specifying uplink version and
it's library dependencies.

Change-Id: Ibaddde4deb93e153ac05c91b676c5b5f1ae1aa37
2020-03-31 15:11:31 +00:00
Kaloyan Raev
6c44512c15 satellite/metainfo: increase stream ID expiration to 48h
Otherwise, it is not possible to upload very large files like 1 TB.

Change-Id: Iadda1ae91174125736684850b906d5bb6d19f0a9
2020-03-23 15:47:39 +02:00
Egon Elbre
1b6ab173a8 private/context2: moved to storj.io/common/context2
Change-Id: Ic1dd1ed645ff3e1057c9b2b143e2c3ddf29d678e
2020-03-20 14:39:46 +00:00
Egon Elbre
eb1d8aab96 satellite/metainfo/pointerverification: service for verifying pointers
This implements a service for pointer verification. This makes the
slightly clearer, because it's not part of metainfo.

It also adds a peer identity cache which reduces database calls and peer
identity decoding.

Change-Id: I45da40460d579c6f5fd74c69bccea215157aafda
2020-03-19 16:27:38 +00:00
Egon Elbre
09e0f3de63 satellite/metainfo/piecedeletion: add Service
Change-Id: Id7e32ed569701fa0be66f9527c43a67052994570
2020-03-18 14:50:08 +00:00
Michal Niewrzal
81afbcc12e satellite/metainfo: check bucket existence on upload and listing
Initial change for checking bucket existence on satellite side for
requests like BeginObject and ListObjects. This is simple implementation
that is just checking bucket in DB but should be improved in future to
avoid DB calls as much as possible.

Part of https://storjlabs.atlassian.net/browse/USR-365

Change-Id: I9076acddc44d7dbfa7612a1c24a007de01621583
2020-03-17 15:43:22 +00:00
Kaloyan Raev
27f811a9e1 metainfo: delete methods return the deleted item
This only happens if Read or List permission is granted together with
the Delete permission

Change-Id: I68b5f04a476bddabe499809ac98097aac75732a8
2020-03-16 16:26:16 +02:00
Michal Niewrzal
16878a22ea satellite/metainfo: stops hiding real validateAuth
Metainfo method validateAuth checks things like API key, user permission
and rate limit but at the end all errors were returned as
rpcstatus.Unauthenticated.

Old Metainfo is not touched to avoid backward compatibility issues.

Change-Id: I78eb276210fc50151da58a5c84e13ecd0961da29
2020-03-10 11:53:00 +00:00
Michal Niewrzal
9f390f37da satellite/metainfo: return default ciphers (path and encryption) for old
uplinks

New libuplink is not storing encryption values in with bucket but old
uplinks are using those values for configuration. If bucket was created
with new libuplink we will send back satellite defaults.

Change-Id: Ie1bf3682847e07b302270b4c4bf1a7219f4bf011
2020-03-05 10:04:50 +00:00