Commit Graph

288 Commits

Author SHA1 Message Date
Moby von Briesen
db6bc6503d satellite/metainfo: Update metainfo RS config to more easily support multiple RS schemes.
Make metainfo.RSConfig a valid pflag config value. This allows us to
configure the RSConfig as a string like k/m/o/n-shareSize, which makes
having multiple supported RS schemes easier in the future.

RS-related config values that are no longer needed have been removed
(MinTotalThreshold, MaxTotalThreshold, MaxBufferMem, Verify).

Change-Id: I0178ae467dcf4375c504e7202f31443d627c15e1
2020-11-09 22:16:13 +00:00
Michal Niewrzal
0205f0d807 satellite/metainfo: fix usage of types from internalpb
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I34e121fce47371ee4cfd92cce03809520b68859f
2020-10-30 16:03:45 +02:00
Michal Niewrzal
517212bfa8 satellite/metainfo: fix usage of types from internalpb
After moving SatStreamID and SatSegmentID from common I missed changing
some methods in metainfo endpoint. This change is a fix for that.

Change-Id: I3344623dc7acfa73db6c20cd3212301e74335857
2020-10-30 13:20:58 +00:00
Michal Niewrzal
8f26f66da0 internalpb: move satellite specific protobuf types storj/storj
We have some types that are only valid for satellite usage. Such types
are SatStreamID and SatSegmentID. This change moves those types to
storj/storj and adds basic infrastructure for generating code.

Change-Id: I1e643844f947ce06b13e51ff16b7e671267cea64
2020-10-30 08:49:16 +00:00
Michal Niewrzal
cb1fea87f8 satellite/metainfo: mark unused methods as 'not implemented'
Some of metainfo endpoint methods are not used but we still have
implementation there. This change removes unused code and returns
unimplemented error for those methods.

Change-Id: I74e75e0caff76a4f5d119ee989b687b4e9d6e6f9
2020-10-28 12:42:47 +00:00
Michal Niewrzal
1adb497a71 satellite/metainfo: remove unused code
This change removed unused 'createRequests' struct. As far I remember it
was used to help validating old metainfo beginObject/commitObject flow.

Change-Id: I0f139b9934196d73f26eafa347ba5605722f3a55
2020-10-28 12:40:14 +01:00
Egon Elbre
2268cc1df3 all: fix linter complaints
Change-Id: Ia01404dbb6bdd19a146fa10ff7302e08f87a8c95
2020-10-13 15:59:01 +03:00
Stefan Benten
2f648fd981 satellite: make limits be nullable
Currently we have no way to actually set one
of the following limits to 0 (meaning not usable):

- maxBuckets
- usageLimit
- bandwidthLimit

With having the field nullable,
NULL corresponds to the global default,
0 now actually 0 and
a set value determines a custom limit.

Change-Id: I92bb77529dcbd0881ae8368921be9d246eb0919e
2020-09-21 19:34:19 +00:00
Michal Niewrzal
d199daa907 satellite/metainfo: replace local constants with metabase values
Change-Id: Ie855c10dd259464658952186251b7d3210eb49ce
2020-09-16 13:25:14 +02:00
Michal Niewrzal
c753d17e8f satellite/metainfo/objectdeletion: replace
objectdeletion.ObjectIdentifier with metabase.ObjectLocation

Another change to use metabase.ObjectLocation across satellite codebase
to avoid duplication and provide better type safety.

Change-Id: I82cb52b94a9107ed3144255a6ef4ad9f3fc1ca63
2020-09-08 21:09:34 +02:00
Michal Niewrzal
aa47e70f03 satellite/metainfo: use metabase.SegmentKey with metainfo.Service
Instead of using string or []byte we will be using dedicated type
SegmentKey.

Change-Id: I6ca8039f0741f6f9837c69a6d070228ed10f2220
2020-09-03 15:11:32 +00:00
Yingrong Zhao
af773ec8a6 cmd/uplink: use DeleteBucketWithObjects for bucket force deletion
This PR updates `uplink rb --force` command to use the new libuplink API
`DeleteBucketWithObjects`.
It also updates `DeleteBucket` endpoint to return a specific error
message when a given bucket has concurrent writes while being deleted.

Change-Id: Ic9593d55b0c27b26cd8966dd1bc8cd1e02a6666e
2020-09-02 16:39:20 +00:00
Isaac Hess
2d54447088 satellite/metainfo: Fix GetObjectIPs loop and test
This change forces the test of GetObjectIPs to use multiple remote
segments (earlier versions of the test were accidentally using inline
segments). This change also revealed a small bug in the for loop code,
which is fixed.

Change-Id: Ic486b079d221952ba13553acf0ca41a8873f3f21
2020-08-31 15:56:12 -06:00
Isaac Hess
351aa70eb7 satellite/metainfo: Implement GetObjectIPs
Change-Id: Ibabbe7c555b790498d28a6ac4c95fcf2f7376978
2020-08-31 10:27:56 -06:00
Egon Elbre
9225fc5aef satellite/accounting: simplify ExceedsBandwidthUsage call
Change-Id: I5376da2329e44da8f060226d2a76432df0acdaa0
2020-08-28 18:10:02 +03:00
Egon Elbre
3ca405aa97 satellite/orders: use metabase types as arguments
Change-Id: I7ddaad207c20572a5ea762667531770a56fd54ef
2020-08-28 15:52:37 +03:00
Yingrong Zhao
bd5213f68b satellite/metainfo: implement batch delete for DeleteBucket
This PR changes DeleteBucket to be able to delete all objects within a
bucket if `DeleteAll` is set in `BucketDeleteRequest`.
It also changes `DeleteBucket` API to treat `ErrBucketNotFound` as a
successful delete operation instead of returning an error back to the
client.

Change-Id: I3a22c16224c7894f2d0c2a40ba1ae8717fa1005f
2020-08-24 13:28:09 +00:00
Yingrong Zhao
0518b16370 satellite/piecedeletion: move node info retrieval into the service
This change will require less work for the user of peiecedeletion
service by moving overlay database call into the package.

Change-Id: I14a150ab71fe885780e7a7a74db006a779507ae5
2020-08-13 16:07:54 +00:00
Yingrong Zhao
b71da59f8a satellite/metainfo: fix client context cancelation
We should still ignore client cancelation.
Also remove logger dependency on `Report`.

Change-Id: I59987c6b1aaa93202b07e5ef6e207d5191542b52
2020-08-11 20:58:23 +00:00
Isaac Hess
a14887e20b satellite/metainfo: Add GetObjectIPs method
This adds the unimplemented GetObjectIPs method to metainfo endpoint so
we can import new common protobuf definitions.

Change-Id: I154f26baccb6bb3c66de3eb25611930545c9754b
2020-08-11 11:39:15 -06:00
Egon Elbre
94a09ce20b all: add missing dots
Change-Id: I93b86c9fb3398c5d3c9121b8859dad1c615fa23a
2020-08-11 17:50:01 +03:00
Yingrong Zhao
2b2cb468bf satellite/metainfo: implement DeleteObjectPieces with objectdeletion package
Use new objectdeletion package for deleting pointers.

In the best case scenario, it will make on database call to fetch
information about the number of segments. And another request to delete
and fetch information about other segments.

This PR also changes our object deletion API to return no error when an
object is not found but instead consider such operation as success. This
behavior is asligned with S3 API and makes the code less complex.

Change-Id: I280c56e8b5d815a8c4dafe8227689467e899775a
2020-08-06 16:10:13 -04:00
Michal Niewrzal
ceb2eee494 satellite/metainfo: make overwrites impossible without delete permission
https://storjlabs.atlassian.net/browse/USR-1007

Change-Id: Ic7fa600a00a53b80a37acee4ac994be6ea4cb279
2020-08-06 18:01:44 +00:00
Michal Niewrzal
88dcc93f3c satellite/metainfo: use user PartnerID for bucket attribution
Change-Id: I20f1bd432333f9b37ca8fb457c349eff94ffb392
2020-08-06 13:14:07 +00:00
Rafael Gomes
935f44ddb7 satellite/metainfo: Add Delete Service config
Change-Id: I0a6e3ce1adfe1488eb23da9dda92877af1834599
2020-08-03 14:28:02 +00:00
Michal Niewrzal
e69cec0b75 satellite/metainfo: stop generating StreamID for BeginDeleteObject
request

We are no longer using `BeginDeleteSegment` or `ListSegments` so we can
avoid generating StreamID as a result of `BeginDeleteObject`.

StreamID from `BeginDeleteObject` is also not used on Uplink side.

Change-Id: I3b068deab17068459849b5cf05811cad4b8a9034
2020-07-17 17:51:06 +00:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
Jennifer Johnson
784a156eea satellite: prevents uplink from creating a bucket once it exceeds the max bucket allocation.
Change-Id: I4b3822ed723c03dbbc0df136b2201027e19ba0cd
2020-07-15 17:27:05 +00:00
Isaac Hess
34b019d16a satellite/metainfo: Handle revocation request
Logic to handle a request to revoke a macaroon.

Change-Id: I5d5c93fcc2e026b0aaf82bfdfacc822185b10f9f
2020-06-22 13:50:07 -06:00
Isaac Hess
2d727bb14e satellite: Check macaroon revocation
When a request comes in on the satellite api and we validate the
macaroon, we now also check if any of the macaroon's tails have been
revoked.

Change-Id: I80ce4312602baf431cfa1b1285f79bed88bb4497
2020-06-22 13:50:07 -06:00
Egon Elbre
6cc7fd5f31 satellite/metainfo: remove old endpoints
Change-Id: Ic85c27caeea08879714e8303b285b6136f9a7fd1
2020-06-16 16:12:14 +03:00
Isaac Hess
05a981bc13 satellite/metainfo: Add RevokeAPIKey stub endpoint
Change-Id: I07fb997dda48ea204aab11f656a1a37762684339
2020-06-16 07:03:02 -06:00
Egon Elbre
19d431ff38 satellite/metainfo: disable old endpoints
First step in removing endpoints that were used by pre 1.0 clients.

Change-Id: Id861c73ffac97139aa9540f4973ea849e572f906
2020-06-16 11:22:44 +00:00
Rafael Gomes
958ea1b9df satellite/accounting: add download limit cache
Change-Id: I722930cab8bd5d240f4878dc6997e9bc7637311f
2020-06-12 16:33:46 -03:00
Michal Niewrzal
e8c4010311 satellite/metainfo: add missing error check
Change-Id: I4043633505a3eaee8304f71f5ddea17de2b3d10f
2020-06-02 09:38:55 +00:00
Moby von Briesen
b82d04e618 satellite/metainfo: limit size of uplink-provided metadata to 2KiB
Change-Id: Id44a46046ddb4a12102525531f4502fcff2b6252
2020-06-01 16:51:29 -04:00
Egon Elbre
d6c90b7ab5 pkg/macaroon: remove aliased package
Change-Id: I8b06e9a794d00752948f190623006e1afffdb031
2020-05-29 13:34:20 +00:00
Egon Elbre
7f323754a4 metainfo/piecedeletion: use NodeURL-s
Change-Id: I247dbfe03e7864e940e4cd1d0f343f38e84099e0
2020-05-21 08:37:13 +03:00
littleskunk
1b06e2a928
satellite/metainfo: remove deprecated error validation (#3876) 2020-05-18 19:58:42 +02:00
Egon Elbre
4e94da3fda satellite/overlay: add feature flag for node selection cache
Also distinguish the purpose for selecting nodes to avoid potential
confusion, what should allow caching and what shouldn't.

Change-Id: Iee2451c1f10d0f1c81feb1641507400d89918d61
2020-05-06 16:13:47 +03:00
Egon Elbre
7e0e74c65c satellite/metainfo: attribution based on useragent
Automatically attach attribution information to bucket during
BeginObject or CreateBucket when the UserAgent is set.

Change-Id: I405cb26c5a2f7394b30e3f2cf5d2214c8781eb8b
2020-04-17 10:47:14 +03:00
Kaloyan Raev
a2ce836761 remove sugar logging
Change-Id: I6b6ca9704837cb3f5f5449ba7f55661487814d9f
2020-04-15 12:37:47 +00:00
Egon Elbre
c97131ae78 satellite/metainfo: organize attribution methods
Change-Id: I4f35599c3f923861b5b05b128bf904480679f5f9
2020-04-14 11:27:43 +03:00
Michal Niewrzal
f36e8548f1 satellite/metainfo: adjust max inline segment size validation to
potential encryption overhead.

This is the same approach we have for validating remote segment size.

https://storjlabs.atlassian.net/browse/USR-619

Change-Id: I2597ee734313a3068fd986001680bbedbf1bed2a
2020-04-09 12:34:10 +00:00
Egon Elbre
11a44cdd88 all: don't depend on gogo/proto directly
Change-Id: I8822dea0d1b7b99e0b828e0373a0308a42dde2be
2020-04-08 17:32:15 +00:00
Michal Niewrzal
2d1a6968b0 satellite/metainfo: revert returning segment size and inline segment
size with BeginObject

Such solution will add one round trip to satellite during upload so for
now we are reverting this until we will have solution for this.

Change-Id: Ic2d826448ab7b0318cd6922df05deee9167cf2f0
2020-04-06 13:36:34 +02:00
Matt Robinson
4b01a8dd18
Descellate Usage Error to Debug (#3780) 2020-04-03 13:20:20 +02:00
Egon Elbre
6492b13d81 all: remove old uuid
Change-Id: I3a137f73456f010c37d3933dbe12cbbb840b809f
2020-04-02 19:30:36 +03:00
Michal Niewrzal
c178a08cb8 satellite/metainfo: add max segment size and max inline size to
BeginObject response

We want to control inline segment size and segment size on satellite
side. We need to return such information to uplink like with redundancy
scheme.

Change-Id: If04b0a45a2757a01c0cc046432c115f475e9323c
2020-04-02 12:41:28 +00:00
Michal Niewrzal
4a79b609e9 satellite/metainfo: fix panic when we batch BeginObjectDelete without
all permissions

Without read and list permissions BeginObjectDelete won't return error
if occurs. This was breaking Batch processing because there was
assumption that without error response will be always not nil.

https://storjlabs.atlassian.net/browse/SM-590

Change-Id: I0fc9539e429110a660eb28725b266d5e4771d198
2020-04-02 12:20:19 +00:00
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
2020-03-31 19:16:41 +03:00
Egon Elbre
1d79228ed0 satellite/metainfo: support uplink useragent
Adds support for parsing user agent and specifying uplink version and
it's library dependencies.

Change-Id: Ibaddde4deb93e153ac05c91b676c5b5f1ae1aa37
2020-03-31 15:11:31 +00:00
Kaloyan Raev
6c44512c15 satellite/metainfo: increase stream ID expiration to 48h
Otherwise, it is not possible to upload very large files like 1 TB.

Change-Id: Iadda1ae91174125736684850b906d5bb6d19f0a9
2020-03-23 15:47:39 +02:00
Egon Elbre
1b6ab173a8 private/context2: moved to storj.io/common/context2
Change-Id: Ic1dd1ed645ff3e1057c9b2b143e2c3ddf29d678e
2020-03-20 14:39:46 +00:00
Egon Elbre
eb1d8aab96 satellite/metainfo/pointerverification: service for verifying pointers
This implements a service for pointer verification. This makes the
slightly clearer, because it's not part of metainfo.

It also adds a peer identity cache which reduces database calls and peer
identity decoding.

Change-Id: I45da40460d579c6f5fd74c69bccea215157aafda
2020-03-19 16:27:38 +00:00
Egon Elbre
09e0f3de63 satellite/metainfo/piecedeletion: add Service
Change-Id: Id7e32ed569701fa0be66f9527c43a67052994570
2020-03-18 14:50:08 +00:00
Michal Niewrzal
81afbcc12e satellite/metainfo: check bucket existence on upload and listing
Initial change for checking bucket existence on satellite side for
requests like BeginObject and ListObjects. This is simple implementation
that is just checking bucket in DB but should be improved in future to
avoid DB calls as much as possible.

Part of https://storjlabs.atlassian.net/browse/USR-365

Change-Id: I9076acddc44d7dbfa7612a1c24a007de01621583
2020-03-17 15:43:22 +00:00
Kaloyan Raev
27f811a9e1 metainfo: delete methods return the deleted item
This only happens if Read or List permission is granted together with
the Delete permission

Change-Id: I68b5f04a476bddabe499809ac98097aac75732a8
2020-03-16 16:26:16 +02:00
Michal Niewrzal
16878a22ea satellite/metainfo: stops hiding real validateAuth
Metainfo method validateAuth checks things like API key, user permission
and rate limit but at the end all errors were returned as
rpcstatus.Unauthenticated.

Old Metainfo is not touched to avoid backward compatibility issues.

Change-Id: I78eb276210fc50151da58a5c84e13ecd0961da29
2020-03-10 11:53:00 +00:00
Michal Niewrzal
9f390f37da satellite/metainfo: return default ciphers (path and encryption) for old
uplinks

New libuplink is not storing encryption values in with bucket but old
uplinks are using those values for configuration. If bucket was created
with new libuplink we will send back satellite defaults.

Change-Id: Ie1bf3682847e07b302270b4c4bf1a7219f4bf011
2020-03-05 10:04:50 +00:00
Jennifer Johnson
1c1750e6be removes bandwidth limiting
On satellite, remove all references to free_bandwidth column in nodes table.
On storage node, remove references to AllocatedBandwidth and MinimumBandwidth and mark as deprecated.

Protobuf message, NodeCapacity, is left intact for backwards compatibility.
Once this is released to all satellites, we can drop the column from the DB.

Change-Id: I2ff6c6537fc9008a0c5588e951afea58ede85838
2020-03-04 14:04:00 +00:00
Michal Niewrzal
4deab5ac6c satellite/metainfo: combine CommitSegment and CommitObject in batch v2
This change is a special case for batch processing. If in batch request
CommitSegment and CommitObject are one after another we can execute
these requests as one. This will avoid current logic where we are saving
pointer for CommitSegment and later we are deleting this pointer and
saving it once again as under last segment path for CommitObject.

This change should handle issue we have in older uplinks with incorrect
order of storing pointers.

Change-Id: I86514c95df169e6fbc91b52e5117472cae70cb8b
2020-02-28 07:40:36 +00:00
Michal Niewrzal
54e38b8986 pkg/miniogw: gateway implementation with new libuplink
Change-Id: I170c3a68cfeea33b528eeb27e6aecb126ecb0365
2020-02-21 16:20:38 +01:00
Egon Elbre
5342dd9fe6 go.mod: update uplink
Change-Id: I867a6a1eef8aa5d60bb676e5112b98c4192ce811
2020-02-21 16:08:12 +02:00
Michal Niewrzal
dbe8428f9f satelite/metainfo: return NotFound on delete non existing bucket
Change-Id: I7f466b5f824eab7b5146c2792f40cb2bcd7976a5
2020-02-18 09:05:30 +00:00
Egon Elbre
ef2f101495 satellite/metainfo: don't allow deleting non-empty bucket
Change-Id: I72a8b959e954c7f52e93fc8ea4006a957cc2941a
2020-02-14 14:36:22 +01:00
Kaloyan Raev
37cf42a9ae satellite/metainfo: overwrite zombie segments
Fixes https://storjlabs.atlassian.net/browse/USER-240

- Adds UnsynchronizedPut method to metainfo service that overwrites any
existing pointer under the same path
- Uses UnsynchronizedPut in the metainfo endpoint for committing the
segments

Change-Id: Icb43f31ea33f14066ca9dfdcf226eb3079b90948
2020-02-12 11:10:38 +00:00
Cameron Ayer
b22bf16b35 satellite/overlay: add config flag for node selection free disk requirement
Currently SNs report their free disk space once per hour. If a node
becomes full, it has to wait until the next contact cycle begins to
report; all the while receiving and failing upload requests. By increasing
the minimum required disk space, we can give the storage nodes more time
to report their space before the completely fill up. This change goes
hand-in-hand with another change we want to implement: trigger capacity
report on SN immediately upon falling below threshold.

Change-Id: I12f778286c6c3f582438b0e2949765ac43325e27
2020-02-11 18:08:25 +00:00
Jeff Wendling
7999d24f81 all: use monkit v3
this commit updates our monkit dependency to the v3 version where
it outputs in an influx style. this makes discovery much easier
as many tools are built to look at it this way.

graphite and rothko will suffer some due to no longer being a tree
based on dots. hopefully time will exist to update rothko to
index based on the new metric format.

it adds an influx output for the statreceiver so that we can
write to influxdb v1 or v2 directly.

Change-Id: Iae9f9494a6d29cfbd1f932a5e71a891b490415ff
2020-02-05 23:53:17 +00:00
Ethan
149273c63f satellite/metainfo: add cache expiration for project level rate limiting
Allow rate limit project cache to expire so we can make project level rate limit changes without restarting the satellite process.

Change-Id: I159ea22edff5de7cbfcd13bfe70898dcef770e42
2020-01-29 16:14:10 +00:00
Stefan Benten
d30d2d920d satellite/metainfo: Adding Monkit Meters to the Request Logs
Change-Id: I33d56510cf72d5f8512c1069ce65856cba7f8957
2020-01-29 15:51:36 +00:00
Jessica Grebenschikov
a1948ed338 satellite/orders: add old method for CreateGetOrderLimitsOld to maintain compatibility with old versions of the uplink
Change-Id: I7ce1f4fbc6217f1d340cf778c4b010d40961b3f0
2020-01-28 18:54:24 -05:00
Jessica Grebenschikov
54dbaaece2 satellite/orders: create as many orderLimits as needed to download a file
Change-Id: I2a39483d35037d9940913c035a78a93ea692ce9f
2020-01-28 20:04:11 +00:00
Michal Niewrzal
90fc1922d0 satellite/metainfo: override bucket RS values with satellite config
Satellite now is keeping RS values for uplink but old uplinks were using
default bucket settings. Because of that we need to override buckets
settings with satellite settings to avoid breaking older uplinks.

Change-Id: Ia1068db70e4adbf741c5e81d27d9e39799049c22
2020-01-28 15:51:04 +00:00
Yingrong Zhao
f3fcbe256c satellite/metainfo: revert combine CommitSegment and CommitObject in batch
This reverts commit 8772867855.

for uplink versions v0.25.0 through v0.30.7, there's a bug with multiplesegment upload
where the last segment is inline caused by this commit.

Change-Id: If375e186b23265586caf08991c25980e99f3cc1a
2020-01-27 13:26:33 -05:00
Michal Niewrzal
ca32ffbfc5 satellite/metainfo: move deletion before upload to satellite
Change is adding object deletion to BeginObject request (before upload).
Now when satellite controls deletion we can move deletion before upload
to satellite. This change improves two things:
* no need for additional request to delete object before upload (need
one more change to storj/uplink)
* fix an issue with lack of permissions to upload if caveat allows only
for writing (e.g. disallow deletes but allows to write)

https://storjlabs.atlassian.net/browse/V3-3362

Change-Id: Ic453146298cdd302df290c532123731a3f99e38e
2020-01-27 12:48:10 +00:00
Ivan Fraixedes
d5a60aec58
satellite/metainfo: Delete segments in reverse order
Change DeleteObjectPieces for deleting the segments' pointers of an
object in a reverse order.

Last segment: L
N: total number of segments

Deleting in reverse order is: L, n-1 to 0

Deleting in reverse order makes BeginDeleteObject usable to delete
partially uploaded objects that were interrupted (e.g. upload
cancellation).

With this change, the uplink upload cancellation, can be changed to use
BeginDeleteObject to cleanup already uploaded segments without having to
retrieve orders and dial every single node which stored a piece.

Ticket: https://storjlabs.atlassian.net/browse/V3-3525
Change-Id: Ieca6fd3801c4b71671811cb5f08a99d5146928a6
2020-01-24 16:05:12 +02:00
Michal Niewrzal
6502454947 satellite/metainfo: move RS configuration to satellite
With this change RS configuration will be set on satellite. Uplink with
get RS values with BeginObject request and will use it. For backward
compatibility and to avoid super large change redundancy scheme stored
with bucket is not touched. This can be done in future.

Change-Id: Ia5f76fc10c37e2c44e4f7b8754f28eafe1f97eff
2020-01-22 09:33:53 +00:00
Ethan
21a5d70a83 satellite/metainfo: Rate limiting - API requests
Limits how many times metainfo APIs can be called per second by project ID. If limit is exceeded, the API will return Unauthorized/Too Many requests.

Limit per second and the size of the limiter cache per project are configurable, as well as whether the limiter is enabled.

Tests added/updated for the new rate_limit field in projects table.
Tests added for exceeding limits and disableing limiter.

Change-Id: Ic8ad102de3b690a475809d4f684156d5715f20fa
2020-01-21 14:25:04 +00:00
Michal Niewrzal
8772867855 satellite/metainfo: combine CommitSegment and CommitObject in batch
This change is a special case for batch processing. If in batch request
CommitSegment and CommitObject are one after another we can execute
these request as one. This will avoid current logic where we are saving
pointer for CommitSegment and later we are deleting this pointer and
saving it once again as under last segment path for CommitObject.

Change-Id: If170e78c8410f5ba5916cbff6a29b9221db9ce2e
2020-01-21 09:30:55 +00:00
stefanbenten
b678b55f83 satellite/metainfo: improve metainfo logging
Change-Id: Ibc06393bfb996e1cad9bb76c451e3faf2393b478
2020-01-20 18:49:55 +00:00
Cameron Ayer
4424697d7f satellite/accounting: refactor live accounting to hold current estimated totals
live accounting used to be a cache to store writes before they are picked up during
the tally iteration, after which the cache is cleared. This created a window in which
users could potentially exceed the storage limit. This PR refactors live accounting to
hold current estimations of space used per project. This should also reduce DB load
since we no longer need to query the satellite DB when checking space used for limiting.

The mechanism by which the new live accounting system works is as follows:

During the upload of any segment, the size of that segment is added to its respective
project total in live accounting. At the beginning of the tally iteration we record
the current values in live accounting as `initialLiveTotals`. At the end of the tally
iteration we again record the current totals in live accounting as `latestLiveTotals`.
The metainfo loop observer in tally allows us to get the project totals from what it
observed in metainfo DB which are stored in `tallyProjectTotals`. However, for any
particular segment uploaded during the metainfo loop, the observer may or may not
have seen it. Thus, we take half of the difference between `latestLiveTotals` and
`initialLiveTotals`, and add that to the total that was found during tally and set that
as the new live accounting total.

Initially, live accounting was storing the total stored amount across all nodes rather than
the segment size, which is inconsistent with how we record amounts stored in the project
accounting DB, so we have refactored live accounting to record segment size

Change-Id: Ie48bfdef453428fcdc180b2d781a69d58fd927fb
2020-01-16 10:26:49 -05:00
Ivan Fraixedes
6dc948da46
satellite/metainfo: Create service for deleting pieces
Create a service for deleting pieces of storage nodes.

Currently the DeletePieces method returns after a success threshold,
completion or a timeout.

The end goal is to return when reaching the success threshold and
leaving the remaining goroutines running after DeletePieces method
returns and add a life cycle to the service that it waits for them when
it closes.

This is the first commit for ticket:
https://storjlabs.atlassian.net/browse/V3-3476

Change-Id: If740bbf57c741f880449980b8176b036dd956c7b
2020-01-15 15:24:20 +01:00
Egon Elbre
02ed2b5a19 satellite/metainfo: disallow cancelling deletion request
Change-Id: I05194d493632c9f454671ddf6b67b4901f2b7884
2020-01-15 14:23:46 +02:00
Ivan Fraixedes
5cd6058318 satellite/metainfo: Add back-pressure mechanism DeleteObjectPieces
Add a back-pressure mechanism to the satellite metainfo
DeleteObjectPieces method for returning once the 75% of successful
deleted pieces is reached.

Change-Id: Ia38df49fba5838f0605c40a77cfff8e3442cb5b0
2020-01-09 08:22:11 +00:00
littleskunk
05e4a86654
satellite/metainfo: Close client DeleteObjectPieces
The DeleteObjectPieces should print out the warning on closing the
connections only if there was an error.

Change-Id: If3d7ab256d8508c08388c1f22c7dd1eb819d2509
2020-01-08 15:54:11 +01:00
Ivan Fraixedes
922c43f921
satellite/metainfo: Close client DeleteObjectPieces
The DeleteObjectPieces must close the storage node client once it has
finished deleting its pieces.

Change-Id: I08eb8af8e4215d77d59b52f5055211b918374ab4
2020-01-08 15:01:34 +01:00
Egon Elbre
082ec81714
uplink: move to storj.io/uplink (#3746) 2020-01-08 15:40:19 +02:00
Ivan Fraixedes
027e3d4f62
satellte/metainfo: Avoid a noisy warning
DeleteObjectPieces must not call overlay cache KnownReliable method with
an empty list of node IDs for avoiding to log a useless noisy warning.

Change-Id: Ibe2a34f2913f003d3ba020f9764c1369fa63123b
2020-01-07 14:15:04 +01:00
Ivan Fraixedes
9618959f1d
satellite/metainfo: Use errs2 IsRPC helper function
Use the helper function IsRPC of the err2 package rather than checking
if an error is of a specific RPC status code with an 'if' conditional.

Change-Id: Ibe89d6c2d836307c3112a6d7cc6bf95f0f985fd2
2020-01-07 08:26:45 +01:00
Ivan Fraixedes
fc4ea28695 satellite/metainfo: Return ErrObjectNotFound
deletePointer must return an ErrObjectNotFound rather than a rpc status
error NotFound because the callers must distinguish such error if it
comes from the getPointer or from the UnsynchronizedDelete.

Change-Id: I68b4e45a2765e63b73bf85c2c39a5fc0198373f6
2020-01-03 22:01:29 +00:00
Michal Niewrzal
38eff60698 satellite/metainfo: adjust old API test to new API
We are missing some tests for new Metainfo API that we have for old API.
This is first change to adjust old tests to new API.

Change-Id: Ie2b16bf85de8633662f952e863dbf3d409d801d9
2020-01-03 11:05:14 +01:00
Ivan Fraixedes
c3b58f1656 satellte/metainfo: Make BeginDeleteObject to delete pieces
For improving the deletion performance we are shifting the
responsibility to delete the pieces of the object from Uplink to the
Satellite.

BeginDeleteObject was the first call to return the stream ID which was
used for after retrieving the list of segments and then get addressed
order limits for deleting the pieces (of each segment) from the storage
nodes.

Now we want the Satellite deletes the pieces of all the object segments
from the storage nodes hence we don't need anymore to have several
network round trips between the Uplink and the Satellite because the
Satellite can delete all of them in the initial BegingDeleteObject
request.

satellite/metainfo.ListSegments has been changed to return 0 items if
the pointer of the last segment of an object is not found because we
need to preserve the backward compatibility with Uplinks that won't be
updated to the last release and they rely on listing the segments after
calling BeginDeleteObject for retrieving the addressed order limits
to contact the storage nodes to delete the pieces.

Change-Id: I5f99ecf27d62d65b0a062936b9b17581ef692af0
2020-01-02 15:53:59 +00:00
Ivan Fraixedes
7266155375 satellite/metainfo: List segments manually limit check
The endpoint listSegmentsManually method misses a check for the limit
parameter, otherwise it can return inconsistent results when it's 0 or
negative.

When 0 or negative, without the check, it returns no segments but also
that there isn't more segments and that isn't correct.

The function is only called from the Endpoint.ListSegments method and
the function cares to ensure that limit is always greater than 0, but if
the method doesn't check that a new future caller could misuse it and
provoke a bug.

Additionally:

* Documentation for the modified function has been written
* The part of the function that repeated the logic of the
  Endpoint.getPointer method has been removed for using that method.
* Added logging before returning an internal error in
  Endpoint.getPointer.

Change-Id: I5c4f0db2292da0162db6b7d63553895808d0925a
2019-12-31 07:44:46 +00:00
Ivan Fraixedes
059537c16e satellite/metainfo: Add new TODOs & remove old ones
Do some cleanup for adding new identified TODOs (associated with ticket
https://storjlabs.atlassian.net/browse/V3-3406) and remove an old one.

Change-Id: I5d20dbe1c4dee0a8279e08b05b907f4cc9dba278
2019-12-27 13:16:09 +00:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Kaloyan Raev
f8d0864630 satellite/metainfo: use KnownReliable in DeleteObjectPieces
This reduces the number of queries to the overlay when deleting objects.

Change-Id: I28ed2c2d225e0c5eb1a8d952235fa7e5837a48d1
2019-12-18 12:38:22 +00:00
Ivan Fraixedes
366f4b9493
satellite: Create method for deleting pieces of SNs
Create a method for deleting pieces of the storage nodes and add a test
for it.

Change-Id: I5fffc8c43d900d317961098b1d135ce3223b73ea
2019-12-18 10:38:16 +01:00
Vitalii Shpital
53d9bc4530
storagenode/notifications: db created (#3707) 2019-12-16 19:59:01 +02:00
Ivan Fraixedes
770123de10 satellite/metainfo: Improve docs & use common funcs
* Use unexported existent method in logic that was duplicated in some
  exported methods.
* Log a forgotten internal error.
* Improve the documentation adding more and fixing some to fit to our
  code style conventions.

Change-Id: Ie6f8bc59f9089f92b8b0d1b4c09c2142c3f273f5
2019-12-13 16:00:35 +00:00
Ivan Fraixedes
03fa0150bc satellite/metainfo: Trace endpoint getPointer method
The Endpoint.getPointer method lacked of tracing.
Also add a dot at the end of documentation comment for following our
code style conventions.

Change-Id: I9b63ad297f04e31825648aae43aa8f9ebba2b4e2
2019-12-12 13:55:19 +00:00
Ivan Fraixedes
0d1ba7bc08 satellite/metainfo: Return error misusing func
Return an error when misusing the endpoint method
'listSegmentsFromNumberOfSegments' because there is the method
'listSegmentsManually' for being used when the number of segments is
less or equal than 0.

If we don't return an error on `listSegmentsFromNumberOfSegments` we
would realize that we have a bug much more later than returning an error
because the clients wouldn't receive an error and would receive an empty
list, making them to wonder what they are doing wrong to receive 0
results before they realize that they could be in front of a bug.

This commit also renames the function to be plural as "numberOfSegments"
parameter and the test function which missed also the end 's'.

Change-Id: I02318685bf36aa3af26545731a1711621a5e2e39
2019-12-11 10:45:56 +00:00
Egon Elbre
72d407559e satellite/metainfo: don't leak error implementation detail (#3722)
* satellite/metainfo: don't leak implementation detail

* add missing wrap
2019-12-10 15:21:30 -05:00
Ivan Fraixedes
c400471bbc
satellite/metainfo: Fix some docs comments
Fix a documentation comment for one method and apply our code
conventions to some that I stumbled.

Change-Id: I3baf5d004a128dcd561c3e27c080aab345c64461
2019-12-06 19:14:35 +01:00
Yingrong Zhao
7af42e3c10
satellite/metainfo, satellite/repair, uplink/eestream: add metric for download failed due to not enough pieces available (#3665) 2019-12-04 16:24:36 -05:00
Ivan Fraixedes
d69482e938
satellite/metainfo: Improve piece hash validation (#3671)
Improve the piece hash validation filtering out a piece when an order
limit is not found for it.

The commit also improves the documentation of an internal metainfo
method and rename the parameters of 2 methods for clarifying what they
are.
2019-12-03 14:36:32 +01:00
Fadila
526a126a5a
Satellite: log the project information for upload, download, and list activities (#3651) 2019-12-02 15:39:19 +01:00
Egon Elbre
36fead0093 satellite/metainfo: add UserAgent support to endpoints (#3548) 2019-11-26 03:12:37 -08:00
Michal Niewrzal
1aa2bc0a83
satellite/metainfo: reduce pointerDB access for CommitObject (#3589) 2019-11-24 13:14:51 -08:00
Egon Elbre
91859f1ccf
satellite/metainfo: fix broken test (#3580) 2019-11-15 17:44:23 +02:00
Yehor Butko
a8e4e9cb03
satellite/payments: project usage charges (#3512) 2019-11-15 16:27:44 +02:00
Ivan Fraixedes
aa7b5b7c53
satellite/metainfo: avoid large conditional block (#3579)
Large conditional blocks are hard to read.
When the conditional block only has one branch it's easy to understand
the logic of the function to early return switching the condition.
2019-11-15 14:33:09 +01:00
paul cannon
0c025fa937 storage/: remove reverse-key-listing feature
We don't use reverse listing in any of our code, outside of tests, and
it is only exposed through libuplink in the
lib/uplink.(*Project).ListBuckets() API. We also don't know of any users
who might have a need for reverse listing through ListBuckets().

Since one of our prospective pointerdb backends can not support
backwards iteration, and because of the above considerations, we are
going to remove the reverse listing feature.

Change-Id: I8d2a1f33d01ee70b79918d584b8c671f57eef2a0
2019-11-12 18:47:51 +00:00
Egon Elbre
cc032d3151 satellite/metainfo: fix some uses of metainfo.Delete (#3513)
* satellite/metainfo: rename Delete to UnsynchronizedDelete

* fix deletes

* make db private

* fix typos

* also verify on commit object
2019-11-06 18:02:14 +01:00
littleskunk
7eb6724c92
logging: unify logging around satellite ID, node ID and piece ID (#3491)
* logging: unify logging around satellite ID, node ID and piece ID

* unify segment index
2019-11-05 22:04:07 +01:00
Ethan Adams
2eb0cc56fe
satellite/gracefulexit: Check if node already has a piece in the pointer (#3434) 2019-11-05 14:13:45 -05:00
Michal Niewrzal
7f6893ea86
satellite/metainfo: fix broken object listing (#3348) 2019-10-24 14:05:08 -07:00
Ivan Fraixedes
071d1c4313
upload: Add more info to returned error response & to logs (#3218)
* uplink/storage/segments: return error no optimal threshold
  Return an error if the store get less uploaded pieces than the indicated
  by the optimal threshold.

* satellite/metainfo: Fix gRPC status error & add reason
  This commit fix the CommitSegment endpoint method to return an
  "Invalid Argument" status code when uplink submits invalid data which is
  detected when filtering invalid pieces by filterInvalidPieces endpoint
  method.

  Because filterInvalidPieces is also used by CommitSegmentOld, such
  method part has been changed accordingly.

  * An initial check in CommitSegment to detect earlier if uplink sends an
    invalid number of upload pieces.
  * Add more information to some log messages.
  * Return more information to uplink when it sends a number of invalid
    pieces which make impossible to finish the operation successfully.

* satellite/metainfo: Swap some "sugar" loggers to normal ones
  Swap "sugar" loggers to normal ones because they impact the performance
  in production systems and they should only be used under specific
  circumstances which were none of the ones changed.
2019-10-17 20:01:40 +02:00
littleskunk
2301a8287f Satellite/PieceHashValidation: Increase time window from 2h to 24h to avoid timezone issues (#3291) 2019-10-16 06:47:08 -06:00
Maximillian von Briesen
08ed50bcaa
satellite/metainfo: add commit interval to prevent long delays between order limit creation and segment commit (#3149) 2019-10-01 12:55:02 -04:00
Maximillian von Briesen
edadf46009 satellite/audit: delete nodes from containment when segment has changed (#3115) 2019-09-29 04:03:15 +02:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
Michal Niewrzal
607da4ab4a
metainfo: move FinishDeleteSegment logic to BeginDeleteSegment (#3104) 2019-09-23 14:41:58 -07:00
JT Olio
946ec201e2
metainfo: move api keys to part of the request (#3069)
What: we move api keys out of the grpc connection-level metadata on the client side and into the request protobufs directly. the server side still supports both mechanisms for backwards compatibility.

Why: dRPC won't support connection-level metadata. the only thing we currently use connection-level metadata for is api keys. we need to move all information needed by a request into the request protobuf itself for drpc support. check out the .proto changes for the main details.

One fun side-fact: Did you know that protobuf fields 1-15 are special and only use one byte for both the field number and type? Additionally did you know we don't use field 15 anywhere yet? So the new request header will use field 15, and should use field 15 on all protobufs going forward.

Please describe the tests: all existing tests should pass

Please describe the performance impact: none
2019-09-19 10:19:29 -06:00
Michal Niewrzal
b07a490f95 satellite/metainfo: fix GetObject method (#3088) 2019-09-19 11:13:57 +02:00
Michal Niewrzal
1c72e80e40 uplink/satellite: fix for case when inline segment is last one (#3062)
* uplink/satellite: fix when inline seg is last one

* review comments
2019-09-19 01:18:14 +02:00
Maximillian von Briesen
574c96c350
satellite/metainfo: Verify storagenode signature on satellite upload (#2985) 2019-09-18 09:50:33 -04:00
Egon Elbre
a801fab66a
all: add archview annotations (#2964) 2019-09-10 16:24:16 +03:00
Michal Niewrzal
a6721ba92f
satellite/metainfo: Improve metainfo ListSegments (#2882) 2019-08-30 23:30:18 +02:00
Michal Niewrzal
5fb823843f
Fix downloading non encrypted segments (#2870) 2019-08-29 10:00:20 +02:00
Egon Elbre
56383de298
satellite/metainfo: use status.Error and fix error codes (#2827) 2019-08-20 16:16:51 +03:00
Michal Niewrzal
83461a7c01 satellite/metainfo: fix storing enc key/nonce (#2759) 2019-08-15 14:45:49 +03:00
Egon Elbre
c8edeb0257
satellite/overlay: rename overlay.Cache to overlay.Service (#2717) 2019-08-06 19:35:59 +03:00
Michal Niewrzal
688d932d93
Make one implementation for SetAttribution/SetBucketAttribution (#2683) 2019-08-05 09:07:40 +02:00
Michal Niewrzal
287fdf9936
Integrate new Metainfo calls (server side) (#2682) 2019-08-01 11:04:31 +02:00
Natalie Villasana
238e264a8f
satellite confirms success threshold reached (#2657) 2019-07-31 14:28:43 -04:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Michal Niewrzal
5710dc3a32
Metainfo RPC segment methods (part 2) (#2616) 2019-07-24 13:33:23 +02:00
Michal Niewrzal
cba008d7df
Add GetObject method to Metainfo (#2611) 2019-07-23 13:09:12 +02:00
Michal Niewrzal
6f2b85603d
Metainfo RPC segment methods (part 1) (#2567) 2019-07-22 16:45:18 +02:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588)
* value attribution merge fix and more test cases
2019-07-19 11:17:34 -04:00
Michal Niewrzal
0b827f1375
Speedup metainfo tests (#2556) 2019-07-16 21:16:41 +02:00
Michal Niewrzal
260d9c49a8
Metainfo RPC objects methods (#2534) 2019-07-16 12:39:23 +02:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Alexander Leitner
1c5db71faf
Change protobuf expirations to use time.Time (#2509)
* Change protobuf expirations to use time.Time instead of timestamp.Timestamp
2019-07-09 17:54:00 -04:00
Egon Elbre
910e4744ad
satellite/metainfo: optimize pointerdb size (#2506) 2019-07-09 16:36:18 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database (#2460)
* add db interface and methods, add sa metainfo endpoints and svc

* add bucket metainfo svc funcs

* add sadb bucekts

* bucket list gets all buckets

* filter buckets list on macaroon restrictions

* update pb cipher suite to be enum

* add conversion funcs

* updates per comments

* bucket settings should say default

* add direction to list buckets, add tests

* fix test bucket names

* lint err

* only support forward direction

* add comments

* minor refactoring

* make sure list up to limit

* update test

* update protolock file

* fix lint

* change per PR
2019-07-08 15:32:18 -07:00
Stefan Benten
16156e3b3d
Ensure we force a segment size and account storage before committing them (#2473) 2019-07-08 18:24:38 -04:00
aligeti
a554752ce0
adding expiration date check on satellite's CreateSegment (#2476)
* adding expiration check on satellite createsegment
2019-07-08 14:10:16 -04:00