We want to be able to limit the number of segments per project for users.
To limit this we need to check limit value associated with project
and value of used segments already in BeginMoveObject, BeginMoveSegment
and increment cache segments usage after each CommitSegment call.
Resolves https://github.com/storj/team-metainfo/issues/1
Change-Id: I6290e67c095a174b9d101c4521802d9bfe0453b8
At some point we missed to add metadata key to list objects
response. Because of that uplink is taking key from pb.SteamMeta.
We need to clean it up.
Tests will be added on uplink side.
https://github.com/storj/uplink/issues/71
Change-Id: I3328e2f1b86bca15aeaf89f8e59cdca3c8e97742
For backward compatibility we are overriding pb.StreamMeta
values returned as encrypted metadata. It turns out that we
should do it not when target values are missing but when
values to override exists.
This was causing problems after move operation. Details can be
found here https://github.com/storj/uplink/issues/70
Backward compatibility test will be added to storj/uplink testsuite.
Change-Id: I72e7a01226b1dd62902cb0d6ebb1ff91a4693005
Previously, only valid partner IDs could be used for bucket level value attribution. Now that any useragent byte slice can be used, we should allow for empty useragent strings to be stored rather than throwing an error or leaving the bucket with no attribution.
Change-Id: I7043f835588dab1c401a27e31afd74b6b5a3e44b
Currently detecting if error NotFound is for object
or bucket is very fragile on uplink side. We cannot
change this messages for now.
Change-Id: I6ec4d98116477812f031134e4f1c9e73bdce8b27
Check if the context is done at the beginning of the download object and
segment and return right away if it's the case.
Check if Redis returned an error due to context cancellation for not
logging the error.
Change some logging messages of Redis errors to reflect on them if the
error happens while downloading an object or a segment.
Change-Id: I8ed8ff9ff7bb170b560f41356ea06820ce6c4e12
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.
Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
Add the project ID to error logs messages about not being able to
retrieve and track storage or bandwidth usage.
Some error logs related to these errors already contained the project ID
but others didn't, having it in the ones that didn't make them more
consistent and will provide more info which may be helpful for
troubleshooting them.
Change-Id: Ia9fc707a7f3aff0867645bb941badc199c2bf832
Don't log as an Error when users make request that cannot be fulfilled
because they exceed their storage or bandwidth limits.
Those are not system errors and the service is handling them correctly
and showing them in the logs an "ERROR" is misleading.
Change-Id: Iac642b7e8ba92840bb943192ad0694b5f4930258
To resolve problem with lack of ability to set metadata while MPU on
gateway we are adding setting metadata with BeginObject. This
change makes also metadata optional while CommitObject. We need
this functionality to not override metadata set with BeginObject in
case when metadata is not set with CommitObject.
Another reson is that we would like to not set metadata at all if user
didn't specify metadata. At the moment we always setting some bytes
for metadata fields e.g. empty EncryptedMetadata field can have key
and nonce set.
Change-Id: Ifee25b7718eb1f919119db9b698b29d8b5ebe2ec
To reduce database load, add option to omit as many object properties
as possible when listing objects.
Change-Id: I817633801b00629a4042d1d1bd2389ee581953de
Removed PathCipher and DefaultSegmentSize from CreateBucketParams
since they are unused anymore and breaks Integration on uplink side
Change-Id: I1393a7f1f436940731aa59edd693043336383290
The UserAgent should be stored as is, with the exception of removing the trailing version from any libuplink user agents
Change-Id: If17ef2fc4b59480a3477300f2585a07d64cc2bf4
Currently slower storagenodes can slow down deletion queue.
To make piece deletion faster reduce the maximum time spent in
either dialing or piece deletion requests.
With this change:
* dial timeout is 3s
* request timeout is 15s
* fail threshold is set to 10min
Similarly, we'll mark storage node as failed when the timeout occurs.
The timeout usually indicates that the storagenode is overwhelmed.
Garbage collection will ensure that the pieces get deleted eventually.
Change-Id: Iec5de699f5917905f5807140e2c3252088c6399b
We needed Redundancy insided sat StreamID when uplink was defining RS values. Now it can be removed.
Change-Id: Id37187493eaa00cf29cb0262a050d71add3deb96
We should improve the way how we are handling metabase errors in
metainfo endpoint.
https://storjlabs.atlassian.net/browse/PG-316
Change-Id: I1da6f333546cabf34d6eb1de8e94a3ef455d75d5
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB
Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
Speedup is done by reducing number of testplanet instances
for tests without changing main test logic.
Change-Id: Ic3849485d37b8ca55c013a45b7191dce65b88b04
Uplink needs only part of columns we are reading from DB.
To improve performance we should read only those that are
realy needed.
Change-Id: Ib39259318169c46afe5fa4c6ce2184da82e960c8
This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.
This reverts commit 1ef06fae99.
Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.
Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
We are not using the benchmark results for anything, they are mostly
there to ensure that we don't break the benchmarks. So we can disable
CockroachDB for them.
Similarly add short versions of other tests.
Also try to precompile test/benchmark code.
Change-Id: I60b501789f70c289af68c37a052778dc75ae2b69
It's safer to create new connection pool for piece deletion
only if dialer have no existing pool assigned.
Change-Id: I26661683ab7c0198587905478057c01c8f533a7e
We should be using object naming insted of path.
This is one place where we can easiliy change it.
To regenerate protbuf I had to remove gogo.proto.
Most probably it was confilicting with gogo.proto
from common/pb.
Change-Id: Ia5972f77994765c8f26bf1c3dc8205d2eadd70fa
We want to enable connection pool for piece deletion to avoid
doing multiple SSL hanshakes to SN while massive deletion process.
Change-Id: Ic917e4eda304ee16a286926ef046fe9e38bf38ca
This PR utilize the new burst limit column from projects table to allow
control on the limit for request per seconds and token bucket size
When no burst limit is explicitly set, rate limit is applied to both so
we don't limit how quickly request can be made in a second.
Change-Id: I883235c60c5d6416aeadd1c80ed2ebd193aa4d9f
server-side move extended with moving between buckets, for this reason
we change bucket name for object in db.
Change-Id: Ie21bcccc170e6ff14dcd8053fdb86fdf6d8438a0
Some processing inside storagenodes is async compared to uplink upload
and download, hence we need to explicitly wait for storagenodes to
finish their pending work before flushing orders to the satellite.
Hopefully this fixes TestAttributionReport flakiness.
Change-Id: I77c651ab6471ae094b5c21d1ab3860c96cb0d039
Second method needed to perform server-side move. It updates
metadata key and nonce and all segments key and nonces.
Change-Id: Ia43b26622a13048269f0ae9e1524b345db112adb
First from two methods needed to perform server-side move. It gets
metadata key and nonce and all segments key and nonces and returns
all of that to uplink.
Change-Id: Ied2c79559e77d3f63091c4d61948f2d6a2147d67
Currently, requests that were successfully passed through the metainfo
endpoints rate-limiter might still fail in the middle of the
corresponding response. The problem is that we perform rate-limiting a
second time, which means other requests would influence whether the
current (already rate-checked) request will fail. This also has other
unintended effects, like responding with rpcstatus.PermissionDenied for
requests that were successfully rate-checked and did not lack
permissions but were rate-checked again in the middle of
(*Endpoint).BeginObject. This situation has been happening on the
gateway side and might affect other uplink clients. This change, where
appropriate, swaps subsequent validateAuth with validateAuthN that
performs rate-limiting once.
Change-Id: I6fc26dedb8c442dd20acaab5942f751279020b08
At some point we moved metabase package outside Metainfo
but we didn't do that for satellite structure. This change
refactors only tests.
When uplink will be adjusted we can remove old entries in
Metainfo struct.
Change-Id: I2b66ed29f539b0ec0f490cad42c72840e0351bcb
Two small cleanups:
* merging private commitObject, commitSegment,
makeInlineSegment with its public versions. We were
using it when pb.Pointer was still used.
* removing unused CreatePath method
Change-Id: Ib18b07473d91259335dab874559ef52412ab813d
