Commit Graph

692 Commits

Author SHA1 Message Date
Qweder93
417df4d73c satellite/metabase: don't return expired objects
Resolves: https://github.com/storj/team-metainfo/issues/112

Change-Id: If48f865826c22764fc1ff83745c34524cd396aac
2022-07-04 21:32:30 +03:00
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Jeremy Wharton
58c5d44f44 satellite/console: integrate sessions into satellite UI
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.

Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
2022-06-13 08:02:02 +00:00
Moby von Briesen
5b79970575 satellite/metainfo: Use project-level attribution at bucket level
If project.UserAgent is set, use this for bucket.UserAgent on bucket
creation. Otherwise, set bucket attribution as before (getting UserAgent
from request headers).

Tests were updated to create the bucket with a different user, added as
a project member. Otherwise, the tests do not catch the bug.

Change-Id: I7ecf79a8eac5957eed361cbea94823190f58b776
2022-06-07 17:08:50 +00:00
Fadila Khadar
4725a3878c satellite/metainfo: parallel deletion tests
- parallel deletion of 50 objects and their 50 copies (one copy per object)
This test is skipped because it's creating deadlocks that are not automatically retried on postgres
- parallel deletion of 1 object and its 50 copies.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Id7a28251c06bb12b5edcc88721f60bf7a4bc0492
2022-05-26 11:00:42 +00:00
Michał Niewrzał
6759ba831c satellite/metainfo: skip TestUserAgentTransferStats
It's hard to make this test stable on CI but still can be
useful for local testing.

Change-Id: Iac5368d1a9179b67ea2409ee99bcf189db862897
2022-05-25 11:15:23 +00:00
Michał Niewrzał
456aea727e satellite: use PieceIDDeriver for derivation
We can use PieceIDDeriver in all places where we are deriving id from
the same id multiple times. We have serveral such places: gc, segment
deletion, segment validation, order limit creation. Using it should
save some resources.

Change-Id: I24668d516c0f7cea4aec6470614067734149501d
2022-05-19 06:31:42 +00:00
Michał Niewrzał
230424eace satellite/metainfo: add IPFS datastore user agent to metrics
Change-Id: I2ac14c97c64542669acef58ca6ded79984d0bd11
2022-05-17 13:09:22 +02:00
Michał Niewrzał
3ebb05ecca satellite/metainfo: add user-agent bandwidth metric
The existing versionCollector metrics can tell us how many times various
metainfo endpoints are called, but they don't tell us how many bytes a
client is transferring.  We currently can't collect precise information
on this, but we can collect information on how much planned traffic is
requested via order limits.

The implementation as provided is intended to measure objects sizes
before erasure encoding is taken into account.

Change-Id: I2f1d2a7831630e8439ecf5342e933df259151792
2022-05-17 09:57:39 +00:00
Ivan Fraixedes
648d6bf0a7 satellite/metabase: Classify pending object error
Create an error class for the "pending object error" for distinguishing
it from other errors for allowing to return it as a "Not Found" DRPC
status code instead an "Internal" status code.

"Internal" errors are logged in the satellite error so this was
polluting the server logs aside of returning an inappropriate status
code.

Change-Id: I10a81adfc887c030c08a228158adc8815834b23c
2022-05-10 10:54:17 +00:00
Michał Niewrzał
e4a4248337 satellite/metainfo: add aws cli user agent for tracking
Change-Id: Ibb00a836dd1dc22b5e20eeb1d1df593ca6ae7d8e
2022-05-09 21:09:33 +00:00
Qweder93
04b22f24eb satellite/metabase: copy expires_at to segments while FinishCopyObject
Resolves: https://github.com/storj/storj/issues/4726

Change-Id: I0bb222b7d5d3c66fe96c78a8aa1ea8f3b0a33f2b
2022-04-27 18:39:54 +00:00
nadimhq
dd6aac9b39
satellite/metainfo: enable server-side copy in production (#4708)
these changes enable server-side copy in production
2022-04-26 09:58:16 -04:00
Michał Niewrzał
db5d781735 satellite/metainfo: simplify version collector logging
Version collector previously returned errors and logged them in the
calling code.  It is cleaner to log inside version collector.

Change-Id: I52cb49a1ef53f3f1f51692ddb26ec095cfd0f100
2022-04-26 10:16:20 +00:00
Michał Niewrzał
e66beb2429 satellite/metainfo: set metadata with BeginObject
This change makes possible to set metadata at the beginning
of upload.

https://github.com/storj/team-metainfo/issues/105

Change-Id: If5b6b90243d9c9318a8b72c2af3fbde400c5d8e7
2022-04-26 08:04:27 +00:00
Michał Niewrzał
6e5a94698e satellite/metabase: add option to override metadata with CommitObject
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.

https://github.com/storj/team-metainfo/issues/105

Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
2022-04-26 08:03:52 +00:00
Qweder93
cb1bf52794 satellite/metainfo: add expireddeletion test for copied objects
We implemented server-side copy feature and we would like to
confirm that it is not affecting expired deletion service.

Resolves: https://github.com/storj/storj/issues/4698

Change-Id: Ia8ca27a7ab7764a48a0c85dc7be80a58bfc83729
2022-04-21 12:07:00 +00:00
Michał Niewrzał
6188c4c0dd satellite/metainfo: enable segment limit by default
Change-Id: I26cc0bbd97d6cc2ffc96213788f261efb749d028
2022-04-14 06:12:09 +00:00
Michał Niewrzał
99ec4c8869 satellite/gc: improve test for copies
Initial space used for pieces is calcualted, not retrieved
from storage nodes and at the end of test we are deleting
also copies that become ancestors to verify that all data
was removed from storage nodes.

Change-Id: I9804adb9fa488dc0094a67a6e258c144977e7f5d
2022-04-11 11:06:01 +00:00
Cameron
d4ad3a3ca6 satellite/satellitedb/attribution: update value attribution query to return byte-hours
Before, the VA query was summing the total and dividing by the number of
rows. This gives the average bytes stored per hour, but we charge for
usage with byte-hours. Why not do value attribution the same way?
To do that, we don't divide by the number of rows. We also have object
and segment fees so return segment-hours and object-hours too.

Change-Id: I1f18b7e1b2bae1d3fae1ca3b93bfc24db5b9b0e6
2022-04-08 16:22:21 -04:00
Michał Niewrzał
e93bbf693d satellite/metainfo: add missing method monitoring
Change-Id: I3a0d8121203a6f543f660de02628967cf7f22040
2022-04-03 15:47:14 +00:00
Michał Niewrzał
8cdadae124 satellite/metainfo: optimize geo config check for move/copy
This change has two purposes. First is to avoid DB call in case
source and destination bucket are the same.
Second is to return bucket not found error in correct order. If
source and destination bucket are different we will first check
source and later destination. Currently we will get first error
about not existing destination bucket.
Because of this change we stop putting bucket placement
into satellite stream id but its not needed as we don't use
this value with finish move/copy object methods.

Change-Id: I0f7b3ba604d53c722e8fa4d7a37843a69d02bebd
2022-04-01 11:49:44 +00:00
Michał Niewrzał
4da3dc8170 satellite/metabase: improve processing of key/nonce while finish copy/move
So far we assumes that metadata key/nonce cannot be empty at all
but at some point we adjusted code to accept empty metadata/key/nonce
to save DB space.

This change is adjusting how we are processing nonce while
FinishMoveObject/FinishCopyObject. We can use storj.Nonce directly
which makes code cleaner. It's also fixing issue in FinishMoveObject
where we didn't convert nonce correctly to []byte.

Part of change is disabling validation for key and nonce until
uplink will be adjusted. We need change uplink to send always
both key and nonce or non of them. Validation will be restored
as soon as change for uplink will be merged.

https://github.com/storj/storj/issues/4646

Change-Id: Ia1772bc430ae591f54c6a9ae0308a4968aa30bed
2022-03-28 14:34:21 +00:00
Erik van Velzen
b6db5c12b4 satellite/metainfo: register php user agent
Add uplink-php and nextcloud as user agents. These sending of these
user agents was added to recent releases of these clients.

Change-Id: Ia2732ade1d9e5cf8d4e41fe246faec3feaa58c25
2022-03-23 18:37:05 +00:00
Michał Niewrzał
ae29a5bb62 satellite/metainfo: use metabase package instead storj
Uplink have some types aliased from storj/common repo. It's like
that for easier type replacement if we decide to use custom type
instead of aliasing. Because in storj/storj we are not using aliases
it's impossible to do refactoring on uplink side. This change is
cleaning up this situation.

Change-Id: I20c8e31b9a821983483af1c67b2e7bb91397fd9d
2022-03-23 15:47:54 +00:00
Bill Thorp
c49c646d4e satellite/metainfo: allow multi-product user-agent metrics
Chronograph statistics indicate that much of our Gateway-MT traffic may
originate from and also is metriced as rclone traffic.  This makes it
difficult to understand what our users are doing.  This solution makes
it clear what products are actually being used, likely without
increasing the cardinality of our metrics by more than one.

Change-Id: I5d5e2af3715fa0864f69f1145fd78caf7e4a4224
2022-03-22 10:40:02 +00:00
Michał Niewrzał
35efb6462c satellite/metainfo: adjust deleting to server side copy
For server-side copy we adjusted one method DeleteObjectExactVersion.
Other deletion methods won't be used directly in code at the moment.
We will adjust other methods later or decide if we will need them at
all.

To handle deletion of objects with copies or just copies correctly we
need to use DeleteObjectExactVersion method in two places while:
* removing object before upload
* explicit object deletion

This change is also changing DeleteObjectExactVersion method to
delete pending objects because we need this functionality to
delete object before new upload.

https://github.com/storj/storj/issues/4481

Change-Id: Ieff5cc95732bb70ed8cc0ecdd62e03c929857c02
2022-03-17 13:57:01 +01:00
Michał Niewrzał
ccd16bbef6 satellite/metabase: handle NewEncryptedMetadata while coping object
Copy object functionality should support setting new metadata for
copy. This change is adjusting FinishCopyObject method to set new
metadata when OverrideMetadata field is set to true.

Fixes https://github.com/storj/storj/issues/4483

Change-Id: Ica37cb57e8edae301cdc483fbda4f3ddba5d2702
2022-03-09 12:31:33 +01:00
Michał Niewrzał
1da9697995 satellite/metainfo: use feature flag for begin/finish object copy metods
We want also to cover with feature flag code for begin/finish
object copy methods.

Change-Id: I97330e7d9b310de98408c47c2c1ed41551631bec
2022-02-28 15:09:40 +00:00
Michał Niewrzał
c0297bae78 satellite/metabase: return object metadata with FinishCopyObject
Updates metadata and metainfo to return object metadata with
FinishCopyObject request.

https://github.com/storj/storj/issues/4474

Change-Id: I32cba5c20a943272e9b5964df1b3d6463ad212dc
2022-02-25 12:46:30 +00:00
Fadila Khadar
2d4760fd09 satellite/metainfo: BeginCopyObject and FinishCopyObject
Metainfo endpoints to use server-side copy.

Fixes https://github.com/storj/storj/issues/4475

Change-Id: Ided06aed9e6187d6d8f030e95dda019ba78fff95
2022-02-24 14:38:04 +01:00
Michał Niewrzał
fbe2680500 satellite/metainfo: add feature flag for server side copy code
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.

Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
2022-02-24 10:43:49 +00:00
Michał Niewrzał
b24379aa91 satellite/metainfo: use single testplanet instance in tests
Tests refactoring to reuse testplanet instance between some
of tests. This should decrease resources needed to run those
tests.

Change-Id: I98f3041ec23085d3903b19acd339904973319ec1
2022-02-16 15:27:26 +00:00
Bill Thorp
cf03209c16 satellite/metainfo: only collect metric "other" once per user agent
A user-agent string can contain multiple "products", in the case of
Gateway-MT at least this includes the HTTP client's full user agent.
This means that "other" is often logged even when we know the Storj
product, and sometimes logged more than once per call to "collect".

This makes sure that "other" is only logged if a product isn't
identified, and only logged once.

Change-Id: I8536f7eb32877e36fec97dab7b8d477ccb10f92e
2022-02-09 18:18:55 +00:00
Stefan Benten
2d3c417c92 satellite/metainfo/version_collector: adding more known user agents:
transfer-sh will be set as of https://github.com/dutchcoders/transfer.sh/pull/467.
filezilla needs to be verified and duplicati is set per info from @TopperDEL
comet and orbiter are added as preparation.

Change-Id: I44d730a7b3ba1969068e48c2477b478831799cd1
2022-02-08 16:24:55 +00:00
Michał Niewrzał
2e31ef3f29 satellite/metabase: better error message while move
Before this change we were returning full DB error message.
That can be very confusing for end user. This change is translating
error message into more user frindly version and fixes also DRPC
error status code.

Fixes https://github.com/storj/team-metainfo/issues/76

Change-Id: I29b06ab4ba50a0d14db7a822a2906d95d65ab524
2022-02-02 15:03:24 +00:00
Michał Niewrzał
bc161794fc satellite/metabase: drop DeleteObjectLatestVersion method
This method was never used, except tests.

Change-Id: Idc1e69b2e2971995b5c4e6cf78a2b5fc69f39ad2
2022-02-02 14:33:48 +00:00
Michał Niewrzał
5041ee0abc satellite/metabase: drop GetObjectLatestVersion method
We don't have different version of object than 1 so at the moment
this method is not needed. Also using GetObjectExactVersion
should be slightly more performant.

Change-Id: I78235d8ae22594cc1d6345dabcc915f41cd7797b
2022-02-02 09:40:53 +00:00
Michał Niewrzał
dffa7845f6 satellite/metainfo: split endpoint tests
We already split main code base, now we need to split test
to reflect new files structure (bucket/object/segment/other).

Fixes https://github.com/storj/team-metainfo/issues/12

Change-Id: Ica1054c4fc7df764483b03f204b4beba094df8e1
2022-02-02 07:46:14 +00:00
Michał Niewrzał
17f56ab636 satellite/metainfo: split endpoint into smaller files (object/segment)
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Updates https://github.com/storj/team-metainfo/issues/12

Change-Id: I6c691e4d0e192fe3ad7974d2d0ab5ced0d272f3c
2022-01-25 16:14:37 +00:00
Michał Niewrzał
65c38c6cef satellite/metainfo: split metainfo endpoint into smaller files (buckets)
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Change-Id: I9b097dcc8fa889f985b7f4ef5f8f435a1ff0ef95
2022-01-24 13:47:51 +00:00
Michał Niewrzał
bfad42f29a satellite/metainfo: split metainfo endpoint into smaller files
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Change-Id: I5128c84e06c82777fe71460bf5f9a6e26e52a243
2022-01-24 08:24:24 +00:00
Stefan Benten
e7e39fe628
satellite/metainfo: reduce default user rate limit to 100
Currently the rate limit has kept per satellite api endpoint.
Since we run 9+ api endpoints in production, we do not need
a limit of 1000, since the intention was to allow 1000 total.
This change reduces the effective limit given 9 instances
down to 900, which should be close enough.

Change-Id: Ia579149ccc3a12e8febe0cfd5586b8a39de40f55
2022-01-21 23:37:59 +01:00
Michał Niewrzał
1c47163eee satellite/metainfo: add more logging for CreateBucket
We were returning pure non rpc errors in two cases.
This change added loging and correct rpc error as a
return.

Change-Id: I581ceb17dcdc00921dfa3c1057015c3b4d04308d
2022-01-21 15:51:37 +00:00
Qweder93
228f465d45 satellite/metainfo: сombine checks for storage and segment limit
We need to combine methods from accounting.Service (ExceedsStorageUsage and ExceedsSegmentUsage)
to run checks concurrently.

Resolves https://github.com/storj/team-metainfo/issues/73

Change-Id: I47831bca92457f16cfda789da89dbd460738ac97
2022-01-19 10:44:04 +00:00
Michał Niewrzał
36c07720bb satellite/metainfo: bound user agent metrics
We need to limit number of results for SeriesTag to avoid
overloading our metric system.

Change-Id: I3d9f4f4f6d489bf4ce87c6240f66d48b29b88abb
2022-01-18 12:42:09 +00:00
Michał Niewrzał
5e9643e1b8 satellite/metainfo: cleanup validation
Refactoring to do few things:
* move simple validation before validations with DB calls
* combine validation check/update for storage and segment
limits together

Change-Id: I6c2431ba236d4e388791d2e2d01ca7e0dd4439fc
2022-01-13 13:00:40 +01:00
Michał Niewrzał
18ab464b51 satellite/metainfo: UpdateProjectSegmentUsage enabled only with segment validation
updating project segment usage entry to redis is enabled only if
segment validation is enabled in config value, so no incompatible
changes will be added while deployment.

Change-Id: I1288cb9ff0a8a00f095dc94e20d2f14393e9a613
2022-01-13 12:12:25 +01:00
Michał Niewrzał
eb0d08d59b satellite/metainfo: collect versions of user tools
We want to know usage statistics for our main tools
like uplink-cli or rclone. Initially we will collect
only usage stats without relation to specific process
e.g. download or upload.

Change-Id: I203b1a6c07ae014e710368f77163f13fdf10763c
2022-01-12 13:49:15 +00:00
Qweder93
b6625cadea satellite/metainfo: uncomment tests, renamed EncryptedPath into EncryptedObjectKey
moved tests back and renamed params that were renamed on uplink side

Change-Id: I4a9592bd955f2217c5809c235c394a4e78d337ea
2022-01-11 17:18:18 +02:00