Commit Graph

592 Commits

Author SHA1 Message Date
Michał Niewrzał
5c91ecd271 satellite/metainfo: endpoint cleanup
Two small cleanups:
* merging private commitObject, commitSegment,
makeInlineSegment with its public versions. We were
using it when pb.Pointer was still used.
* removing unused CreatePath method

Change-Id: Ib18b07473d91259335dab874559ef52412ab813d
2021-08-26 09:52:38 +00:00
Artur M. Wolff
2cd68bf4fb private/lrucache: import from common
Change-Id: Ia1f43d0440fef21122b071b05da59b4cf2689d6c
2021-08-16 10:04:32 +00:00
Bill Thorp
549e799bbe satellite: wrap notfound on failed deletes as DRPC errors
We're seeing BeginDeleteObject in metaclient returning object not found:
metabase: no rows deleted in the Gateway-MT mint tests.  There's a
client check for rpcStatus.NotFound, but the metabase endpoint isn't
wrapping the db error as a DRPC error.

Here's the chain:

  gateway.AbortMultipartUpload()
  project.AbortUpload()
  metainfoClient.BeginDeleteObject() <- understands DRPC errors
  endpoint.DeletePendingObject() <- where this code is
  db.DeletePendingObject() <- returns error

Change-Id: I93991de76487426df0a807b0d1e69fc975196a1a
2021-08-16 09:02:35 +00:00
Artur M. Wolff
2f7e670996 satellite/metainfo: allow per-project rate limit of zero
Change-Id: I237c67dfa3d24ed4837175556f72b9c761644435
2021-08-11 11:45:58 +00:00
Clement Sam
6613133a1c satellite/metabase: return metadata from IterateObjectsAllVersionsWithStatus on demand
Change-Id: I6bb717cf941e2b5fd7c411bd1e5a77fad0835995
2021-08-06 16:20:39 +00:00
Michał Niewrzał
f87ad240ed satellite/metainfo: add DeletePart method
We need a way to delete whole part. This especially
needed for uplink multipart API to do cleanup after
aborted or failed part upload.

Test will be added when uplink part will be merged.

Change-Id: I9ba69a49e1adcdce0f42dd3a76f938fcf931155a
2021-08-05 14:43:30 +00:00
Qweder93
5bb3836312 satellite/metainfo: objectEntryToProtoListItem fills metadata on demand
Added includeMetadata parameter which represents if metadata should be included in response
by default true, in case of new uplink version - ObjectIncludes will be used instead.

Change-Id: I2f8d3b4cc354cd655f8093bbbebe0e3c2ae14e6f
2021-08-03 13:41:11 +00:00
Michał Niewrzał
a2b30e42a3 satellite/metainfo: log internal errors
We are missing some logging around errors returned as
Internal.

Change-Id: Ifc41634816648172ff944808d23d1e6a344e41f1
2021-07-27 09:57:59 +00:00
Michał Niewrzał
b12d29935a satellite/metabase: remove metaloop package
We moved everything to segment loop so we can now
remove metaloop from code.

Change-Id: I9bd8d2349e5638d7cdad50f2f313f9bd89a8165c
2021-07-22 13:00:45 +00:00
Michał Niewrzał
27a714e8b0 satellite/accounting/tally: use objects iterator instead metaloop
Bucket tally calculation will be removed from metaloop and will
use metabase objects iterator directly.

At the moment only bucket tally needs objects so it make no sense
to implement separate objects loop.

Change-Id: Iee60059fc8b9a1bf64d01cafe9659b69b0e27eb1
2021-07-20 15:52:18 +00:00
Egon Elbre
5a56021bca satellite/metabase: add intLimitRange
We have quite a bit code duplication in ensuring valid int ranges.

Change-Id: Ib31db8c4b2be6a677625cab5bcdddefb60401fe3
2021-07-19 07:51:16 +00:00
Kaloyan Raev
a767aed591
satellite/{metabase,metainfo}: require StreamID for UpdateObjectMetadata
This avoids corrupting objects if reuploads and metadata updates happen
concurrently.
2021-07-08 17:50:37 +03:00
Kaloyan Raev
73b9223758 satellite/metainfo: implement UpdateObjectMetadata
Change-Id: I7d791bb84b73299d395b587074e721c14e4be31c
2021-07-07 12:57:32 +00:00
Michał Niewrzał
4e645059be satellite: rename Endpoint2 to Endpoint
Change-Id: I91faaa532d5a24f033b9865a89b91bc6821588be
2021-06-22 09:25:05 +00:00
Michał Niewrzał
bc79f01aaa satellite/metabase: set expires_at while committing segment
We added expires_at column to segments table and now
we need to populate this column while committing segment.
We still need to migrate existing segments with
separate tool.

Change-Id: Ibac8c63d97201dd98cc2cb9db385f4cb73bc3f7e
2021-06-10 16:32:28 +00:00
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
Michał Niewrzał
053e58b683 satellite/metabase: add segmentloop service
We want to move some of current metainfo loop observers to
segment loop. This change adds new service, similar to metainfo
loop but which is iterating only over segments.

Change-Id: I67f7f461781723a4476e2b83377f31736d7c4870
2021-06-01 11:15:07 +00:00
Egon Elbre
10a0216af5 satellite/metainfo: use range for specifying download limit
Previously the object range was not used for calculating order limit.
This meant that even if you were downloading only a small range it would
account bandwidth based on the full segment.

This doesn't fully address the accounting since the lazy segment
downloads do not send their requested range nor requested limit.

Change-Id: Ic811e570c889be87bac4293547d6537a255078da
2021-06-01 09:36:55 +00:00
Egon Elbre
910eec8eee satellite/metainfo: remove MetabaseDB interface
Currently the interface is not useful. When we need to vary the
implementation for testing purposes we can introduce a local interface
for the service/chore that needs it, rather than using the large api.

Unfortunately, this requires adding a cleanup callback for tests, there
might be a better solution to this problem.

Change-Id: I079fe4dbe297b0ae08c10081a1cea4dfbc277682
2021-05-13 13:22:14 +00:00
Egon Elbre
2af7e4ef26 satellite/metabase/metaloop: use database time
The system and database time may drift. We should use database time for
absolute "as of system time" to ensure that it's not newer than the
current database time. When the "as of system time" is in the future,
then the query will fail.

Change-Id: I5423f6aaad966ca03a76b5ff805bfba932e44a51
2021-05-07 09:08:04 +00:00
Egon Elbre
5269596c7d satellite/{metabase,metainfo}: use ObjectStream as argument
Change-Id: I5a7f096002b4d7a6162b24d7a64346b058c4c89c
2021-05-04 15:30:59 +00:00
Egon Elbre
a8533042a3 mod: bump uplink
Renamed a private package in uplink and some of the tests need to be
updated.

Change-Id: I8735fc16c2066ecba669618f0c1dadf68273704d
2021-04-30 16:53:54 +03:00
Egon Elbre
2c657f594e satellite/metainfo: don't rely on exact error name
Change-Id: I7975a00c32891a43b4f1e6dc4f5847201844f2ec
2021-04-29 18:35:56 +00:00
Egon Elbre
961e841bd7 all: fix error naming
errs.Class should not contain "error" in the name, since that causes a
lot of stutter in the error logs. As an example a log line could end up
looking like:

    ERROR node stats service error: satellitedbs error: node stats database error: no rows

Whereas something like:

    ERROR nodestats service: satellitedbs: nodestatsdb: no rows

Would contain all the necessary information without the stutter.

Change-Id: I7b7cb7e592ebab4bcfadc1eef11122584d2b20e0
2021-04-29 15:38:21 +03:00
Michał Niewrzał
02460fcc4c satellite/metainfo/expireddeletion: change chore interval to 24h
We want run deletion query more often to avoid long running DB operations.

Change-Id: I72c63bb8ec05cc3ab2e7ab63479ba6cdf830386a
2021-04-27 12:30:23 +00:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Egon Elbre
c641ddcb54 pkg/lrucache: rename package
cache is really common variable and type name and we have already used
the package name alias in multiple places.

Change-Id: I6435785b7549b541d533de59ec94557b9bd11e04
2021-04-23 16:37:28 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
4c9ed64f75 satellite/metabase/metaloop: move loop under metabase
Currently the loop handling is heavily related to the metabase rather
than metainfo.

metainfo over time has become related to the "public API" for accessing
the metabase data.

Currently updates monkit.lock, because monkit monitoring does not handle
ScopeNamed correctly. Needs a followup change to monitoring check.

Change-Id: Ie50519991d718dfb872ec9a0176a82e732c97584
2021-04-22 12:58:09 +03:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00
Kaloyan Raev
2ee3030275 all: remove code related to PointerDB
Change-Id: I6675c9597f87019020f6233b83ab2f1119d2bc46
2021-04-21 12:35:31 +00:00
Egon Elbre
6a805b2891 satellite/metainfo/metabase: use pgx.Batch to delete expired segments
Change-Id: I51eeaadeeef0ca2faabac4b38158278aea177eb8
2021-04-19 11:19:29 +03:00
Egon Elbre
b2be1f1629 satellite/metainfo/metabase: fix delete expiration
The cursor was not being used in the batch deletion.

The stream ID was not being used while deleting, which could in rare
circumstaces delete a newly uploaded object.

Use the stream id in deletion, rather than passing that information from
one query to another.

Change-Id: I03271c6e72747e345dfb0bb70989f29e835efd8e
2021-04-19 11:19:29 +03:00
Fadila Khadar
bde367ae73 satellite/gc: check on bloom filter creation date
Check that the bloom filter creation date is earlier than the
metainfo loop system time used for db scanning.

Change-Id: Ib0f47c124f5651deae0fd7e7996abcdcaac98fb4
2021-04-14 16:40:37 +00:00
Michał Niewrzał
c3f8e06798 satellite/metainfo/metainfo: simplify expired objects deletion query
Change-Id: Iba5fc498527338ed6d2c5dd779c508ec8b6ce443
2021-04-13 19:34:32 +00:00
Michał Niewrzał
027554a15c satellite/metainfo: add tests for commit segment validation
Change-Id: I012bf2f7481003591cc873fe738a7cd8f525db29
2021-04-12 16:33:17 -06:00
Egon Elbre
3236f91ae0 satellite/metainfo: reenable piece hash verification
Change-Id: I9312d8fde0c1c392cbad82b947222b4b6e0dbbdd
2021-04-12 15:12:42 -06:00
Michał Niewrzał
6e79923996 satellite/metainfo: bring back validation while committing segment
During metainfo refactor we disabled some validation as it was designed to validate pointer. Now part of this validation is restored. This is first part.

Change-Id: I6132f922fe23d60118bbccfdb77fd93c3c81afed
2021-04-12 15:12:42 -06:00
Egon Elbre
ec7af97a17 satellite/metainfo: document migrated objects
Document the fields that migrated objects have missing, it's easy to
forget that they might not exist.

Avoid downloading the segment, if we're not sure whether it's the
correct one. We'll later improve the code with an heuristic to get a
best guess, which segment to download.

Change-Id: I12395c17bbf0edf25e0d00c8d072fce6085e303b
2021-04-09 15:28:27 +00:00
Michał Niewrzał
6b88a675c5 metabase-createdat-migration: add migration tool
We recently added create_at column to segments table.
Old segments needs to get this value from objects table.
This tool will iterate over all objects and update corresponding
segments if create_at column is not set.

Change-Id: Ib5aedc384637e739ee9af84454af0639e2559416
2021-04-08 14:20:09 +00:00
Egon Elbre
ffc8aca8b0 satellite/metainfo: update bandwidth usage in DownloadObject
Change-Id: I3c77fd46c57903435bd9618bb6cf33647217adbf
2021-04-07 14:17:59 +03:00
Egon Elbre
98b7e1a991 satellite/metainfo: fix returning []{nil} in DownloadObject
When object doesn't contain segments the implementation would have
returned []*pb.SegmentDownloadResponse{nil} instead of nil.

Change-Id: If38f6d3d9d119f514f63ad1a8762055f657f3004
2021-04-07 09:55:01 +03:00
Egon Elbre
47be501188 satellite/metainfo: add DownloadObject endpoint
Add endpoint for getting object information, list segments in a range
and download the first segment in the range.

Change-Id: I056d697ae87c9aa34e7deccba8713902db260457
2021-04-06 19:47:36 +03:00
Egon Elbre
17817d84e9 satellite/metainfo: add PlainOffset to SegmentDownloadResponse
Change-Id: Ib68f2fd0ad86556f443a01a514243247dc7989a3
2021-04-06 15:21:00 +03:00
Egon Elbre
d2101bd017 satellite/metainfo: add Position to SegmentDownloadResponse
Change-Id: I538c4cf2a9d81788f0df2581defe166871d72cd7
2021-04-05 19:27:34 +03:00
Jeff Wendling
2462ac5917 metainfo+satellite/satellitedb: use HasBucket when possible
Change-Id: I8318f5581eef559f0dfbf20d07ea1fe36f81ce87
2021-04-02 13:20:36 -04:00
Michał Niewrzał
a5224e7a6c satellite/metainfo/metaloop: use segment CreatedAt and RepairedAt
Repair checker expects to have information about CreatedAt and RepairedAt fields to calculate segment age metric.

Change-Id: I6b41df880d77133be541e14d10d91cc75759b339
2021-04-02 08:46:54 +00:00
Michał Niewrzał
71d0adf90b satellite/metainfo: avoid error on inserting existing attribution
Currently we can have an error about duplicated entry while inserting into value_attributions table. This change is changing simple insert into insert that is doing nothing on conflict.

Change-Id: I3efd8dc0b63115e8e2ed8f4196ccf969ee942295
2021-04-02 08:18:17 +00:00
Egon Elbre
5c038c4325 satellite/metainfo/metabase: add stream range for listing segments
Change-Id: I32833e805a1046b9752b04888f830b51809a1efd
2021-04-01 15:27:10 +00:00