Commit Graph

720 Commits

Author SHA1 Message Date
Egon Elbre
2c657f594e satellite/metainfo: don't rely on exact error name
Change-Id: I7975a00c32891a43b4f1e6dc4f5847201844f2ec
2021-04-29 18:35:56 +00:00
Egon Elbre
961e841bd7 all: fix error naming
errs.Class should not contain "error" in the name, since that causes a
lot of stutter in the error logs. As an example a log line could end up
looking like:

    ERROR node stats service error: satellitedbs error: node stats database error: no rows

Whereas something like:

    ERROR nodestats service: satellitedbs: nodestatsdb: no rows

Would contain all the necessary information without the stutter.

Change-Id: I7b7cb7e592ebab4bcfadc1eef11122584d2b20e0
2021-04-29 15:38:21 +03:00
Michał Niewrzał
02460fcc4c satellite/metainfo/expireddeletion: change chore interval to 24h
We want run deletion query more often to avoid long running DB operations.

Change-Id: I72c63bb8ec05cc3ab2e7ab63479ba6cdf830386a
2021-04-27 12:30:23 +00:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Egon Elbre
c641ddcb54 pkg/lrucache: rename package
cache is really common variable and type name and we have already used
the package name alias in multiple places.

Change-Id: I6435785b7549b541d533de59ec94557b9bd11e04
2021-04-23 16:37:28 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
4c9ed64f75 satellite/metabase/metaloop: move loop under metabase
Currently the loop handling is heavily related to the metabase rather
than metainfo.

metainfo over time has become related to the "public API" for accessing
the metabase data.

Currently updates monkit.lock, because monkit monitoring does not handle
ScopeNamed correctly. Needs a followup change to monitoring check.

Change-Id: Ie50519991d718dfb872ec9a0176a82e732c97584
2021-04-22 12:58:09 +03:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00
Kaloyan Raev
2ee3030275 all: remove code related to PointerDB
Change-Id: I6675c9597f87019020f6233b83ab2f1119d2bc46
2021-04-21 12:35:31 +00:00
Egon Elbre
6a805b2891 satellite/metainfo/metabase: use pgx.Batch to delete expired segments
Change-Id: I51eeaadeeef0ca2faabac4b38158278aea177eb8
2021-04-19 11:19:29 +03:00
Egon Elbre
b2be1f1629 satellite/metainfo/metabase: fix delete expiration
The cursor was not being used in the batch deletion.

The stream ID was not being used while deleting, which could in rare
circumstaces delete a newly uploaded object.

Use the stream id in deletion, rather than passing that information from
one query to another.

Change-Id: I03271c6e72747e345dfb0bb70989f29e835efd8e
2021-04-19 11:19:29 +03:00
Fadila Khadar
bde367ae73 satellite/gc: check on bloom filter creation date
Check that the bloom filter creation date is earlier than the
metainfo loop system time used for db scanning.

Change-Id: Ib0f47c124f5651deae0fd7e7996abcdcaac98fb4
2021-04-14 16:40:37 +00:00
Michał Niewrzał
c3f8e06798 satellite/metainfo/metainfo: simplify expired objects deletion query
Change-Id: Iba5fc498527338ed6d2c5dd779c508ec8b6ce443
2021-04-13 19:34:32 +00:00
Michał Niewrzał
027554a15c satellite/metainfo: add tests for commit segment validation
Change-Id: I012bf2f7481003591cc873fe738a7cd8f525db29
2021-04-12 16:33:17 -06:00
Egon Elbre
3236f91ae0 satellite/metainfo: reenable piece hash verification
Change-Id: I9312d8fde0c1c392cbad82b947222b4b6e0dbbdd
2021-04-12 15:12:42 -06:00
Michał Niewrzał
6e79923996 satellite/metainfo: bring back validation while committing segment
During metainfo refactor we disabled some validation as it was designed to validate pointer. Now part of this validation is restored. This is first part.

Change-Id: I6132f922fe23d60118bbccfdb77fd93c3c81afed
2021-04-12 15:12:42 -06:00
Egon Elbre
ec7af97a17 satellite/metainfo: document migrated objects
Document the fields that migrated objects have missing, it's easy to
forget that they might not exist.

Avoid downloading the segment, if we're not sure whether it's the
correct one. We'll later improve the code with an heuristic to get a
best guess, which segment to download.

Change-Id: I12395c17bbf0edf25e0d00c8d072fce6085e303b
2021-04-09 15:28:27 +00:00
Michał Niewrzał
6b88a675c5 metabase-createdat-migration: add migration tool
We recently added create_at column to segments table.
Old segments needs to get this value from objects table.
This tool will iterate over all objects and update corresponding
segments if create_at column is not set.

Change-Id: Ib5aedc384637e739ee9af84454af0639e2559416
2021-04-08 14:20:09 +00:00
Egon Elbre
ffc8aca8b0 satellite/metainfo: update bandwidth usage in DownloadObject
Change-Id: I3c77fd46c57903435bd9618bb6cf33647217adbf
2021-04-07 14:17:59 +03:00
Egon Elbre
98b7e1a991 satellite/metainfo: fix returning []{nil} in DownloadObject
When object doesn't contain segments the implementation would have
returned []*pb.SegmentDownloadResponse{nil} instead of nil.

Change-Id: If38f6d3d9d119f514f63ad1a8762055f657f3004
2021-04-07 09:55:01 +03:00
Egon Elbre
47be501188 satellite/metainfo: add DownloadObject endpoint
Add endpoint for getting object information, list segments in a range
and download the first segment in the range.

Change-Id: I056d697ae87c9aa34e7deccba8713902db260457
2021-04-06 19:47:36 +03:00
Egon Elbre
17817d84e9 satellite/metainfo: add PlainOffset to SegmentDownloadResponse
Change-Id: Ib68f2fd0ad86556f443a01a514243247dc7989a3
2021-04-06 15:21:00 +03:00
Egon Elbre
d2101bd017 satellite/metainfo: add Position to SegmentDownloadResponse
Change-Id: I538c4cf2a9d81788f0df2581defe166871d72cd7
2021-04-05 19:27:34 +03:00
Jeff Wendling
2462ac5917 metainfo+satellite/satellitedb: use HasBucket when possible
Change-Id: I8318f5581eef559f0dfbf20d07ea1fe36f81ce87
2021-04-02 13:20:36 -04:00
Michał Niewrzał
a5224e7a6c satellite/metainfo/metaloop: use segment CreatedAt and RepairedAt
Repair checker expects to have information about CreatedAt and RepairedAt fields to calculate segment age metric.

Change-Id: I6b41df880d77133be541e14d10d91cc75759b339
2021-04-02 08:46:54 +00:00
Michał Niewrzał
71d0adf90b satellite/metainfo: avoid error on inserting existing attribution
Currently we can have an error about duplicated entry while inserting into value_attributions table. This change is changing simple insert into insert that is doing nothing on conflict.

Change-Id: I3efd8dc0b63115e8e2ed8f4196ccf969ee942295
2021-04-02 08:18:17 +00:00
Egon Elbre
5c038c4325 satellite/metainfo/metabase: add stream range for listing segments
Change-Id: I32833e805a1046b9752b04888f830b51809a1efd
2021-04-01 15:27:10 +00:00
JT Olio
86c41790ce satellite/metainfo/metaloop: add observability
we want to know a lot more about what's going on during
the operation of the metainfo loop. this patchset adds
more instrumentation to previously unmonitored but
interesting functions, and adds metrics that keep track
of how far through a specific loop we are. it also
adds mon:lock annotations, especially to the metainfo
loop run task, which recently changed, silently broke
some queries, and thus failed to alert us to spiking
run time issues.

Change-Id: I4358e2f2293d8ebe30eef497ba4e423ece929041
2021-03-30 14:32:05 -06:00
Fadila Khadar
8d4df93d45 satellite/metainfo: remove TODO
remove TODO that we won't do.

Change-Id: I6ab3ddf2d9285006bed0fc5700da8c6738aa7792
2021-03-30 13:55:54 +00:00
Egon Elbre
794dc6d6d5 satellite/metainfo/metabase: add PlainOffset to segment listing
Change-Id: I8bbb5fcebf9f171033938eaae9f415b3a4b5aaa0
2021-03-30 16:20:21 +03:00
Egon Elbre
c4cb79ff1f satellite/metainfo/metabase: update segment plain offset, when needed
CommitObject was not updating plain_offset on segments.

Change-Id: I7e56c873bb15b893bf298b73e0349eb18d855a37
2021-03-30 16:20:21 +03:00
Kaloyan Raev
035c393da0 satellite: update tests to pass etag.Reader to multipart.PutObjectPart
Change-Id: Ibe99357945ae7a91f5b5d4f87b83d425c9fa84a5
2021-03-29 13:18:11 +00:00
Michał Niewrzał
141444f6d6 satellite/repair/repairer: fix segmentAge metric
Change-Id: I146b3163aa1bfab5ee060298e6bf9822ca6820a0
2021-03-29 12:29:47 +00:00
Egon Elbre
86e698f572 pb: use *UnimplementedServer to avoid breaking API changes
Change-Id: I99a34eeb37ac4453411f273511710562a519f57a
2021-03-29 12:26:10 +03:00
Michał Niewrzał
3e91d22ce6 satellite/metainfo/metabase: update repair_at with UpdateSegmentPieces
We need to update repair_at column while repairing segments.

Change-Id: I9770fe2e11a7e4b81a39470c50bb1d2aa6cfae29
2021-03-26 11:47:25 +00:00
Fadila Khadar
1c696168c5 satellite/metainfo/metabase: delete objects and segments in one query
We were using two queries to delete one or more objects and its/their
segments in DeleteObjectExactVersion, DeletePendingObject, DeleteObjectLatestVersion,
DeleteObjectAnyStatusAllVersions,DeleteObjectsAllVersions.

This change delete objects and their segments in one query.

Change-Id: Ib2c0eb501f00b091ee32519e02155350c4dcb8b0
2021-03-25 11:13:18 +00:00
Kaloyan Raev
8e1aa4bb74 satellite/metainfo: etag support for committing and listing segments
Change-Id: I9a4cdf0cffa887efc856b191ee0939c11e8391e7
2021-03-25 10:08:21 +02:00
Kaloyan Raev
ec929ca1da satellite/metainfo/metabase: add support for encrypted ETag
Change-Id: I142d5f759a452b12bf2ce67e6c939846a5e86ce7
2021-03-25 10:01:16 +02:00
Kaloyan Raev
cc42dfcc78 satellite/metainfo: add EncryptionParameters to SatStreamID
We have a use case for this in ListSegments. ListSegments is going to
return the EncryptedETag along with EncryptedKey and EncryptedKeyNonce.
It also must return the EncryptionParameters.

Since the EncryptionParameters are in the objects DB table, it would be
more efficient for ListSegment to avoid querying that DB table, but take
it from the SatStreamID.

Change-Id: I16c98641c0fe0c98e3303329d0da6ef137ca55cf
2021-03-25 07:54:34 +00:00
Moby von Briesen
27bcb46718 satellite/console: change default project and usage limits
This is one step for implementing the free tier:
* Change the default project limit from 10 to 3
* Move storage and bandwidth project usage limits from the metainfo
package to the console package (otherwise there is a cyclical
dependency, and metainfo doesn't use these values anyway)
* Change the default storage usage limit per project from 500gb to 50gb
* Change the default bandwidth usage limit per project from 500gb to 50gb
* Migrate the database so that old users and projects continue to have
the old defaults (10 projects/500gb usage)

Change-Id: Ice9ee6a738bc6410da18c336c672d3fcd0cab1b9
2021-03-24 14:24:27 +00:00
Egon Elbre
f19ef4afe5 satellite/metainfo/metaloop: move loop to a separate package
Change-Id: I94c931a27c1af6062185ec62688624ec02050f11
2021-03-23 15:37:34 +00:00
Michał Niewrzał
9d52451464 satellite/metainfo/expireddeletion: add config value to set list limit
We want to control how many expired objects we will query at once.

Change-Id: Iba7c79a6236c6b720b53bb8f078d55514690d8e5
2021-03-23 09:51:56 +00:00
Fadila Khadar
a8b5f8f990 satellite/metainfo/metabase: select then delete expired objects
We were deleting expired objects by directly executing a delete query.
With this change, we first select the objects to be deleted and then
delete them (as recommended by cockroachdb for deleting using a non indexed
column).

Change-Id: Ied150fbdc7031a343a74e0b9dab316598188ef66
2021-03-22 17:39:13 +01:00
Michał Niewrzał
27ae0d1f15 satellite/metainfo/metabase: add NewRedundancy parameter for UpdateSegmentPieces method
At some point we might try to change original segment RS values and set Pieces according to the new values. This change adds add NewRedundancy parameter for UpdateSegmentPieces method to give ability to do that. As a part of change NewPieces are validated against NewRedundancy.

Change-Id: I8ea531c9060b5cd283d3bf4f6e4c320099dd5576
2021-03-22 08:12:56 +00:00
Fadila Khadar
e613c641d7 satellite/metainfo/metabase: benchmark expired deletion
Change-Id: Icc2e8227b8a96002563ff69f3e9cd8e71369f0d1
2021-03-19 11:30:00 +00:00
Michał Niewrzał
1c02158620 satellite/metainfo/metabase: add RepairedAt to RawSegment
Change-Id: I3f8bbed0f815f8394fc6ab16a914cae276fbab56
2021-03-19 11:06:59 +00:00
Kaloyan Raev
be8a70309d satellite/metainfo/metabase: rename etag column to encrypted_etag
Change-Id: I79662d9a2cec2da6ea9bba250f0e71978119558d
2021-03-18 18:34:29 +02:00
Kaloyan Raev
36bc17ef5b satellite/metainfo/metabase: add etag column to segments table
We will use it keep track of ETag for object parts.

Change-Id: Ib610397afd44d71a1b5fe4301940de6f36ba39ae
2021-03-18 10:40:55 +00:00
Michał Niewrzał
fa083a7f05 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ib5ce5965b77b81c254d08c27ab30c7eccefbd4c6
2021-03-17 15:37:17 +01:00
Isaac Hess
1de6315b83 metainfo: Stub DownloadObject metainfo endpoint
This is the stub "not implemented" endpoint for the new DownloadObject
RPC.

Change-Id: Ifa724e20d89d839f2b6968d6283bc66a3bab8781
2021-03-15 10:52:57 -06:00
Michał Niewrzał
65957c7525 satellite/metainfo/expireddeletion: temporarily log errors instead failing whole system
At the moment we are trying to optimize deletion queries but its hard to verify deletion performance. Until we are sure that the queries are good we will just log errors instead shutting down whole satellite core.

Change-Id: I5625251d4518c35f0d46d6bf37b2f3ea7950675e
2021-03-15 16:00:20 +01:00
Kaloyan Raev
c4f0289904 satellite/metainfo: ListSegments returns CreatedAt in response
If a non-nil value is read from created_at column of the segments table,
it will be set to the CreatedAt field if SegmentListItem.

Change-Id: I02691d8e11fad12c1b0e4c443bdebb568016ffe3
2021-03-15 11:45:25 +00:00
Kaloyan Raev
6e661da0a0 satellite/metainfo/metabase: read created_at from DB
All SQL queries for reading segments are updated to read the created_at
column where appropriate.

Change-Id: Icd7c7672fa71e992673078598b28229bb898c728
2021-03-15 11:37:14 +00:00
Michał Niewrzał
887f3b04e1 satellite/metainfo/metabase: pieces vs redundancy scheme validation while commiting segment
Additional validation for pieces based on RS.

Change-Id: Ib25bfc0e38ca78e2c3194a3223d59257489049fb
2021-03-11 16:12:52 +00:00
Fadila Khadar
018b0684b8 satellite/metabase: different queries for delete_expired
use custom queries for pg and crdb when deleting expired objects.

Change-Id: I2e230b3b34d5128c877ed8f8f77d83b9a935df42
2021-03-11 09:54:03 +00:00
Kaloyan Raev
bdb33b381b satellite/metainfo/metabase: add created_at and repaired_at to segments
The created_at columns is first added without a default value to avoid
setting the current time to existing segments.

Change-Id: Ic2fe3da238422e2949e6f3016fbac04eb89ba037
2021-03-10 19:59:47 +00:00
Egon Elbre
5e954ad487 satellite/metainfo/metabase: optimize ConvertAliasesToPieces
old time/op    new time/op    delta
    7.56µs ± 5%    4.93µs ± 2%  -34.75%  (p=0.000 n=5+15)

  old alloc/op   new alloc/op   delta
    6.86kB ± 0%    3.85kB ± 0%  -43.87%  (p=0.000 n=5+18)

  old allocs/op  new allocs/op  delta
      19.0 ± 0%      17.0 ± 0%  -10.53%  (p=0.000 n=5+18)

Change-Id: Iedf24087766b3bd90934f2daa7ac186c3503a341
2021-03-10 08:39:23 +00:00
Egon Elbre
461d2cd233 satellite/metainfo{,/metabase}: add GetStreamPieceCountByNodeID
GetObjectIPs endpoint was loading all the segment information rather
than only what it needed.

Change-Id: I3f684cb7020b1771d78b5d43c6862d03ada464c0
2021-03-08 16:43:17 +02:00
Egon Elbre
981f1ca449 satellite/metainfo: use ListStreamPositions
ListSegments loads all the segment data into memory, however this can
add up to a lot of data with inline segments and large objects.

Change-Id: I037738f0e70b810ecbea7d83b00ea7ca9eb90c7a
2021-03-08 14:10:33 +02:00
Egon Elbre
5bc0f3bf23 satellite/metainfo/metabase: add ListStreamPositions
Change-Id: I0b5a5803be8987e7cafbcc80460217d9eb8f0238
2021-03-08 14:09:22 +02:00
Michał Niewrzał
903cb172c2 satellite/metainfo/metabase: add iteration with prefix benchmark
Change-Id: I7af77cdf0f11e28ad946f2eaff4a9aa8fe5d0bec
2021-03-08 09:43:38 +01:00
Egon Elbre
b5e95ec79f satellite/metainfo/metabase: add extra validation to IterateLoopSegments
Change-Id: I377bf9f661dcff6ba29d10f2909eb59d6d361c5d
2021-03-06 11:19:36 +02:00
Egon Elbre
150d75da28 satellite/metainfo/metabase: remove old TODO notes
Change-Id: Id216eb8cc5014652cce77a0f8f17ac66715bd46f
2021-03-05 15:16:46 +02:00
Michał Niewrzał
8de1116eeb satelite/metainfo: remove unused method from metainfo loop
IterateDatabase method was used by zombie segment reaper which is removed for multipart implementation.

Change-Id: I93e1294236612d6d82b2ab57053bb84e653f72b4
2021-03-04 12:48:27 +00:00
Egon Elbre
ec67413776 satellite/metainfo: add as of system time to object iteration
Change-Id: Idc1d5c7d983f331c60f5b75c5a977a25e197faf9
2021-03-03 13:39:47 +02:00
Egon Elbre
b0b7b81105 satellite/metainfo/metabase: streams iterator
Iterate over streams/segments rather than loading all of them into
memory. This reduces the memory overhead of metainfo loop.

Change-Id: I9e98ab98f0d5f6e80668677269b62d6549526e57
2021-03-03 13:16:49 +02:00
Michał Niewrzał
c51ea68ad3 satellite/metainfo/metabase: reduce number of fields for LoopSegmentEntry
For metainfo loop we need only some of Segment fields. By removing some of them we will reduce memory consumption during loop.

Change-Id: I4af8baab58f7de8ddf5e142380180bb70b1b442d
2021-03-02 15:04:54 +01:00
Michał Niewrzał
c223c2d845 satellite/metainfo/metabase: rename ListObjectsSegments to ListLoopSegmentEntries
This method will be used only with metainfo loop and we need to customize query to consume less memory.

Change-Id: Iaa97392f483c5df5609d501b3847b80eb1ea2583
2021-03-02 12:24:59 +00:00
Michał Niewrzał
8e97111513 satellite/metainfo/metabase: reduce fields number for LoopObjectEntry
We want to read from DB only those fields that are used by metainfo loop so we need to remove most of fields from LoopObjectEntry.

Change-Id: I14ecae288f631dc0ff54f4c560ce43b736eccdcf
2021-03-02 12:21:19 +00:00
Egon Elbre
7e72a231c2 satellite/metainfo/metabase: cast bucket_name to []byte
Currently our metabase assumption is that it may contain arbitrary
bucket names and endpoint applies the naming constraints as it sees fit.
However by passing bucket_name as TEXT pg and crdb automatically try to
convert it to []byte, which may or not may work as intended... or in
some cases not work at all.

Cast all bucket name arguments to []byte to make it work.

Change-Id: I44650f5c873010997398bb0163d7f56ff6d9b5cf
2021-03-02 13:22:49 +02:00
Egon Elbre
ba0197a9b7 satellite/metainfo/metabase: avoid full table scan
Change-Id: Id47ffb5e1287d9303ce6fb530e87dbdc23cf8307
2021-03-01 23:34:53 +02:00
Michał Niewrzał
646cf229a2 satellite/metainfo/metabase: rename FullIterateObjects method to IterateLoopObjects
We want to have custom loop iterator to avoid reading all object fields to reduce memory consumpion. This is first step to just rename existing iterator to IterateLoopObjects.

Change-Id: I8878ff21a49ba224db2d497cc8f9076e75c7609e
2021-03-01 17:19:05 +01:00
Egon Elbre
994854dcda satellite/metainfo/metabase: use key with prefix as the next cursor
We shouldn't use a trimmed object key as the next cursor.

Change-Id: I8513f7d94ed366dff9e9eca732c0e1ded6f5cd71
2021-02-28 22:05:42 +02:00
Egon Elbre
261a4c1c09 satellite/metainfo/metabase: fix iterator boundaries
Currently the old encrypted keys may not match the path component
encoding. Change the iterator such that the prefixes handle arbitrary
byte sequences.

Change-Id: I0a50049f4ef9887e1c4df6f9692f967a054430eb
2021-02-28 21:19:51 +02:00
Michał Niewrzał
68605f32ed satellite/metainfo: limit number of batched segments in metainfo loop
New metainfo loop can have memory issues when in one batch we will have object with many segments. This change limits number of batched segments to defined limit. Solution is not perfect as if we will have single object with extreme large segments count it can cross defined limit a lot. We need to prepare safer solution soon.

Change-Id: Iefcf466d5bac76513d4219b1a9d99adc361c54ae
2021-02-26 12:32:43 +01:00
Michał Niewrzał
a0cc7e8c5e satellite/metainfo/metabase: use segment.Pieces to check if segment is inline
It looks that we cannot use root piece id as indicator if segment is inline as we have case in SLC satellite that inline segment have root piece id set. Pieces should be better thing to check.

Change-Id: I2377ff88861390342273f5e71871373eaf462615
2021-02-24 22:14:18 +01:00
Michał Niewrzał
846bb895e0 satellite/metainfo: loop cleanups
Change-Id: I89b4f519ce18cb0ce34a61a4cbe1675d93741637
2021-02-23 17:48:36 +01:00
Michał Niewrzał
95b78e8011 satellite/metainfo: select segments in batches for metainfo loop
Segments are not read in batches. For each batch of objects
we are reading all segments for those objects.

Change-Id: Idaf19bbe4d4b095065d59399dd326e22c57499a6
2021-02-23 11:38:44 +00:00
Ethan
79ecd80c0a satellite/metabase: Remove subquery on delete objects for CRDB
The subquery for DELETE FROM obects returns a stream_id field for filtering. Unfortunately stream_id is not indexed. This change removed the subquery from the CockroachDB delete bucket query.

Change-Id: If1abe21668c593e6d4bdc3ba8cdbad26c09d234e
2021-02-19 16:51:01 +00:00
Egon Elbre
1137620baf satellite/satellitedb: move tests to their domains
Testing interfaces is slightly clearer when it's in the package needing
the database rather than each individual implementation.

Change-Id: I10334c214a205f7e510b939b4359a2214c4e060a
2021-02-19 17:29:15 +02:00
Kaloyan Raev
e0f15130a2 satellite/metainfo: ensure list prefix is prepended in satStreamID
When listing pending objects with prefix, the prefix should be prepended
to the EncryptedPath in satStreamID. Otherwise, listing multipart
uploads may display different UploadID than expected.

Change-Id: I27e9f9af9348783e053ad123121b6ddd051739e4
2021-02-19 16:32:55 +02:00
Michał Niewrzał
8b9da01817 satellite/metainfo/metabase: allow committing empty inline segments
We need to keep empty inline segments as we did it with pointerDB because otherwise old uplinks after uploading data won't be able to download such file. To reduce number of empty inline segments on uplink side we need to implement skipping empty last inline segments for multipart upload.

Change-Id: Ice86c805babba1ad17149754cbd6b3f4fd652722
2021-02-19 13:41:34 +00:00
Egon Elbre
4c0ea717eb satellite/metainfo: remove unneeded dependencies from Loop
metainfo.Loop doesn't require buckets nor pointerdb anymore.

Also:
* fix comments
* update full iterator limit to 2500

Change-Id: I6604402868f5c34079197c407f969ac8015e63c5
2021-02-19 15:11:16 +02:00
Egon Elbre
a25e35f0b0 cmd/metainfo-loop-benchmark: add benchmark
Change-Id: I0745cfcf9f8c9d73fd025dcba6ee8a7480273fe2
2021-02-19 10:59:41 +02:00
Egon Elbre
1cb6376daa satellite/metainfo: remove BucketsDB.ListAllBuckets
The ListAllBuckets implementation was buggy, remove it altogether.

Change-Id: Id457ba5f4d793156af3fc2071f74ce1be17ba804
2021-02-19 10:59:41 +02:00
Egon Elbre
8e95e76c35 satellite/metainfo: fix metainfo loop
ListAllBuckets could skip buckets when the total number of buckets
exceeds list limit. Replace listing buckets with looping directly
on the objects table.

Change-Id: I43da2fdf51e83915a7854b782f0e9ec32c373018
2021-02-19 10:27:08 +02:00
Egon Elbre
adf687aebb satellite/metainfo/metabase: add full iterator
Add FullIterateObjects that iterates over all objects in the metabase.

Change-Id: I4b8dbda32dfce2d7729e9574af79e4f20faed98a
2021-02-18 16:51:35 +02:00
Michał Niewrzał
dd9ad09301 satellite/metainfo: return RS value with DownloadSegment response
Until now we where using single RS per object but it turns out that we
need to be able to support RS per segment. We need to give uplink such information while downloading.

As an addition we are using RedundancySchemePerSegment flag for GetObject request to detect if
we should try to get RS from segment for this request response.

Change-Id: I209dad324496ff59b521b11d2343da61dcdbe7f5
2021-02-17 12:33:48 +00:00
Michał Niewrzał
f1169ad7bd satellite/metainfo: return RS value with DownloadSegment response
Until now we where using single RS per object but it turns out that we
need to be able to support RS per segment. We need to give uplink such information while downloading.

Change-Id: I6565b7c08962b3a1429f6079e7c2023a0a7c8b72
2021-02-17 12:39:53 +01:00
Egon Elbre
61f0fb67a9 satellite/metainfo/metabase: refresh alias cache only once
When there are concurrent refreshes to the cache and the entries are
missing, it could end up causing multiple database calls, even though
only one is needed.

Change-Id: I1ae7a124bbdd1570473cf3a032d375d2f25a8426
2021-02-17 10:00:04 +00:00
Fadila Khadar
259b030b3a satellite/metabase: add tests for BeginObject
adds tests to BeginObjectNextVersion and BeginObjectExactVersion
to check the behavior when an older or a newer committed version
exists.

The current behavior is: everything is committed.

Change-Id: Ia8facbe0dc038a5d214e4e56da3c8e4df2f18900
2021-02-17 09:02:26 +00:00
Michał Niewrzał
6ebe06cd1b satellite/metainfo: fix for getting real RS values
Change-Id: Ib2cd01b2a68baab9f342dc0ff1ab8e5f12f4557f
2021-02-16 17:26:09 +01:00
Michał Niewrzał
b8513e2fcb satellite/metainfo: get encryption from StreamMeta for old uplinks
Old uplinks sends some additional information inside marshaled protobufs and we need to extract things like encryption parameters. Newer uplinks are passing it directly in request.

Change-Id: I0b575e68c3ed98481247fe38344e7d61cbd542ba
2021-02-12 13:15:49 +00:00
Michał Niewrzał
341a4b7f59 metainfo-migration: extend test case
Change-Id: I9d1ac41a18bb08200ef20f7ff2f8df5531140f99
2021-02-11 19:28:41 +01:00
Egon Elbre
2848bf488f satellite/metainfo/metabase: use alias pieces in segments table
This makes all tables automatically convert between aliases and piece
ID-s.

Change-Id: I27fa42c82bbb09e05e3327f85e13a000b48faffd
2021-02-11 16:40:37 +02:00
Egon Elbre
25f81f353c satellite/metainfo/metabase: add AliasPiece
This adds AliasPieces run length encoding. On average it should
make our pieces encoding:

   repair=50,optimal=85,total=90   152.0 bytes
   repair=16,optimal=37,total=50    65.4 bytes

Change-Id: I391a9183164828f05383a3cde9ab0e4549c2d440
2021-02-11 14:23:51 +02:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Michał Niewrzał
dc2bec9f89 satellite/metainfo: collect uplink versions with monkit
This change adds metrics for uplink version and method which was used by this version.

Change-Id: I05fc425a024805cfcd6d8add810a2a0749405f4f
2021-02-10 17:47:13 +01:00
Egon Elbre
0c223f288a satellite/metainfo/metabase: don't use random node ID-s
We will add a cache to nodes, so using completely random nodes wouldn't
show the actual performance.

Change-Id: I94f18283712812f05f7795efd3c7cf57499fa52c
2021-02-08 11:36:05 +02:00
Egon Elbre
d0a7ab60ec satellite/metainfo/metabase: add NodeAliasCache
This implements a node alias cache that automatically refreshes
and adds new nodes when needed.

Change-Id: I4af121f7307b7b7ac4f87ad465f02d4eeb17d42f
2021-02-08 11:36:03 +02:00
Egon Elbre
0260966de4 satellite/metainfo/metabase: add NodeAliasMap
We need to keep an inmemory cache to avoid lookups into aliases table.
This adds the inmemory state of the cache.

Change-Id: Ief2b9bb19e10b46839b9208472dfc3035eb49af3
2021-02-08 10:28:22 +02:00
Egon Elbre
322c3a167b satellite/metainfo/metabase: add node alias table
This is first step in supporting node aliases. It adds a table
that automatically assigns aliases to nodes inserted into the table.

Change-Id: Ibdf40097c3c1e5b371500203f8db203505a48adc
2021-02-05 14:34:15 +02:00
Caleb Case
f011a5e8cc
Use WithNonce when creating caveats
This ensures the caveats are unique even when they contain the same
permissions and will result in unique macaroons. This is important to
ensure revocation doesn't impact more macaroons than intended.

Change-Id: I6354edd0119f2d85eaf580f2d1926a3de9151b88
2021-02-04 13:46:32 -05:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Fadila Khadar
f34519c969 metainfo/metabase: iterate through pending objects by key
Change-Id: I451453d52278a2ff15ade1a6fccede136ab314be
2021-02-03 11:44:05 +01:00
Kaloyan Raev
038bd0a4da satellite/repair/repairer: fix repair for pending objects
https://storjlabs.atlassian.net/browse/PG-160

Change-Id: Ice7a0dcfc591bcde85a355cf95fff1eb3411f508
2021-02-02 19:50:10 +02:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Egon Elbre
0bbe2f1d61 satellite/metainfo: add unimplemented ListPendingObjectStreams
Change-Id: I6c0fd240ce5be82c1f464470a6f147289b1cdf9d
2021-02-02 16:51:36 +02:00
Kaloyan Raev
b519bb377d satellite/metainfo/metabase: GetSegmentByLocation
We need this method to fix repairing pending objects. In another PR, it
will replace the GetObjectLatestVersion + GetSegmentByPosition calls
that are currently executed.

Change-Id: I4c5c2ab604edf898452b6fd21b86d4d3f970ce79
2021-02-01 21:52:12 +02:00
Kaloyan Raev
6f441960ec satellite/metainfo/metabase: fix ParseSegmentKey to parse part numbers
ParseSegmentKey did not parse part numbers correctly from an encoded
SegmentKey.

Change-Id: I4ec7e15e0a094ed2a659c8c09d2f024eaceda678
2021-02-01 15:18:18 +02:00
Kaloyan Raev
0f1961b7d1 satellite/metainfo/metabase: improve subquery comparision performance
Comparing the result from a subquery with the "IN" operator instead of
"=" makes a huge difference in the execution time of the SQL query on
CockroachDB.

Change-Id: I76e8f75a7bc95951667345d1ed9bd60f9aef3edb
2021-01-27 17:45:27 +02:00
Michał Niewrzał
3fc0d2a83e satellite/metainfo: add testing method from multipart-upload branch
We wanto have single uplink branch for standard and multipart-upload satellite but some tests are using helper methods from multipart. This change adds methods used by uplink test.

Change-Id: I82352ed56674ff7e8743b58061ba594018e78e3b
2021-01-26 09:13:12 +00:00
Michał Niewrzał
50bea3ab1a satellite/metainfo: adjust tests to changes in uplink
One of uplink method changed its signature and we need to fix test on satellite side.

Change-Id: Ib89ea6aa25c57bac11bc3e0e9c2c89a4b9debd7c
2021-01-26 08:54:35 +00:00
Egon Elbre
a700a1bdab satellite/metainfo/metabase: add benchmark
Change-Id: I4289aac252f9c951548444aa5a82e007e9f47806
2021-01-25 14:59:24 +00:00
Kaloyan Raev
ba2907be1b satellite/metainfo: do not expire satStreamID
We are checking if satStreamID is created in the last 48 hours. If it is
older we treat is as expired an fail to unmarshal it.

Since the satStreamID is also the Upload ID for multipart uploads, this
means that all calls fail for multipart uploads older than 48 hours.
Even aborting old multipart uploads is not possible.

To resolve this issue, we should stop checking satStreamID for
expiration.

Change-Id: Ieaf53ed3cd800cdd08843676c2d9490b007d962e
2021-01-25 13:35:20 +00:00
Egon Elbre
b8fd8c775a satellite/metainfo/metabase: fix FixedSegmentSize for gapped parts
Parts that have segment index gaps should be treated similarly how
multipart objects are, because direct calculation of the segment does
not work.

Change-Id: I2717eac36f085b5100f3d600fcf0ce056202a9eb
2021-01-25 13:30:56 +02:00
Egon Elbre
2ce829a8b1 satellite/metainfo/metabase: fix segment index in tests
Change-Id: Iaf9591229454d54d6e6927893738be678926b31c
2021-01-25 13:27:24 +02:00
Michał Niewrzał
3c13aae61e satellite/metainfo: remove unused method
CreateGetOrderLimits is not used anymore because we have CreateGetOrderLimits2. We need to remove old method and fix name of second.

Change-Id: I59148b8d28fc9dbab7d452c884319125a02745d1
2021-01-21 17:00:13 +01:00
Michał Niewrzał
ce675b7707 satellite/metainfo/metabase: remove sql transaction while committing segments
Change-Id: Ia2cb1df63e1cbb8b9d36b0161a75b9c9cb3a56f9
2021-01-21 13:20:42 +00:00
Michał Niewrzał
292caa9043 satellite/metainfo/metabase: allow to set Encryption while committing object
In some cases we need to set encryption parameters later, with CommitObject method. This change makes Encryption optional with BeginObject* methods and mandatory with CommitObject if not set earlier.

Change-Id: I812c9b0e8fc213ca32d4758e0e68227e0e9bdd32
2021-01-20 18:28:25 +01:00
Michał Niewrzał
2d087c54b1 satellite/metainfo/metabase: use PlainSize for fixed segment calculation
In the past we were storing fixed segment size with StreamInfo, encrypted in metadata. The value was unencrypted size of segment, not encrypted one.

Change-Id: Id6b18440c674223eabbb152b1636c83e1ab6462c
2021-01-20 16:40:27 +00:00
Kaloyan Raev
ea48322dd3 satellite/metainfo: use deterministic signing for satStreamID
So we can have stable UploadID for multipart uploads.

Change-Id: Iac6780394c8cc0f96c0b9c4b850b92ed3627a9b0
2021-01-19 15:56:29 +00:00
Michał Niewrzał
d54ae9f10f satellite/metainfo: close project in TestObjectOverrideOnUpload
Change-Id: If19245474f54cea1657f704e155e09e1511eaeec
2021-01-19 16:33:30 +01:00
Egon Elbre
8e47adfad5 satellite/metainfo/metabase: optimize a query
Change-Id: I6395225f68cd8fbc8e3078c80ac6cb079c3c6a4f
2021-01-19 15:37:03 +02:00
Michał Niewrzał
38beecc7ad satellite/metainfo: Override pending object on upload
On upload we need to override pending and committed object. This change is adjusting DeleteObjectAllVersions to delete both.

Change-Id: Ib66c2af207c618119f7bf0de7fa9d3e5145d8641
2021-01-14 15:16:02 +00:00
Michał Niewrzał
95320912b4 satellite/metainfo: simplify metainfo delete test
Simplifies tests for pending and committed objects.

Change-Id: I0c683198f8fce62898142c6c343aca985d91a77f
2021-01-14 15:06:06 +00:00
Kaloyan Raev
6dff40f5c5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/metainfo.go

Change-Id: Ib5c49f3c911c58319855a171f9ce73657da976d9
2021-01-14 14:33:59 +02:00
Egon Elbre
85fb964afe satellite/{metainfo,overlay}: improvements to GetObjectIPs
* Deduplicate NodeID list prior to fetching IPs.
* Use NodeSelectionCache for fetching reliable IPs.
* Return number of segements, reliable pieces and all pieces.

Change-Id: I13e679caab275488b4037624b840a4068dad9589
2021-01-14 09:12:45 +00:00
Egon Elbre
d11c2b709e go.mod: bump storj.io/common
* Add missing endpoints.
* Fix deprecated packages and funcs.

Change-Id: I756090c46a4d15eabf6d413a593cdc64c5809bc7
2021-01-13 14:51:08 +00:00
Ivan Fraixedes
a4d06b9b1e satellite/metainfo: Don't response errors when Redis down
For being able to have resilient multi-region satellites we cannot stop
processing uploads/download client request when Redis isn't responding
properly.

These changes avoid to stop the processing of the client requests when
we cannot check if the client exceeds its storage or bandwidth limits
and we cannot update its used storage/bandwidth limits because Redis is
not responding successfully or the satellite database returns an error.

Change-Id: Ia7f12c07fc9ffdfad0e7ff052ff3fd81eca0f0e3
2021-01-13 14:30:44 +00:00
Michał Niewrzał
ac058e5ecc metainfo-migration: basic pointerdb->metabase migrator
Change-Id: If183b9c99ce816521fcae6e7189a6f85ddd4eb48
2021-01-12 12:59:53 +00:00
Egon Elbre
0df0e980a0 satellite/metainfo/metabase: fix FixedSegmentSize calculation
Commiting multipart object with same sized segments should always
end with FixedSegmentSize = -1.

Change-Id: Iedeafb763b62b2513de0a753dce20ddad5b32d90
2021-01-11 14:47:49 +02:00
Egon Elbre
24833465e6 satellite/metainfo/metabase: avoid magic constant
Change-Id: I4f01e38f67e18ae9cb9845a8e75a987acba66427
2021-01-11 10:22:21 +00:00
Michał Niewrzał
ec88d21a3c Merge 'main' branch.
Change-Id: I6e8162d1a6caf75e89c9f9c9f9522730aebf83ae
2021-01-11 10:26:58 +01:00
Michał Niewrzał
71ca18134b satellite/metainfo: reject downloading a segment from multipart object
Non-multipart uplink implementation is always trying to download object
by downloading last segment first (PartNumber=0, Index=-1) but this
approach won't work with multipart object. We need to reject such old
style request with reasonable message.

Change-Id: I9221e019933565a8d25136bdfef3e054320bac3d
2021-01-11 08:04:33 +00:00
Egon Elbre
51731db121 satellite/orders: use smaller encrypted metadata
Avoid using project uuid string representation, because
it uses more bandwidth.

This reduces the encrypted metadata size from 118 -> 97 bytes.

Change-Id: Ic53a81b83acc065f24f28cd404f9c0b1fe592594
2021-01-08 16:40:31 +00:00
Michał Niewrzał
da316c27aa satellite/metainfo/metabase: disable PlainSize validation
We need to accept segments with PlainSize == 0 to be backward compatible
for old uplinks.

Change-Id: I926abe751e9e0ad7e06e42a5301ab78a3f23aafa
2021-01-07 14:14:20 +00:00
Kaloyan Raev
4fc61f7afa satellite/metainfo/metabase: allow commmitting objects > 2 GiB
The total_plain_size and total_encrypted_size columns in the objects
table were set as INT4, which limits the size of committed objects to
just 2 GiB.

This patch migrates the DB to change the type of these fields to INT8.

Change-Id: Iad7e7b44a652e6c5b8e17b80588637bb48390fe6
2021-01-06 18:23:03 +02:00
Michał Niewrzał
ad3e3a38c5 Merge 'main' branch
Change-Id: Ia0db1b1f9ef3e0671d3f2208881b0abc3064e200
2021-01-04 12:13:45 +01:00
Rafael Gomes
8b2e4bfa7e satellite/metainfo/piecedeletion Remove spaces from metrics.
Change-Id: Iaf1d8a96a43087f2fcc579347f581e8a78a0fb58
2020-12-30 14:27:39 -03:00
Kaloyan Raev
777f6e583b satellite/metainfo: replace a call to PointerDB with call to Metabase
Change-Id: I10aa89dacf91cb6c1528698031e6b53c52915bd9
2020-12-22 12:44:28 +02:00
Michal Niewrzal
66d4d5eb48 satellite/metainfo/metabase: implement IterateObjectsAllVersions for
pending/committed objects

Change-Id: Ibf390821b6a23919078de4f18c2653e308320904
2020-12-22 10:27:09 +01:00
Michal Niewrzal
dad8360b39 satellite/metainfo/metabase: rename IterateObjectsAllVersions to
IterateObjectsAllVersionsWithStatus

We need different implementation for IterateObjectsAllVersions because
we want to iterate over all object without specifying object status.
Existing method will have new name but implementation details are not
changed.

Change-Id: I01b987996772fa7f8fd73da9910d52db2d1aa0d7
2020-12-21 16:47:32 +00:00
Kaloyan Raev
bafc6af992 ci: remove workaround for failing tests
Change-Id: I3eb673fae6c81bee17d7437cb870d5f5ba6978d5
2020-12-21 18:07:40 +02:00
Michal Niewrzal
f3ef8088e7 satellite/metainfo/metabase: add Verify method for Pieces
This change adds Verify method for pieces to do some basic checks.

Change-Id: I0ff4313b594d2cb3aad7da545f940e10ee654b77
2020-12-21 12:49:03 +00:00
Michal Niewrzal
18825d1e0b satellite/{metainfo,gracefulexit}: fix failing tests
Change-Id: I3428ea601255c36a316732c9f75135d6e5fa4d79
2020-12-21 12:22:32 +00:00
Kaloyan Raev
4d37d14929 satellite/{metrics,repair}: adjust monitoring to new metainfo loop
Change-Id: I87a2145daa5ed49bb2c08d6967baa09c0b14b4c6
2020-12-21 09:05:17 +02:00
Egon Elbre
65b22be417 satellite/accounting/tally: use metabase
Change-Id: I6d49dc103a18e8a110bfa7775d53a65d208b6c2c
2020-12-18 16:18:03 +02:00
Michal Niewrzal
311b082838 satellite/metainfo: fix metainfo loop
This fix issues with passing observers between iteration methods.

It's not best implementation but I think we will need to optimize it
soon one way or another.

Change-Id: I574599bfd10822d84e2d2f1800bcd88e176a76ea
2020-12-18 10:38:12 +00:00
Michal Niewrzal
2111740236 Merge 'master' branch
Change-Id: Ib73af0ff3ce0e9a1547b0b9fc55bf88704f6f394
2020-12-18 09:13:24 +01:00
littleskunk
3feee9f4f8
satellite/accounting: default project limits (#4001) 2020-12-17 22:27:05 +01:00
Michal Niewrzal
70ba4deea9 satellite/repair/checker: adjust irreparable part of repair checker
Change-Id: I0732104a97ba18a5359de3966cd692677a0ff790
2020-12-17 14:11:22 +00:00
Kaloyan Raev
9aa61245d0 satellite/audits: migrate to metabase
Change-Id: I480c941820c5b0bd3af0539d92b548189211acb2
2020-12-17 14:38:48 +02:00
Michal Niewrzal
8d3ea9c251 satellite/repair/repairer: implement SegmentRepairer with metabase
Change-Id: I647c625e00a626c44e812602ad9bc3e85a7b602c
2020-12-17 10:47:21 +00:00
Kaloyan Raev
6206aa88e4 {satellite/metainfo,private/testplanet} use TestingAllSegments in tests
Change-Id: I8c641a24fabf3ea537312978a42501eab8d6a339
2020-12-17 09:58:31 +00:00
Egon Elbre
1728e45e55 satellite/metainfo/metabase: optimize DeleteBucket
Change-Id: If34cdeae0f688cb96717905fc8287c66ea3034be
2020-12-16 14:39:33 +00:00
Egon Elbre
4706f01876 satellite/metainfo: add TestingAll{Segments,Objects}
Change-Id: Ia758c119d5ebd7bbb21216a463c99c2e9afcdeb0
2020-12-16 10:21:36 +00:00
Kaloyan Raev
4fba9921f6 satellite/metainfo/metabase: define ErrSegmentNotFound error class
This makes it easier to callers of GetSegmentByPosition to determine if
the segment is missing.

Change-Id: I2d8546dddf07dcf790d2f7c08d308ed589b34f2f
2020-12-15 15:48:41 +02:00
Michal Niewrzal
934ae32ca4 satellite/repair/checker: fix checker tests
Change-Id: I63d3368a07b800fdb10bb93b847eb32927b8c0dc
2020-12-15 10:47:42 +00:00
Kaloyan Raev
fc85179a19 satellite/metainfo: refactor SegmentLocation.Index to SegmentPosition
Change-Id: Ic9403c8126712693326dd83d6ba4f3b84be3e0c7
2020-12-14 13:35:53 +02:00
Kaloyan Raev
7d8f19e94d satellite/metainfo: metainfo loop should yield StreamID for segments
Change-Id: If6c86add75ce79ffcfe95353225719c7d4b5a459
2020-12-14 13:04:38 +02:00
Michal Niewrzal
e7e6985ae9 satellite/metainfo/metabase: add UpdateSegmentPieces method
We need to be able to update just remote_pieces column in DB. This is
needed at least for repair process.

Change-Id: I20dcc9b06babfefbbf102f32b1d14946379f26c2
2020-12-14 10:25:09 +00:00
Kaloyan Raev
2bb010e7c5 cmd: remove segment reaper
It was designed to detect and remove zombie segments in the PointerDB.
This tool should be not relevant with the MetabaseDB anymore.

Change-Id: I112552203b1329a5a659f69a0043eb1f8dadb551
2020-12-14 09:36:37 +00:00
Michal Niewrzal
7e6e0d3e2e satellite/metainfo: metainfo loop implementation with metabase
Change-Id: Iadac469519de605a88e624df23265289771b2006
2020-12-11 16:15:57 +01:00
Michal Niewrzal
b3acc1101a Merge 'master' branch
Change-Id: Iee99400c7095770e61cde94b3b2c8eb0ddec463d
2020-12-10 15:42:52 +01:00
Michal Niewrzal
c2a97aeb14 satellite/satellitedb: add ListAllBuckets method
We need to be able to list all buckets in DB without knowing project ID.
This method will be used to list buckets for metainfo loop
implementation based on metabase.

Change-Id: Iac75af0eee4f31e80a15577575a8249cbca787b2
2020-12-10 14:19:27 +00:00
Fadila Khadar
15add3c752 satellite/metainfo: enable deletion of a pending object.
Change-Id: I1ac24ad340d411c52b92ef18f51a684d9214ed47
2020-12-10 13:56:14 +01:00
Egon Elbre
365410d10b satellite/metainfo/metabase: add commit with specifying a list of segments
Change-Id: Ibb9999545691d150f36e20ce70ac00a3802ad7b2
2020-12-09 15:16:38 +02:00
Michal Niewrzal
f077564bb7 satellite/metainfo: setup access to metabase in metainfo loop
This is initial change to replace metainfo loop implementation with
metabase.

Change-Id: Ib23cb047bbbe6cff146305935a16ba833d40351a
2020-12-09 09:50:04 +01:00
Kaloyan Raev
3f5e07fb1f satellite/metainfo: more tests fixed
- TestBucketNameValidation
- TestBatch
- TestCommitObjectMetadataSize
- TestIDs

TestOverwriteZombieSegments is removed as not relevant to metabase.

Change-Id: I13cf5abe342089960628f185061303fd4f9d09a4
2020-12-08 14:00:24 +02:00
Kaloyan Raev
d0d0a192cd satellite/metainfo: fix TestEndpoint_DeleteObjectPieces tests
This also removes the
TestEndpoint_DeleteObjectPieces_ObjectWithoutLastSegment test case as it
does not seem relevant to metabase.

Change-Id: I06a0ecaa8232c10c15e433517a7ba056933bf858
2020-12-08 10:22:05 +00:00
Michal Niewrzal
fb5244e8f1 satellite/metainfo: migrate GetObjectIP endpoint method to metabase
Change-Id: I0da346fc2ce84787a09d2bb7c4ffc02a5d8a3b2f
2020-12-08 08:50:15 +00:00
Michal Niewrzal
218bbeaffa Merge 'master' branch
Change-Id: Ica5c25607a951076dd9f77e35e308062f71ce3f0
2020-12-07 15:05:52 +01:00
Stefan Benten
494bd5db81
all: golangci-lint v1.33.0 fixes (#3985) 2020-12-05 17:01:42 +01:00
Ethan Adams
f90ea10a4a
Allow for DB application names per process. (#3983) 2020-12-04 11:24:39 +01:00
Kaloyan Raev
a7685f50c9 satellite/metainfo/metabase: set maxParts to MaxListLimit if greater
We should set the client requested maxParts to MaxListLimit if it is
greater than that value instead of returning an error.

MinIO default value for maxParts is 10,000 while the satellite's
MaxListLimit is 1,000. If we return an error, the ListParts with default
maxParts will throw an error.

Change-Id: I06739e1d8d8f96803eba491585395da0443aec04
2020-12-03 11:32:07 +00:00
Fadila Khadar
fd7cc20a7e satellite/metainfo/metabase: add DeletePendingObject
Change-Id: I7ba7d3d7f179712940e91b05501310279bbb97a4
2020-12-03 10:21:04 +00:00
Kaloyan Raev
bc25cc8ddd satellite/metainfo: add encryption info in metadata of list items
This is needed to resolve 'encryption type 0 is not supported' errors in
gateway.

Change-Id: I33c486968d5a1a1310b82e9778a8ce01429f7043
2020-12-02 13:34:41 +02:00
Michal Niewrzal
83e7cd2a40 satellite/metainfo/metabase: replace SQL substring with go code
We have some issues with SUBSTRING function on cockroachdb so for now we
are removing it from SQL query and replacing with go code.

Change-Id: I5be921211067d42e7d1a4997076bcfdbed9617a1
2020-12-02 10:47:18 +01:00
Kaloyan Raev
e8378a87bf satellite/metainfo: include redundancy in satStreamID of ListObjects
This fixes the issue with using UploadID from the ListMultipartUploads
response.

Change-Id: If077a5804d03cc139f75febf472d5fb73f0f7929
2020-12-01 15:11:34 +00:00
Michal Niewrzal
d5c0264163 satellite/metainfo: implement deleteBucketObjects with metabase objects
iterator

This method replaces `deleteByPrefix` as at the moment only function of
this method was to delete objects in a bucket.

Change-Id: I5266103672003fbd64f3847f53760b1ba0016fe2
2020-12-01 14:54:23 +00:00
Michal Niewrzal
f08e34f15e satellite/metainfo: fix listing objects when cursor is specified
We need to combine prefix + cursor to iterate objects correctly.

Change-Id: I2f4a29ba3ee65ce52eb1e8b88a3b09cf93e0b2bb
2020-12-01 15:01:44 +01:00
Michal Niewrzal
e4c4ab06ba satellite/metainfo: add testing methods for getting objects and segments
Change-Id: I4229d80090609df27d998a90bdb3137469dba042
2020-12-01 12:44:08 +00:00
Kaloyan Raev
46839b992a satellite/metainfo: expired deletion service to use Metabase
It also sets the Expires time correctly in the Metabase when uploading
objects.

Change-Id: Iec1b3ec8d11346a91dfc2ba23a1b08edec4a84d3
2020-12-01 09:12:14 +00:00
Michal Niewrzal
5a7bc9657d Merge 'master' branch
Change-Id: If583132a821274dc4b78cf5f72b853ba8460c619
2020-11-30 12:57:22 +01:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
2020-11-30 10:49:23 +01:00
Michal Niewrzal
cfb45a785c satellite/metainfo/metabase: add TotalPlainSize to objects table
We need to know how large is object in general and we shouldn't query
segments for that.

Change-Id: Ie316e1ec38bfc39065e6046b79b55d58f3ce9d53
2020-11-25 14:35:28 +01:00
Michal Niewrzal
8ceef9f357 satellite/metainfo: temporary disable one assertion in test
This is need to merge https://review.dev.storj.io/c/storj/uplink/+/3208
, after that this code will be back.

Change-Id: If9f2f1db95c7a1bba64a41c45a39bd3096a519e7
2020-11-25 13:21:41 +00:00
Michal Niewrzal
983b1737c2 satellite/metainfo: implement IsBucketEmpty with metabase
Use objects iterator to check if bucket is empty.

Change-Id: I590bb7258a47dd816f079039a1151df809dd2637
2020-11-25 12:20:30 +00:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
2020-11-24 17:50:33 +01:00
Egon Elbre
dad36179c6 satellite/metainfo/metabase: fix segment listing with cursor.Part
Change-Id: I910888d435b11d2d4687ea0042998a1078cd6443
2020-11-24 15:31:57 +02:00
Kaloyan Raev
67c210a64c satellite/metainfo: set list cursor to version 1
Otherwise, if left to default version 0, the iterator will include the
cursor item in the result, which fails some tests.

Change-Id: I85103a36852477f371ec46c673a82c2e129978b7
2020-11-24 10:17:42 +00:00
Michal Niewrzal
5be4d413b0 satellite/metainfo: set PlainSize for inline segment
Plain size is required by metabase implementation.

Change-Id: I6739744bff4ce485e0066f2a49acdf38560d31fb
2020-11-20 13:54:14 +01:00
Kaloyan Raev
05fe497a6f satellite/metainfo: add stream id when listing pending objects
Change-Id: Idcdffd70b82c0fe4f79635fc193604b66101ef76
2020-11-19 14:22:09 +02:00
Michal Niewrzal
1465539220 satellite/metainfo: ignore empty inline segments for backward
compatibility

Change-Id: I2567d77e172391945dee24cea8e110fa1963d663
2020-11-18 14:12:03 +00:00
Michal Niewrzal
70b1c7aa17 satellite/metainfo: check if EncryptedMetadataNonce was set
It possible that old uplinks won't set it.

Change-Id: Ifd3c4d4a45c5636cfeb70b0a37bcf1305a097727
2020-11-18 14:04:31 +01:00
Kaloyan Raev
a30f5d7ec0 satellite/metainfo: use Object.Status from ObjectListRequest
Change-Id: Ie2e48d2eedf69bd0f04b7c9d6e7e217fec6b0267
2020-11-18 13:31:52 +02:00
Michal Niewrzal
9bdc8ebf75 satellite/metainfo: adjust GetObject to handle
EncryptedMetadataEncryptedKey

Change-Id: If159fdcc9f37e9449a7d8459a277366d6d6b5802
2020-11-17 16:38:10 +00:00
Egon Elbre
72fed3b3fb satellite/metainfo/metabase: add slow non-recursive listing
Change-Id: Ide60640400caea6474465af5224010fde60826c7
2020-11-17 16:47:29 +02:00