Commit Graph

31 Commits

Author SHA1 Message Date
Egon Elbre
4469d229f8 satellite/metabase/{meta,segment}loop: avoid passing config
Currently the iterate is being called in only one location so there's no
benefit in passing them as arguments over using the receiver.

Change-Id: I433a5d8b795b1bcc1f1e9320d87b10820cf537f1
2021-06-02 15:55:59 +00:00
Egon Elbre
2cf10a7bf4 satellite/metabase/{segment,meta}loop: avoid no observers error
In a rare case it's possible to start the loop iteration without
observers. The most likely case is that the observer is cancelled and
the coalesce timer trigger asynchronously, although being stopped.

Nevertheless, all the observers may also exit during the iteration, in
either case it should not result in an error.

If there's a probem with the observers, then they can report their own
error as they see fit.

Change-Id: Ie423fec41e6295be05536a4b7b0b6623ffebf2fb
2021-06-02 11:57:59 +00:00
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
Michał Niewrzał
053e58b683 satellite/metabase: add segmentloop service
We want to move some of current metainfo loop observers to
segment loop. This change adds new service, similar to metainfo
loop but which is iterating only over segments.

Change-Id: I67f7f461781723a4476e2b83377f31736d7c4870
2021-06-01 11:15:07 +00:00
Egon Elbre
0ef537a685 satellite/metabase/metaloop: limit max as of system time
When using a system time too far in the past, the query gets slower.

Change-Id: I7014d13a16b1ec44391549f829e7a57942f1fea7
2021-05-28 12:07:36 +03:00
Michał Niewrzał
3af91e7a90 satellite/metabase: add iteration over segments
Method IterateLoopSegments can be used to iterate over all segments in metabase without touching objects.

Change-Id: I3cc0e783884b603b47ef3f8233e357aa8a391250
2021-05-26 09:59:33 +00:00
Egon Elbre
e7cf369cd8 satellite/metabase: use pgxutil.Conn helper
Clean up underlying pgx.Conn extraction with helper.

Change-Id: Ie43f663a6036c439e675df54fccd3605c4d8d69e
2021-05-25 05:56:35 +00:00
Egon Elbre
10372afbe4 ci: fix lint errors
Change-Id: Ib5893440807811f77175ccd347aa3f8ca9cccbdf
2021-05-17 13:37:31 +00:00
Egon Elbre
910eec8eee satellite/metainfo: remove MetabaseDB interface
Currently the interface is not useful. When we need to vary the
implementation for testing purposes we can introduce a local interface
for the service/chore that needs it, rather than using the large api.

Unfortunately, this requires adding a cleanup callback for tests, there
might be a better solution to this problem.

Change-Id: I079fe4dbe297b0ae08c10081a1cea4dfbc277682
2021-05-13 13:22:14 +00:00
Michał Niewrzał
547a6e9930 satellite/metabase: add DeletePart method
Method will delete all segments for given part.

Change-Id: I11bdfdf30176749aee03cea0d1d2fa467f7cebea
2021-05-11 22:45:43 +02:00
Egon Elbre
2ae80690cb satellite/metabase/metabasetest: remove STORJ_TEST_DATABASES
Initially metabase was developed separately and it was useful to have a
separate environment flag for tests, however, it's more convenient to
use the same as rest of the testsuite.

Change-Id: Ia4d79be27ce5911cbae68d57cdf0b30f63459444
2021-05-11 13:31:01 +00:00
Egon Elbre
0858c3797a satellite/{metabase,satellitedb}: deduplicate AS OF SYSTEM TIME code
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.

As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.

Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.

Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
2021-05-11 12:40:36 +03:00
Michał Niewrzał
033006403f satellite/metainfo: fix setting object ZombieDeletionDeadline
So far we were setting ZombieDeletionDeadline alwasy as nil and because of that DB default was never set. This change adds separate query for inserting object if deadline is not set.

Change-Id: I3d6a16570e7c74b5304e13edad8c7adcd021340c
2021-05-11 07:26:42 +00:00
Egon Elbre
6161436d8b satellite/metabase/metabasetest: sort results
Deletion results are not guaranteed to be sorted, hence we need to sort
them before comparing.

Change-Id: I6b571dd812d4256327c6c64fff74c780c62b2948
2021-05-07 12:36:02 +03:00
Egon Elbre
6e6051b172 satellite/metabase: drop alias migration code
We have migrated all of the satellites and we shouldn't keep dead-code
around.

Change-Id: I539d6766cfafa2f278ff7767ceb2d39f6777ace3
2021-05-07 12:13:28 +03:00
Egon Elbre
2af7e4ef26 satellite/metabase/metaloop: use database time
The system and database time may drift. We should use database time for
absolute "as of system time" to ensure that it's not newer than the
current database time. When the "as of system time" is in the future,
then the query will fail.

Change-Id: I5423f6aaad966ca03a76b5ff805bfba932e44a51
2021-05-07 09:08:04 +00:00
Michał Niewrzał
0d3865950f satellite/metabase: rename delete_expired.go to delete_objects.go
Logic for deleting specific object type can be reused so file name can be more general.

Change-Id: I75182c0d5e1b653eaa454c895f347dd33b4bebbc
2021-05-05 08:57:32 +00:00
Egon Elbre
5269596c7d satellite/{metabase,metainfo}: use ObjectStream as argument
Change-Id: I5a7f096002b4d7a6162b24d7a64346b058c4c89c
2021-05-04 15:30:59 +00:00
Ethan
dba9321483 satellite/metabase: Remove pending_index
Remove pending_index until more performance testing can be done.

Change-Id: I31d29b1f37998eb62aee8bd42c7dcb79859446d2
2021-05-04 09:49:39 -04:00
Michał Niewrzał
e8ef689929 satellite/metabase: add DeleteZombieObjects method
We will use this method to delete objects where zombie_deletion_deadline passed.

Change-Id: Id24322e40dad5b8bceb0f84080922c746c0b802d
2021-05-04 07:08:16 +00:00
Egon Elbre
961e841bd7 all: fix error naming
errs.Class should not contain "error" in the name, since that causes a
lot of stutter in the error logs. As an example a log line could end up
looking like:

    ERROR node stats service error: satellitedbs error: node stats database error: no rows

Whereas something like:

    ERROR nodestats service: satellitedbs: nodestatsdb: no rows

Would contain all the necessary information without the stutter.

Change-Id: I7b7cb7e592ebab4bcfadc1eef11122584d2b20e0
2021-04-29 15:38:21 +03:00
Fadila Khadar
5af898fcc9 satellite/metabase: add migration step for index on pending objects
Add an index on (project_id, bucket_name) for objects with pending status

Change-Id: Ibd3c949a99e49e8e6bb884564331ec16f9184942
2021-04-28 09:54:10 +00:00
Michał Niewrzał
7944df20d6 storj: use multipart API
Change-Id: I10b401434e3e77468d12ecd225b41689568fd197
2021-04-26 13:15:09 +00:00
Egon Elbre
948529a600 satellite/metabase/metabasetest: move test utils to a separate package
Change-Id: Iaa607996478497d5c97c6a6973d099b315b3abf0
2021-04-26 13:12:58 +03:00
Egon Elbre
8c62788b24 satellite/metabase/metaloop: stop timer properly
There's a rare chance that `Stop` returns false, however doesn't have
time triggered. Use a non-blocking drain to remove the token.

Change-Id: I1ae18a197424017f0ca76656602709a029b56bfd
2021-04-23 16:35:21 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
c1fbecb96b satellite/metabase/metaloop: add Monitor
We need some chores to join without triggering the loop.

For example it's fine to run metrics, only when something else is
running.

Change-Id: I9d8bd16f59c28c540c8d72971bc4e233a8660c02
2021-04-23 12:20:22 +03:00
Egon Elbre
fff21b330d cmd/metabase-verify: tool for verifying metabase state
Currently the tool verifies:
* validity of plain_offset
* whether plain_size is smaller than encrypted_size

Change-Id: I9ec4fb5ead3356a196392c26ca377fcdb367138e
2021-04-23 07:50:36 +00:00
Egon Elbre
4c9ed64f75 satellite/metabase/metaloop: move loop under metabase
Currently the loop handling is heavily related to the metabase rather
than metainfo.

metainfo over time has become related to the "public API" for accessing
the metabase data.

Currently updates monkit.lock, because monkit monitoring does not handle
ScopeNamed correctly. Needs a followup change to monitoring check.

Change-Id: Ie50519991d718dfb872ec9a0176a82e732c97584
2021-04-22 12:58:09 +03:00
Ethan
a9c9f080bb satellite/metaloop: Use AS OF SYSTEM TIME when querying the objects table
(*loopIterator).doNextQuery is not using AS OF SYSTEM TIME because the value is 0

Change-Id: I3c671c1b2aa32bd6379b5cdea6e16246b4def9fd
2021-04-21 17:18:52 +00:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00