Commit Graph

21 Commits

Author SHA1 Message Date
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Márton Elek
84d02f7fbf metabase: use predefined snapshot for unit tests avoid migration
Change-Id: I6225ae32c312030e3559d8ed0faa137ffc3cc5b7
2022-06-01 12:07:00 +00:00
Michał Niewrzał
fbe2680500 satellite/metainfo: add feature flag for server side copy code
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.

Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
2022-02-24 10:43:49 +00:00
Stefan Benten
321bf26c85
cmd,private,satellite: add application_name to metabase connections
Currently the metainfo/metabase DB connections are missing the proper
application_name in order to differentiate and filter queries on the DB
side for analytics.
Without it, it is very time-consuming to correlate processes and their load.

This change adds the "check" on DB connection init and passes the fallbacks
in all places to catch connection strings, that do not set it.

Change-Id: Iea5cea8658bc63778ff89038e5c1c352bf482cfd
2022-02-20 22:29:41 +01:00
Fadila Khadar
b1d9a4426f satellite/metabase: adjust segment_copies table primary key.
segment_copies table has (stream_id, ancestor_stream_id) as primary key,
but it should have only stream_id as primary as a stream_id can only have one ancestor_stream_id.

https://github.com/storj/team-metainfo/issues/84

Change-Id: I0218295e86e449c9dfefa5a21319f3083bf80afd
2022-02-10 13:52:16 +00:00
Michał Niewrzał
95921b8b39 satellite/metabase: add segment_copies table
Part of server-side copy implementation.

Adds new table to metabase for keeping references between
copied segments and its ancestors.

Fixes https://github.com/storj/team-metainfo/issues/84

Change-Id: I436d4b533a3d951bcd752e222e3b721cee7f0844
2022-02-08 11:15:19 +00:00
Elek, Márton
27091826ba satellite/{satellitedb|metabase}: add SQL fields for geofencing
Change-Id: Ia698d71db1d3775b908c24bd1c9bc8abfd015ccc
2021-11-04 08:49:56 +00:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
Clement Sam
1f353f3231 segment/{metabase,repair}: change segment created_at column to not accept nulls
This change adds a NOT NULL constraint to the created_at column in the segment table.
All occurrences of CreatedAt as a pointer are changed to non pointer version (metabase, segment loop, etc)

Change-Id: I3efd476ebd1edd3327b69c9223d9edc800e1cc52
2021-08-06 08:16:28 +00:00
Michał Niewrzał
ae345320fe satellite/metabase: use db methods that pass context correctly
It turns out that some DB methods are not
passing context to to lower levels even if have
context as a parameter.

Change-Id: I82f4fc1a47c362687a91364d85e4468057e53134
2021-07-29 09:32:31 +00:00
Egon Elbre
0df4a27bf7 satellite/metabase: add method to get table statistics
In loop we need to start verifying it's correctness. This allows to
gather these stats.

Change-Id: I146fb50e2b3658b6f3c2682cdc1983e6abd73c29
2021-06-16 18:34:55 +03:00
Egon Elbre
f3a52d1da5 satellite/metabase/segmentloop: limit max interval
Ensure that we don't query too far in the history, which slows things
down.

Change-Id: Ia77aa522f7f4c5d43629d51bb9a51a49fab6fa14
2021-06-10 17:06:33 +00:00
Fadila Khadar
21731ff8d0 satellite/metabase: add expires_at column to segments
Change-Id: If798198d87edb68cda8859ce2277fdfbd6f8380b
2021-06-07 19:11:20 +00:00
Egon Elbre
910eec8eee satellite/metainfo: remove MetabaseDB interface
Currently the interface is not useful. When we need to vary the
implementation for testing purposes we can introduce a local interface
for the service/chore that needs it, rather than using the large api.

Unfortunately, this requires adding a cleanup callback for tests, there
might be a better solution to this problem.

Change-Id: I079fe4dbe297b0ae08c10081a1cea4dfbc277682
2021-05-13 13:22:14 +00:00
Egon Elbre
0858c3797a satellite/{metabase,satellitedb}: deduplicate AS OF SYSTEM TIME code
Currently we were duplicating code for AS OF SYSTEM TIME in several
places. This replaces the code with using a method on
dbutil.Implementation.

As a consequence it's more useful to use a shorter name for
implementation - 'impl' should be sufficiently clear in the context.

Similarly, using AsOfSystemInterval and AsOfSystemTime to distinguish
between the two modes is useful and slightly shorter without causing
confusion.

Change-Id: Idefe55528efa758b6176591017b6572a8d443e3d
2021-05-11 12:40:36 +03:00
Egon Elbre
6e6051b172 satellite/metabase: drop alias migration code
We have migrated all of the satellites and we shouldn't keep dead-code
around.

Change-Id: I539d6766cfafa2f278ff7767ceb2d39f6777ace3
2021-05-07 12:13:28 +03:00
Egon Elbre
2af7e4ef26 satellite/metabase/metaloop: use database time
The system and database time may drift. We should use database time for
absolute "as of system time" to ensure that it's not newer than the
current database time. When the "as of system time" is in the future,
then the query will fail.

Change-Id: I5423f6aaad966ca03a76b5ff805bfba932e44a51
2021-05-07 09:08:04 +00:00
Ethan
dba9321483 satellite/metabase: Remove pending_index
Remove pending_index until more performance testing can be done.

Change-Id: I31d29b1f37998eb62aee8bd42c7dcb79859446d2
2021-05-04 09:49:39 -04:00
Fadila Khadar
5af898fcc9 satellite/metabase: add migration step for index on pending objects
Add an index on (project_id, bucket_name) for objects with pending status

Change-Id: Ibd3c949a99e49e8e6bb884564331ec16f9184942
2021-04-28 09:54:10 +00:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Egon Elbre
267506bb20 satellite/metabase: move package one level higher
metabase has become a central concept and it's more suitable for it to
be directly nested under satellite rather than being part of metainfo.

metainfo is going to be the "endpoint" logic for handling requests.

Change-Id: I53770d6761ac1e9a1283b5aa68f471b21e784198
2021-04-21 15:54:22 +03:00