Commit Graph

1873 Commits

Author SHA1 Message Date
Kaloyan Raev
1156d8b4ed mod: update drpc to v0.0.20
Change-Id: I0ac6ff9bebe5b16d4fb63b30242c87c899e7d358
2021-03-24 19:06:56 +02:00
Moby von Briesen
27bcb46718 satellite/console: change default project and usage limits
This is one step for implementing the free tier:
* Change the default project limit from 10 to 3
* Move storage and bandwidth project usage limits from the metainfo
package to the console package (otherwise there is a cyclical
dependency, and metainfo doesn't use these values anyway)
* Change the default storage usage limit per project from 500gb to 50gb
* Change the default bandwidth usage limit per project from 500gb to 50gb
* Migrate the database so that old users and projects continue to have
the old defaults (10 projects/500gb usage)

Change-Id: Ice9ee6a738bc6410da18c336c672d3fcd0cab1b9
2021-03-24 14:24:27 +00:00
Vitalii Shpital
3e37d1e71c satellite/console: delete api key by name and project id endpoint
WHAT:
new endpoint to be able to delete apiKey/accessGrant by name and project id

WHY:
it will be called to delete special pregenerated access grant which will be used to generate gateway credentials for file browser component or bucket management

Change-Id: I7467ebaab27a7da33efd062536c6da41e6ed4c30
2021-03-23 20:21:07 +00:00
Egon Elbre
f19ef4afe5 satellite/metainfo/metaloop: move loop to a separate package
Change-Id: I94c931a27c1af6062185ec62688624ec02050f11
2021-03-23 15:37:34 +00:00
Egon Elbre
54c2ace483 mod: update drpc to v0.0.19
Change-Id: Ia2e0e0a371368700c596ce89e24748ea4b5c016a
2021-03-23 17:13:28 +02:00
Michał Niewrzał
9d52451464 satellite/metainfo/expireddeletion: add config value to set list limit
We want to control how many expired objects we will query at once.

Change-Id: Iba7c79a6236c6b720b53bb8f078d55514690d8e5
2021-03-23 09:51:56 +00:00
Michał Niewrzał
237782813b Merge remote-tracking branch 'origin/multipart-upload'
Change-Id: If6c5a450b238adab55d1e0dea67d01e5f5768a9f
2021-03-23 09:44:49 +01:00
Moby von Briesen
d2b365bf9f satellite/payments: Disable paywall (free tier)
We are implementing the free tier, which will give all new users 3
projects, 50gb storage, and 50gb bandwidth per project. All users will
receive a recurring coupon to cover this amount of usage.

With the free tier, we no longer need a paywall. Users will not need to
enter a payment method unless they want to increase their project or
usage limits.

Change-Id: If3b026e91858e5f557a2758e366616cecc8f21c7
2021-03-22 17:11:20 -04:00
Cameron Ayer
37602d86d3 satellite/satellitedb: drop columns uptime_reputation_alpha and uptime_reputation_beta
Change-Id: I1e0c5f2f96a920070afb76851da98208654b02cf
2021-03-22 14:33:17 -04:00
Cameron Ayer
864ad70fe2 satellite/overlay/straynodes: set --stray-nodes.enable-dq release default to true
Since we will enable this on all satellites, just set default to true

Change-Id: Ibc86a0afd0b0f57e86bd067abb9cdf06c295a467
2021-03-22 17:25:09 +00:00
Cameron Ayer
2607b16070 satellite/{overlay/straynodes,satellitedb}: rework DQNodesLastSeenBefore to return DQd node IDs and last contact successes
We would like to log Node IDs and last contact successes of nodes DQd
in this manner. We would also like to avoid returning an unbounded list
of items from the db. Therefore we change the query to select a limited
number of nodes that meet the DQ conditions and iterate until 0 rows are
returned. Each column of the query is already indexed.

Change-Id: Iaec2d9b56e7202b7c2028ba21750d40c8dd506ee
2021-03-22 13:01:30 -04:00
Fadila Khadar
a8b5f8f990 satellite/metainfo/metabase: select then delete expired objects
We were deleting expired objects by directly executing a delete query.
With this change, we first select the objects to be deleted and then
delete them (as recommended by cockroachdb for deleting using a non indexed
column).

Change-Id: Ied150fbdc7031a343a74e0b9dab316598188ef66
2021-03-22 17:39:13 +01:00
Michał Niewrzał
27ae0d1f15 satellite/metainfo/metabase: add NewRedundancy parameter for UpdateSegmentPieces method
At some point we might try to change original segment RS values and set Pieces according to the new values. This change adds add NewRedundancy parameter for UpdateSegmentPieces method to give ability to do that. As a part of change NewPieces are validated against NewRedundancy.

Change-Id: I8ea531c9060b5cd283d3bf4f6e4c320099dd5576
2021-03-22 08:12:56 +00:00
Moby von Briesen
78e132cb74 satellite/satellitedb: Add coupon_codes table and allow optional link to coupons table
The coupon_codes table will allow for administrators to create new promo
codes associated with coupon information (amount, duration, etc...).

A user will be able to enter a promo code (aka coupon code) in order to
apply a new coupon to their account. The coupon in the coupons table is
linked to the template defined in the coupon_codes table.

Change-Id: I50e49fa92afbc6aa9d01d8a895c069efb59e472b
2021-03-19 20:46:28 +00:00
Fadila Khadar
e613c641d7 satellite/metainfo/metabase: benchmark expired deletion
Change-Id: Icc2e8227b8a96002563ff69f3e9cd8e71369f0d1
2021-03-19 11:30:00 +00:00
Michał Niewrzał
1c02158620 satellite/metainfo/metabase: add RepairedAt to RawSegment
Change-Id: I3f8bbed0f815f8394fc6ab16a914cae276fbab56
2021-03-19 11:06:59 +00:00
Vitalii Shpital
c3ae122aa7 web/satellite: enter passphrase step for objects page
WHAT:
enter passphrase step for users who has already created passphrase

WHY:
to let users proceed to upload step

Change-Id: I084aec5b863981978cf190f99ee95154fbed9aab
2021-03-19 10:59:32 +00:00
Cameron Ayer
a04495713d satellite/audit: add missing logs for audit failure conditions
Among other conditions, nodes fail audits by returning incorrect
data and by reaching the max reverify count, but we weren't logging
these events. This commit adds the missing logs.

Change-Id: I80749a7e95e8cb97bc8dd7dac1e523e223114b7f
2021-03-18 17:33:11 +00:00
Kaloyan Raev
be8a70309d satellite/metainfo/metabase: rename etag column to encrypted_etag
Change-Id: I79662d9a2cec2da6ea9bba250f0e71978119558d
2021-03-18 18:34:29 +02:00
Ivan Fraixedes
4c1098e571 Redis: Update Redis package to last major version
Update the Redis dependency to use the last major production version.
The last version accepts a context parameter in all the network methods
so it allows us to pass it through them.

Change-Id: I34121b2ec3c2728602115c724933ad24c9e6e4fd
2021-03-18 14:19:49 +00:00
Kaloyan Raev
36bc17ef5b satellite/metainfo/metabase: add etag column to segments table
We will use it keep track of ETag for object parts.

Change-Id: Ib610397afd44d71a1b5fe4301940de6f36ba39ae
2021-03-18 10:40:55 +00:00
Michał Niewrzał
fa083a7f05 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ib5ce5965b77b81c254d08c27ab30c7eccefbd4c6
2021-03-17 15:37:17 +01:00
Egon Elbre
9491df76e7 satellite/satellitedb: optimize StoragenodeAccounting.SaveRollup
Change-Id: I758049872b4ea1ae22657dd9fcb47c228468b1d0
2021-03-16 18:11:40 +02:00
Vitalii Shpital
6a553ec9c5 web/satellite: change banner for beta satellites with URLs
WHAT:
beta satellite top banner's copy is changed to include support/feedback URLs

WHY:
so users using our beta satellite will be able to report feedback somewhere

Change-Id: Ibc349c8b3354b577275fcf1d2b75bfdd267729d9
2021-03-15 17:12:07 +00:00
Isaac Hess
1de6315b83 metainfo: Stub DownloadObject metainfo endpoint
This is the stub "not implemented" endpoint for the new DownloadObject
RPC.

Change-Id: Ifa724e20d89d839f2b6968d6283bc66a3bab8781
2021-03-15 10:52:57 -06:00
Michał Niewrzał
65957c7525 satellite/metainfo/expireddeletion: temporarily log errors instead failing whole system
At the moment we are trying to optimize deletion queries but its hard to verify deletion performance. Until we are sure that the queries are good we will just log errors instead shutting down whole satellite core.

Change-Id: I5625251d4518c35f0d46d6bf37b2f3ea7950675e
2021-03-15 16:00:20 +01:00
Kaloyan Raev
c4f0289904 satellite/metainfo: ListSegments returns CreatedAt in response
If a non-nil value is read from created_at column of the segments table,
it will be set to the CreatedAt field if SegmentListItem.

Change-Id: I02691d8e11fad12c1b0e4c443bdebb568016ffe3
2021-03-15 11:45:25 +00:00
Kaloyan Raev
6e661da0a0 satellite/metainfo/metabase: read created_at from DB
All SQL queries for reading segments are updated to read the created_at
column where appropriate.

Change-Id: Icd7c7672fa71e992673078598b28229bb898c728
2021-03-15 11:37:14 +00:00
Michał Niewrzał
887f3b04e1 satellite/metainfo/metabase: pieces vs redundancy scheme validation while commiting segment
Additional validation for pieces based on RS.

Change-Id: Ib25bfc0e38ca78e2c3194a3223d59257489049fb
2021-03-11 16:12:52 +00:00
Fadila Khadar
018b0684b8 satellite/metabase: different queries for delete_expired
use custom queries for pg and crdb when deleting expired objects.

Change-Id: I2e230b3b34d5128c877ed8f8f77d83b9a935df42
2021-03-11 09:54:03 +00:00
Kaloyan Raev
bdb33b381b satellite/metainfo/metabase: add created_at and repaired_at to segments
The created_at columns is first added without a default value to avoid
setting the current time to existing segments.

Change-Id: Ic2fe3da238422e2949e6f3016fbac04eb89ba037
2021-03-10 19:59:47 +00:00
Cameron Ayer
8138939a2a satellite/satellitedb: remove bad migration step
Migration step 148 will cause errors because we missed some
references to the columns being dropped. Removing the step
altogether causes problems with backwards compatibility tests
because the change already exists in the latest release tag.
To circumvent, we change v148 to an empty migration.

Add methods FindTable and RemoveColumn in private/dbutil/dbschema

Change-Id: Ia527e95b88a88c5dc82800928ce6f8cfb879e334
2021-03-10 14:36:52 -05:00
Egon Elbre
5e954ad487 satellite/metainfo/metabase: optimize ConvertAliasesToPieces
old time/op    new time/op    delta
    7.56µs ± 5%    4.93µs ± 2%  -34.75%  (p=0.000 n=5+15)

  old alloc/op   new alloc/op   delta
    6.86kB ± 0%    3.85kB ± 0%  -43.87%  (p=0.000 n=5+18)

  old allocs/op  new allocs/op  delta
      19.0 ± 0%      17.0 ± 0%  -10.53%  (p=0.000 n=5+18)

Change-Id: Iedf24087766b3bd90934f2daa7ac186c3503a341
2021-03-10 08:39:23 +00:00
Michał Niewrzał
67e26aafcd Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I9b183323cb470185be22f7c648bb76917d2e6fca
2021-03-10 08:53:38 +01:00
Ivan Fraixedes
84b844a2a7 redis-server: Move testing type to specific testing pkg
Move a specific interface & types used for testing to be a private
subpackage with a name that clearly identifies it for testing purpose.

Change-Id: I646cf3b6f0a3b518a6f9a125998dc5a02df02db6
2021-03-10 06:09:46 +00:00
Egon Elbre
461d2cd233 satellite/metainfo{,/metabase}: add GetStreamPieceCountByNodeID
GetObjectIPs endpoint was loading all the segment information rather
than only what it needed.

Change-Id: I3f684cb7020b1771d78b5d43c6862d03ada464c0
2021-03-08 16:43:17 +02:00
Egon Elbre
981f1ca449 satellite/metainfo: use ListStreamPositions
ListSegments loads all the segment data into memory, however this can
add up to a lot of data with inline segments and large objects.

Change-Id: I037738f0e70b810ecbea7d83b00ea7ca9eb90c7a
2021-03-08 14:10:33 +02:00
Egon Elbre
5bc0f3bf23 satellite/metainfo/metabase: add ListStreamPositions
Change-Id: I0b5a5803be8987e7cafbcc80460217d9eb8f0238
2021-03-08 14:09:22 +02:00
Michał Niewrzał
903cb172c2 satellite/metainfo/metabase: add iteration with prefix benchmark
Change-Id: I7af77cdf0f11e28ad946f2eaff4a9aa8fe5d0bec
2021-03-08 09:43:38 +01:00
Egon Elbre
b5e95ec79f satellite/metainfo/metabase: add extra validation to IterateLoopSegments
Change-Id: I377bf9f661dcff6ba29d10f2909eb59d6d361c5d
2021-03-06 11:19:36 +02:00
Egon Elbre
150d75da28 satellite/metainfo/metabase: remove old TODO notes
Change-Id: Id216eb8cc5014652cce77a0f8f17ac66715bd46f
2021-03-05 15:16:46 +02:00
Michał Niewrzał
8de1116eeb satelite/metainfo: remove unused method from metainfo loop
IterateDatabase method was used by zombie segment reaper which is removed for multipart implementation.

Change-Id: I93e1294236612d6d82b2ab57053bb84e653f72b4
2021-03-04 12:48:27 +00:00
Ethan
7af245c3e7 satellite/payments: De-shadow customer page var in stripe mock
Fixes infinite loop when there are more records than the limit.

Change-Id: Ib61a8fcad6d08f4e0f1c5715d710acec607004b3
2021-03-03 20:51:43 -05:00
Egon Elbre
ec67413776 satellite/metainfo: add as of system time to object iteration
Change-Id: Idc1d5c7d983f331c60f5b75c5a977a25e197faf9
2021-03-03 13:39:47 +02:00
Egon Elbre
b0b7b81105 satellite/metainfo/metabase: streams iterator
Iterate over streams/segments rather than loading all of them into
memory. This reduces the memory overhead of metainfo loop.

Change-Id: I9e98ab98f0d5f6e80668677269b62d6549526e57
2021-03-03 13:16:49 +02:00
Jeff Wendling
376547c33c satellite/compensation: smaller txns for RecordPeriod
cockroach is having problems with huge transactions and
having them complete before timeouts or whatever, so
do smaller transactions.

because we can have partial recording of payments which
are not unique, we have to do a thing where we read and
check if it already exists before writing. this is not
concurrency safe.

Change-Id: Ia7d59499a43ce6d70cb2a23754edbdd1b643ef1a
2021-03-02 20:14:25 +00:00
Cameron Ayer
a44974a2f9 satellite/audit: fix pointless containment deletions
Previously if node was not found in containment, it was
given the status, 'skipped'.

We later try to delete skipped nodes from containment.

To fix this, add a new status called 'remove' to differentiate
nodes which should be skipped and nodes which should be deleted.

Change-Id: Ic09e62dc9723c89d0c9f968ce68c039114a9d74e
2021-03-02 13:40:18 -05:00
Michał Niewrzał
c51ea68ad3 satellite/metainfo/metabase: reduce number of fields for LoopSegmentEntry
For metainfo loop we need only some of Segment fields. By removing some of them we will reduce memory consumption during loop.

Change-Id: I4af8baab58f7de8ddf5e142380180bb70b1b442d
2021-03-02 15:04:54 +01:00
Natalie Villasana
c290e5ac9a satellite/orders: decrease FlushBatchSize default to 1000
The previous default FlushBatchSize of 10000 was causing major
slow down in select and insert statements on bucket_bandwidth_rollups.
We saw on the saltlake satellite that a FlushBatchSize of 1000 helped
reduce contention and query latency.

Change-Id: Ib95e73482219bc5aedc11925b1849fa5999774ba
2021-03-02 14:00:48 +00:00
Michał Niewrzał
c223c2d845 satellite/metainfo/metabase: rename ListObjectsSegments to ListLoopSegmentEntries
This method will be used only with metainfo loop and we need to customize query to consume less memory.

Change-Id: Iaa97392f483c5df5609d501b3847b80eb1ea2583
2021-03-02 12:24:59 +00:00