Commit Graph

708 Commits

Author SHA1 Message Date
Moby von Briesen
04294e3e1c satellite/payments: Update coupon defaults for free tier
The new default promotional coupon is $10/month, and doesn't expire.

This change also migrates the coupon.duration column over to the new
coupon.billing_periods, and switches to rely completely on
billing_periods.

Change-Id: Ic3341e9fa4040449bab5e66ca4ee2640b095cf3d
2021-04-05 10:12:26 -04:00
Jeff Wendling
2462ac5917 metainfo+satellite/satellitedb: use HasBucket when possible
Change-Id: I8318f5581eef559f0dfbf20d07ea1fe36f81ce87
2021-04-02 13:20:36 -04:00
Cameron Ayer
523dd823db satellite/satellitedb: remove references to total_uptime_count and uptime_success_count columns
These columns are no longer used

Change-Id: I0400544e4e98752c9b90662c037ab3a1884c234b
2021-04-02 13:22:12 +00:00
Michał Niewrzał
71d0adf90b satellite/metainfo: avoid error on inserting existing attribution
Currently we can have an error about duplicated entry while inserting into value_attributions table. This change is changing simple insert into insert that is doing nothing on conflict.

Change-Id: I3efd8dc0b63115e8e2ed8f4196ccf969ee942295
2021-04-02 08:18:17 +00:00
Moby von Briesen
c334fd090e satellite/payments: Convert coupon duration to be nullable
* Add a nullable billing_periods column in the coupons table
* Add nullable billing_periods column to the currently unused
coupon_codes table
* Drop the duration column from the coupon_codes table
* Replace duration config type so that the default promotional coupon
can be configured to never expire

Zero downtime migration plan:
* Add billing_periods column to coupons and coupon_codes tables (this change)
* After one release, remove all references to the old duration column,
replacing with references to billing_periods. At this point, we can also
change the defult promotional coupon to never expire and migrate over
values from the old duration column.
* After another release, drop the duration column.

Change-Id: I374e8dc9fab9f81b4a5bc681771955662d4c007a
2021-04-01 16:28:23 +02:00
Jeff Wendling
a65aecfd98 compensation: always generate invoices for every node
instead of only generating invoices for nodes that had some
activity, we generate it for every node so that we can find
and pay terminal nodes that did not meet thresholds before
we recognized them as terminal.

Change-Id: Ibb3433e1b35f1ddcfbe292c034238c9fa1b66c44
2021-03-29 14:15:45 +00:00
Cameron Ayer
05f8d2d0b1 satellite/satellitedb: filter offline suspended nodes from selection
Change-Id: I5a6f413453332238d579a7bf50eb30e9156f96c2
2021-03-27 23:36:46 +00:00
Cameron Ayer
1a51049ac0 satellite/{overlay,satellitedb}: add flag to toggle suspending nodes for offline audits
This change introduces a new config flag,
--overlay.audit-history.offline-suspension-enabled,
to toggle suspending nodes for offline audits.

If the flag is set to true, nodes will be suspended if they meet the
requirements.

If the flag is false, nodes will not be suspended. If they are already
suspended and/or under review, these will be cleared.

Change-Id: Ibeba759c42d6e504f6b7598120d4fd4dab85ca74
2021-03-27 16:28:27 +00:00
Cameron Ayer
eb44dc21b4 satellite/satellitedb: select stray nodes for DQ in separate tx from update
Previously we would select a limited number of nodes for DQ in a
CTE and run the update on that set in a single transaction. This
could lead to locking on the table, so instead we select and update
in separate transactions.

Change-Id: I1e802c0845e829eeadcee4fa382f58462515fdb1
2021-03-27 00:00:23 +00:00
Moby von Briesen
27bcb46718 satellite/console: change default project and usage limits
This is one step for implementing the free tier:
* Change the default project limit from 10 to 3
* Move storage and bandwidth project usage limits from the metainfo
package to the console package (otherwise there is a cyclical
dependency, and metainfo doesn't use these values anyway)
* Change the default storage usage limit per project from 500gb to 50gb
* Change the default bandwidth usage limit per project from 500gb to 50gb
* Migrate the database so that old users and projects continue to have
the old defaults (10 projects/500gb usage)

Change-Id: Ice9ee6a738bc6410da18c336c672d3fcd0cab1b9
2021-03-24 14:24:27 +00:00
Michał Niewrzał
237782813b Merge remote-tracking branch 'origin/multipart-upload'
Change-Id: If6c5a450b238adab55d1e0dea67d01e5f5768a9f
2021-03-23 09:44:49 +01:00
Cameron Ayer
37602d86d3 satellite/satellitedb: drop columns uptime_reputation_alpha and uptime_reputation_beta
Change-Id: I1e0c5f2f96a920070afb76851da98208654b02cf
2021-03-22 14:33:17 -04:00
Cameron Ayer
2607b16070 satellite/{overlay/straynodes,satellitedb}: rework DQNodesLastSeenBefore to return DQd node IDs and last contact successes
We would like to log Node IDs and last contact successes of nodes DQd
in this manner. We would also like to avoid returning an unbounded list
of items from the db. Therefore we change the query to select a limited
number of nodes that meet the DQ conditions and iterate until 0 rows are
returned. Each column of the query is already indexed.

Change-Id: Iaec2d9b56e7202b7c2028ba21750d40c8dd506ee
2021-03-22 13:01:30 -04:00
Moby von Briesen
78e132cb74 satellite/satellitedb: Add coupon_codes table and allow optional link to coupons table
The coupon_codes table will allow for administrators to create new promo
codes associated with coupon information (amount, duration, etc...).

A user will be able to enter a promo code (aka coupon code) in order to
apply a new coupon to their account. The coupon in the coupons table is
linked to the template defined in the coupon_codes table.

Change-Id: I50e49fa92afbc6aa9d01d8a895c069efb59e472b
2021-03-19 20:46:28 +00:00
Michał Niewrzał
fa083a7f05 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ib5ce5965b77b81c254d08c27ab30c7eccefbd4c6
2021-03-17 15:37:17 +01:00
Egon Elbre
9491df76e7 satellite/satellitedb: optimize StoragenodeAccounting.SaveRollup
Change-Id: I758049872b4ea1ae22657dd9fcb47c228468b1d0
2021-03-16 18:11:40 +02:00
Cameron Ayer
8138939a2a satellite/satellitedb: remove bad migration step
Migration step 148 will cause errors because we missed some
references to the columns being dropped. Removing the step
altogether causes problems with backwards compatibility tests
because the change already exists in the latest release tag.
To circumvent, we change v148 to an empty migration.

Add methods FindTable and RemoveColumn in private/dbutil/dbschema

Change-Id: Ia527e95b88a88c5dc82800928ce6f8cfb879e334
2021-03-10 14:36:52 -05:00
Michał Niewrzał
67e26aafcd Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I9b183323cb470185be22f7c648bb76917d2e6fca
2021-03-10 08:53:38 +01:00
Jeff Wendling
376547c33c satellite/compensation: smaller txns for RecordPeriod
cockroach is having problems with huge transactions and
having them complete before timeouts or whatever, so
do smaller transactions.

because we can have partial recording of payments which
are not unique, we have to do a thing where we read and
check if it already exists before writing. this is not
concurrency safe.

Change-Id: Ia7d59499a43ce6d70cb2a23754edbdd1b643ef1a
2021-03-02 20:14:25 +00:00
Cameron Ayer
aeac6264cd sallite/satellitedb: add metric stray_nodes_dq_count
Add metric so we can see how many nodes are DQd due to
this.

Change-Id: Ie4bdd1375fb9bd948af14fed9a2962b783b6a526
2021-03-01 21:06:36 +00:00
Natalie Villasana
856db68fd9 satellite/gracefulexit: extend GE data cleanup to include exit_progress
The new 'consistency ge-cleanup-orphaned-data' cli command deleted
orphaned transfer queue items, but not entries in the
graceful_exit_progress table. This will delete orphaned entries
from the exit progress table too.

Change-Id: I5f927aac1f258490678deaf179be92ccfe10fcd8
2021-03-01 15:52:43 +00:00
Cameron Ayer
411a7ad0bc satellite/satellitedb: drop uptime_reputation_alpha and uptime_reputation_beta from nodes table
Change-Id: Ib46e783bf1a5c036394b4cac281382d0380bb1be
2021-03-01 15:30:51 +00:00
Michał Niewrzał
1af9400a23 satellite/satellitedb/dbx: remove unused methods
Turns out that many methods generated with dbx where not used at all.
Lets remove them.

As a next step we can think about dropping tables like:
* user_credit
* offer

Change-Id: Id6cda81a701348db2a6b8b26daa22ae9c4f87cb4
2021-02-25 13:50:50 +00:00
Michał Niewrzał
d995fb497f Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I367da03351ab80f7343332420490dde9282aa47a
2021-02-23 12:31:31 +01:00
Egon Elbre
0178ec7771 satellite/satellitedb: clearer testdata version parsing
Change-Id: I93a23625b30472e826b962dab31ff3e1081b1b8f
2021-02-23 10:22:19 +02:00
Egon Elbre
32afbd0d15 satellite/satellitedb: faster test database setup
Pregenerate the database schema we should use for most tests.
Currently, Cockroach is slow with regards to migration and it's
better if it happens in as few transactions as possible.

This reduces test time from ~21min to ~15min.

Change-Id: Ife8117053e6b9ecf3c93fe63677edf15d4d7c254
2021-02-22 21:13:00 +02:00
Cameron Ayer
549033f2e6 satellite/satellitedb: don't include DQd and exited nodes in DQStrayNodes
Don't update DQ time of already DQd nodes. Don't DQ nodes who exited.

Change-Id: I4528a9ba9f8e278987165ad337a9b34dadb9788b
2021-02-19 15:12:30 -05:00
Egon Elbre
1137620baf satellite/satellitedb: move tests to their domains
Testing interfaces is slightly clearer when it's in the package needing
the database rather than each individual implementation.

Change-Id: I10334c214a205f7e510b939b4359a2214c4e060a
2021-02-19 17:29:15 +02:00
Egon Elbre
1cb6376daa satellite/metainfo: remove BucketsDB.ListAllBuckets
The ListAllBuckets implementation was buggy, remove it altogether.

Change-Id: Id457ba5f4d793156af3fc2071f74ce1be17ba804
2021-02-19 10:59:41 +02:00
JT Olio
b2ed7edd30 cmd/satellite: restore-trash parallel workers
Change-Id: Ic7466b21c20bda334e7ba4268a494e96b6528ac1
2021-02-18 19:11:19 +02:00
JT Olio
3ae3389ddc cmd/satellite: restore-trash command
Change-Id: I80fc932c12147692d49cde277784871ac611fcad
2021-02-18 09:19:22 -07:00
Michał Niewrzał
12402eb729 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I38adf8218c1415c7ea1910f8bd6bed13544b0f03
2021-02-17 08:50:38 +01:00
Malcolm Bouzi
4b2e46a0c9 satellite/satellitedb: add employee size column to users
Change-Id: I21f5904331f0ceb92f494729c22a52c256a69163
2021-02-12 09:15:15 -05:00
Egon Elbre
63c7f8b7fc satellitedb/satellitedbtest: creating a database shouldn't auto-migrate
Some tests need to control the migration progress manually.

Change-Id: I776c69b6d56dc35c7cb88688c4b827d6bba4b7ac
2021-02-11 14:21:49 +02:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Yaroslav Vorobiov
966535e9de {storagenode,satellite}/nodeoperator: add wallet features
Change-Id: Iac7eb40a52b8fddcc573aebaad2e3a30a10cded9
2021-02-08 22:09:45 +02:00
Yingrong Zhao
3b49d3cddf satellite: remove referral program related code
This PR removes all back-end related referral program code including the
marketing portal.

We will have a separate PR for front-end code and database migration to
drop `offers` and `usercredits` table

Change-Id: If59f952cddfe0558a7dc03a0eac7cc1081517f88
2021-02-08 13:52:50 +00:00
Jeff Wendling
e114cfe86d satellite/satellitedb: fix broken query
Change-Id: I6d412a673d75264bf9751c6f15b1fb0ab94e1394
2021-02-04 16:00:51 -05:00
Malcolm Bouzi
db3a3088f9
satellite/satellitedb: add professional user fields to db interface (#4034) 2021-02-04 10:00:15 -05:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
JT Olio
c8e42df2f8 satellitedb: reorder migrations 140-144
These changes are independently tracked on
https://github.com/storj/storj/tree/jt/migration-reorder

The point of this is to make the distributed column
migration, needed for SNO invoice generation, the very
next one, so we can release it as a point release.

Change-Id: I26e1c03629c4f079b9ad12485e2b71a715d82b3b
2021-02-03 18:28:42 +00:00
Ethan
9506b67ca2 satellite/projectaccounting: Improve performance of ProjectAccounting.getBuckets
Limit bucket name lookup to date range of the calling methods since we only need distinct bucket names for that time period.
Adds new index and removes an index specific to project ID since it is no longer needed.

Change-Id: Ic07bbfb1c32280e0c0e39f8da020b284e1e5d974
2021-02-03 14:05:12 +00:00
Cameron Ayer
a17934cb51 satellite/satellitedb: remove reference to uptime counts
Change-Id: I26ac540b720a8ba5d6ca44526900228352dcaf4e
2021-02-02 14:51:27 -05:00
Jeff Wendling
759bdd6794 satellite/compensation: add total-paid and total-distributed to invoices
Change-Id: Id4414867917cbf8aad77795f764d6381e88d9a34
2021-02-02 18:14:31 +00:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Ivan Fraixedes
cc0d88f9c3
satellite/satellitedb: Fix GE flaky test
Fix an issue due to copy-paste problem that made that the Graceful Exit
test to be flaky.

The test uses a time created at the beginning of the test for avoiding
to get undeterministic time differences due to the fact of the response
time variation by the DB queries, however some part of the test were
using a current time rather than this base time, so they have been
addressed.

Change-Id: I4786f06209e041269875c07798a44c2850478438
2021-02-02 13:24:42 +01:00
Ivan Fraixedes
d93944c57b satellite/orders: Delete unused methods & DB tables
Delete satellite order methods and DB tables which aren't used anymore
after we have done a refactoring on the orders to stuck bucket
information in the orders' encrypted metadata.

There are also configuration parameters and a satellite chore that
aren't needed anymore after the orders refactoring.

Change-Id: Ida3682b95921df70792284b42c96d2508bf8ca9c
2021-02-01 18:01:29 +00:00
Ivan Fraixedes
076804eac9 cmd/satellite: Add command for GE data cleanup
Add a command to the satellite for cleaning up the Graceful Exit (a.k.a
GE) transfer queue items of nodes that have exited.

The commit adds to the GE satellite DB a couple of new methods, and its
corresponding test, for performing the operations of the new command.

Change-Id: I29a572a59689d63b24990ac13c52e76d65aaa917
2021-02-01 17:30:58 +00:00
Jeff Wendling
1cf3d89a56 satellite/satellitedb: add distributed column and migration
using redash i manually checked that the only times the sum of
the payments does not match the paid column is for 2020-12 and
if it does not match then there are no payments.

Change-Id: I71ce0571de7e38e21548d7d6757b25abc3bfa781
2021-02-01 16:33:14 +00:00
Natalie Villasana
91bd4191dd satellite/accounting: add rollup archiver chore
The rollup archiver chore moves bucket bandwidth rollups and
storagenode rollups that are older than a given duration
to two new archive tables.

Change-Id: I1626a3742ad4271bc744fbcefa6355a29d49c6a5
2021-02-01 09:29:54 -05:00