Commit Graph

262 Commits

Author SHA1 Message Date
Malcolm Bouzi
c30c7def44 satellite/{console,payments},web/satellite: save signup promo code to newly registered users
Users signing up through a url containing a promo code will have that code applied to their stripe account instead of the free tier coupon.

Change-Id: I071041b0934648ef3f5bdb05b6ec97c400f89ae4
2022-01-21 15:02:22 +00:00
paul cannon
171f908d8b cmd/satellite: rename monkit stat with '_', not '-'
Change-Id: Idb7d2971fbef1145f281b728a492e40e04add155
2022-01-06 02:30:30 +00:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
Yingrong Zhao
d441c8da15 satellite: use segment count for billing
Change-Id: Iafaf41cb6cde4cb2abdac3f75ce0fead838cb758
2021-10-27 19:21:50 +00:00
paul cannon
1951450c57 cmd/satellite: add register-lost-segments command
Usage: from a host for the affected satellite, issue this command
and supply the number of segments that have been permanently lost. For
example, to indicate 2 segments permanently lost:

    satellite register-lost-segments 2

If the unthinkable happens and this is necessary to use, it is presumed
that we will also remove the non-recoverable segments from the metainfo
db, so they will not continue to appear as 'temporarily unavailable' to
the repair checker.

The influxQL query for this will probably look something like:

    SELECT sum(total) AS "sum_total"
    FROM "v3_stats_new"."autogen"."lost-segments"
    WHERE time > :dashboardTime: AND "scope" = 'segment-durability'
    GROUP BY time(:interval:), "application", "instance"
    FILL(null)

Or use the Flux language in order to get the benefit of the
cumulativeSum function:

    from(bucket: "v3_stats_new/autogen")
      |> range(start: dashboardTime)
      |> filter(fn: (r) => r._measurement == "lost-segments"
                           and r.scope == "segment-durability"
                           and r._field == "total")
      |> group(columns: ["application", "instance"])
      |> cumulativeSum()

Change-Id: I73d364937705aa815af7520ab79c00bc2aea09f6
2021-10-26 15:03:52 +00:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
dlamarmorgan
4bbf667ad1 satellite/{satellitedb,attribution,console}: value attribution changes that add userAgent field to buckets table and all tables that have partner_id
Change-Id: I36a13bb651b86bfc14fe5a0a2258f719e6cd2b48
2021-10-18 13:56:19 +00:00
Egon Elbre
c54bcd63a1 Makefile: bump node to v16
Change-Id: I0587168f373230710c7b50b9aefd34718f4a5b3e
2021-10-14 12:46:28 +00:00
Jeremy Wharton
3b751a35c5 satellite/{payments,satellitedb}: Remove custom coupon implementation
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.

Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
2021-10-11 19:47:00 +00:00
Yaroslav Vorobiov
469ae72c19 satellite/repair: update audit records during repair
Change-Id: I788b2096968f043601aba6502a2e4e784f1f02a0
2021-09-24 00:48:13 +00:00
Jeremy Wharton
eb11899acb cmd/satellite: Remove billing 'check-paid-tier' command
Now that the command has been run on all production satellites
(US1, AP1, EU1), we should not need it again.

Change-Id: I25a4ffb03a7172445d90a04ec539be36c4eb2c8e
2021-08-26 22:02:16 +00:00
Moby von Briesen
ef9a5210a4 satellite/payments: Add command to ensure free tier coupons
This command is intended to be run as part of invoice generation - it
iterates over Stripe customers, and applies the free tier coupon to any
customer who doesn't already have a coupon.

This way, we can ensure that all customers have at least the free tier
coupon before and after invoice generation, in case a different coupon
has expired.

Change-Id: I33a4aff9174049f9e051de53ef65298ca65ed688
2021-08-18 13:48:44 +00:00
Artur M. Wolff
2cd68bf4fb private/lrucache: import from common
Change-Id: Ia1f43d0440fef21122b071b05da59b4cf2689d6c
2021-08-16 10:04:32 +00:00
Yingrong Zhao
f8914ccce0 satellite/{repair, overlay}: use reputation store in repair
Change-Id: I48db9e68f48239d48621ccc77d33618ecb83ce1a
2021-07-28 13:22:05 -04:00
Moby von Briesen
8f072bdeee cmd/satellite: Skip non-existing users in paid tier conversion
There are some users in our QA satellite which are no longer in Stripe,
and there are some users in Stripe which are not on our QA satellite.
This change allows us to test the paid tier conversion script in QA
despite these problems.

Change-Id: If94c9e882327841d1fd294d75fd302e6a7feee41
2021-07-27 12:53:58 -04:00
Michał Niewrzał
27a714e8b0 satellite/accounting/tally: use objects iterator instead metaloop
Bucket tally calculation will be removed from metaloop and will
use metabase objects iterator directly.

At the moment only bucket tally needs objects so it make no sense
to implement separate objects loop.

Change-Id: Iee60059fc8b9a1bf64d01cafe9659b69b0e27eb1
2021-07-20 15:52:18 +00:00
Moby von Briesen
5870502589 cmd/satellite: Add billing command for converting customers to paid tier
We have implemented the paid tier, but it currently only handles new
users entering paid tier. It does not convert users who have already
added a credit card previously. We still want to convert these users'
project limits. This billing command can be run once to convert all old
customers with a credti card. Afterwards, we should be able to safely
remove it.

Change-Id: Ia496580b8e72ef436375b74f590fe57cca704fa8
2021-07-12 13:37:37 +00:00
Michał Niewrzał
d53aacc058 satellite/repair: migrate to new repair_queue table
We want to use StreamID/Position to identify injured
segment. As it is hard to alter existing injuredsegments
table we are adding a new table that will replace existing
one. Old table will be dropped later.

Change-Id: I0d3b06522645013178b6678c19378ebafe485c49
2021-06-30 17:12:24 +02:00
Moby von Briesen
4e95d27033 web,satellite: Remove paywall-related functionality
Because of our free/paid tier plan, we do not need a paywall anymore. We
have not used it in a while, but still have leftover code laying around.

Change-Id: Iaea8c39faf042a2f7a6b837727bb135c8bdf2907
2021-06-29 02:47:48 +02:00
Michał Niewrzał
a93e47514a satellite: remove irreparabledb
This is part of metaloop refactoring. We plan to remove
irreparable at some point but there was not time for it.
Now instead refatoring it for segmentloop its just easier
to drop it.

Later we still need to drop table with migration step.

Change-Id: I270e77f119273d39a1ecdcf5e1c37a5662a29ab4
2021-06-17 07:20:15 +00:00
Egon Elbre
910eec8eee satellite/metainfo: remove MetabaseDB interface
Currently the interface is not useful. When we need to vary the
implementation for testing purposes we can introduce a local interface
for the service/chore that needs it, rather than using the large api.

Unfortunately, this requires adding a cleanup callback for tests, there
might be a better solution to this problem.

Change-Id: I079fe4dbe297b0ae08c10081a1cea4dfbc277682
2021-05-13 13:22:14 +00:00
Ivan Fraixedes
7fb86617fc satellite/satellitedb: Use CRDB AS OF SYSTEM & batch for GE
Use the 'AS OF SYSTEM TIME' Cockroach DB clause for the Graceful Exit
(a.k.a GE) queries that count the delete the GE queue items of nodes
which have already exited the network.

Split the subquery used for deleting all the transfer queue items of
nodes which has exited when CRDB is used and batch the queries because
CRDB struggles when executing in a single query unlike Postgres.

The new test which has been added to this commit to verify the CRDB
batch logic for deleting all the transfer queue items of the exited
nodes has raised that the Enqueue method has to run in baches when CRDB
is used otherwise CRDB has return the error "driver: bad connection"
when a big a amount of items are passed to be enqueued. This error
didn't happen with the current test implementation it was with an
initial one that it was creating a big amount of exited nodes and
transfer queue items for those nodes.

Change-Id: I6a099cdbc515a240596bc93141fea3182c2e50a9
2021-05-07 13:09:19 -04:00
JT Olio
a89a2b4b43 satellite/billing: make stripe invoice generation work with multiregion satellites
multiregion satellites have complex database connection strings
largely due to using a different backend for the repair queue than
cockroach.

billing stuff didn't work right with this.

Change-Id: Ie8759a8c47e71347c3a190abfc9d53945d7b8855
2021-05-06 11:51:03 -06:00
Egon Elbre
7802ab714f pkg/,private/: merge with private package
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.

Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
2021-04-23 16:37:28 +03:00
Egon Elbre
c641ddcb54 pkg/lrucache: rename package
cache is really common variable and type name and we have already used
the package name alias in multiple places.

Change-Id: I6435785b7549b541d533de59ec94557b9bd11e04
2021-04-23 16:37:28 +03:00
Egon Elbre
a2e20c93ae private/dbutil: use dbutil and tagsql from storj.io/private
Initially we duplicated the code to avoid large scale changes to
the packages. Now we are past metainfo refactor we can remove the
duplication.

Change-Id: I9d0b2756cc6e2a2f4d576afa408a15273a7e1cef
2021-04-23 14:36:52 +03:00
Kaloyan Raev
2ee3030275 all: remove code related to PointerDB
Change-Id: I6675c9597f87019020f6233b83ab2f1119d2bc46
2021-04-21 12:35:31 +00:00
Moby von Briesen
c334fd090e satellite/payments: Convert coupon duration to be nullable
* Add a nullable billing_periods column in the coupons table
* Add nullable billing_periods column to the currently unused
coupon_codes table
* Drop the duration column from the coupon_codes table
* Replace duration config type so that the default promotional coupon
can be configured to never expire

Zero downtime migration plan:
* Add billing_periods column to coupons and coupon_codes tables (this change)
* After one release, remove all references to the old duration column,
replacing with references to billing_periods. At this point, we can also
change the defult promotional coupon to never expire and migrate over
values from the old duration column.
* After another release, drop the duration column.

Change-Id: I374e8dc9fab9f81b4a5bc681771955662d4c007a
2021-04-01 16:28:23 +02:00
Jeff Wendling
a65aecfd98 compensation: always generate invoices for every node
instead of only generating invoices for nodes that had some
activity, we generate it for every node so that we can find
and pay terminal nodes that did not meet thresholds before
we recognized them as terminal.

Change-Id: Ibb3433e1b35f1ddcfbe292c034238c9fa1b66c44
2021-03-29 14:15:45 +00:00
Ivan Fraixedes
c5cb4dce4d redis: Rename functions prefixed with New by Open
Rename the functions that are prefixed with 'New' which connect with
Redis by 'Open' to  make clear that they perform network operations.

Change-Id: I1351e89a642e8e2c2586626646315ad0fb2c6242
2021-03-25 06:09:27 +00:00
Michał Niewrzał
237782813b Merge remote-tracking branch 'origin/multipart-upload'
Change-Id: If6c5a450b238adab55d1e0dea67d01e5f5768a9f
2021-03-23 09:44:49 +01:00
Ivan Fraixedes
4c1098e571 Redis: Update Redis package to last major version
Update the Redis dependency to use the last major production version.
The last version accepts a context parameter in all the network methods
so it allows us to pass it through them.

Change-Id: I34121b2ec3c2728602115c724933ad24c9e6e4fd
2021-03-18 14:19:49 +00:00
Michał Niewrzał
67e26aafcd Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I9b183323cb470185be22f7c648bb76917d2e6fca
2021-03-10 08:53:38 +01:00
Natalie Villasana
856db68fd9 satellite/gracefulexit: extend GE data cleanup to include exit_progress
The new 'consistency ge-cleanup-orphaned-data' cli command deleted
orphaned transfer queue items, but not entries in the
graceful_exit_progress table. This will delete orphaned entries
from the exit progress table too.

Change-Id: I5f927aac1f258490678deaf179be92ccfe10fcd8
2021-03-01 15:52:43 +00:00
Michał Niewrzał
d995fb497f Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I367da03351ab80f7343332420490dde9282aa47a
2021-02-23 12:31:31 +01:00
Egon Elbre
3c49f0ff73 cmd/satellite: restore-trash, cleanup workers
Use a limiter instead of a custom work queue.

Change-Id: I1506daaaaeb2935f6011ddaeb83a0904abb69c98
2021-02-19 19:08:10 +02:00
JT Olio
ac8c70e20d cmd/satellite: restore trash connection close
Change-Id: I96ca0ba653005f519a2d5e5e7eeb9e2c9f894056
2021-02-18 11:25:28 -07:00
JT Olio
b2ed7edd30 cmd/satellite: restore-trash parallel workers
Change-Id: Ic7466b21c20bda334e7ba4268a494e96b6528ac1
2021-02-18 19:11:19 +02:00
Michał Niewrzał
69151965b9 cmd/satellite: add logic to send RestoreTrash to nodes
Change-Id: I69815fa766a40a7266787e96bf9662854ae56937
2021-02-18 09:19:30 -07:00
JT Olio
3ae3389ddc cmd/satellite: restore-trash command
Change-Id: I80fc932c12147692d49cde277784871ac611fcad
2021-02-18 09:19:22 -07:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Yaroslav Vorobiov
1e328f3c30 satellite/compensation: add wallet features to invoice csv
Change-Id: I90a72c0a4c8d07604096913b6680263b6defc0a2
2021-02-08 22:13:48 +02:00
Yingrong Zhao
3b49d3cddf satellite: remove referral program related code
This PR removes all back-end related referral program code including the
marketing portal.

We will have a separate PR for front-end code and database migration to
drop `offers` and `usercredits` table

Change-Id: If59f952cddfe0558a7dc03a0eac7cc1081517f88
2021-02-08 13:52:50 +00:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Jeff Wendling
759bdd6794 satellite/compensation: add total-paid and total-distributed to invoices
Change-Id: Id4414867917cbf8aad77795f764d6381e88d9a34
2021-02-02 18:14:31 +00:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Ivan Fraixedes
d93944c57b satellite/orders: Delete unused methods & DB tables
Delete satellite order methods and DB tables which aren't used anymore
after we have done a refactoring on the orders to stuck bucket
information in the orders' encrypted metadata.

There are also configuration parameters and a satellite chore that
aren't needed anymore after the orders refactoring.

Change-Id: Ida3682b95921df70792284b42c96d2508bf8ca9c
2021-02-01 18:01:29 +00:00
Ivan Fraixedes
076804eac9 cmd/satellite: Add command for GE data cleanup
Add a command to the satellite for cleaning up the Graceful Exit (a.k.a
GE) transfer queue items of nodes that have exited.

The commit adds to the GE satellite DB a couple of new methods, and its
corresponding test, for performing the operations of the new command.

Change-Id: I29a572a59689d63b24990ac13c52e76d65aaa917
2021-02-01 17:30:58 +00:00
Kaloyan Raev
d0612199f0 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/config.go
	satellite/metainfo/metainfo_test.go

Change-Id: I95cf3c1d020a7918795b5eec63f36112fdb86749
2021-02-01 14:32:12 +02:00
Jeff Wendling
66e15fb7f1 satellite/compensation: remove ytd paid amounts
they aren't right and we aren't using them.

Change-Id: I5ca024e38d055696696886278863e941b5bc51bf
2021-01-27 17:31:01 +00:00