storj/satellite/satellitedb/coinpaymentstxs.go

387 lines
12 KiB
Go
Raw Normal View History

// Copyright (C) 2019 Storj Labs, Inc.
// See LICENSE for copying information.
package satellitedb
import (
"context"
"time"
"github.com/shopspring/decimal"
"github.com/zeebo/errs"
"storj.io/common/uuid"
"storj.io/storj/satellite/payments/coinpayments"
"storj.io/storj/satellite/payments/monetary"
"storj.io/storj/satellite/payments/stripecoinpayments"
"storj.io/storj/satellite/satellitedb/dbx"
)
// ensure that coinpaymentsTransactions implements stripecoinpayments.TransactionsDB.
var _ stripecoinpayments.TransactionsDB = (*coinPaymentsTransactions)(nil)
// applyBalanceIntentState defines states of the apply balance intents.
type applyBalanceIntentState int
const (
// apply balance intent waits to be applied.
applyBalanceIntentStateUnapplied applyBalanceIntentState = 0
// transaction which balance intent points to has been consumed.
applyBalanceIntentStateConsumed applyBalanceIntentState = 1
)
// Int returns intent state as int.
func (intent applyBalanceIntentState) Int() int {
return int(intent)
}
// coinPaymentsTransactions is CoinPayments transactions DB.
//
// architecture: Database
type coinPaymentsTransactions struct {
satellite/satellitedb: unexport satellitedb.DB Backstory: I needed a better way to pass around information about the underlying driver and implementation to all the various db-using things in satellitedb (at least until some new "cockroach driver" support makes it to DBX). After hitting a few dead ends, I decided I wanted to have a type that could act like a *dbx.DB but which would also carry information about the implementation, etc. Then I could pass around that type to all the things in satellitedb that previously wanted *dbx.DB. But then I realized that *satellitedb.DB was, essentially, exactly that already. One thing that might have kept *satellitedb.DB from being directly usable was that embedding a *dbx.DB inside it would make a lot of dbx methods publicly available on a *satellitedb.DB instance that previously were nicely encapsulated and hidden. But after a quick look, I realized that _nothing_ outside of satellite/satellitedb even needs to use satellitedb.DB at all. It didn't even need to be exported, except for some trivially-replaceable code in migrate_postgres_test.go. And once I made it unexported, any concerns about exposing new methods on it were entirely moot. So I have here changed the exported *satellitedb.DB type into the unexported *satellitedb.satelliteDB type, and I have changed all the places here that wanted raw dbx.DB handles to use this new type instead. Now they can just take a gander at the implementation member on it and know all they need to know about the underlying database. This will make it possible for some other pending code here to differentiate between postgres and cockroach backends. Change-Id: I27af99f8ae23b50782333da5277b553b34634edc
2019-12-14 02:29:54 +00:00
db *satelliteDB
}
// Insert inserts new coinpayments transaction into DB.
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
func (db *coinPaymentsTransactions) Insert(ctx context.Context, tx stripecoinpayments.Transaction) (createTime time.Time, err error) {
defer mon.Task()(&ctx)(&err)
dbxCPTX, err := db.db.Create_CoinpaymentsTransaction(ctx,
dbx.CoinpaymentsTransaction_Id(tx.ID.String()),
dbx.CoinpaymentsTransaction_UserId(tx.AccountID[:]),
dbx.CoinpaymentsTransaction_Address(tx.Address),
dbx.CoinpaymentsTransaction_AmountNumeric(tx.Amount.BaseUnits()),
dbx.CoinpaymentsTransaction_ReceivedNumeric(tx.Received.BaseUnits()),
dbx.CoinpaymentsTransaction_Status(tx.Status.Int()),
dbx.CoinpaymentsTransaction_Key(tx.Key),
dbx.CoinpaymentsTransaction_Timeout(int(tx.Timeout.Seconds())),
)
if err != nil {
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
return time.Time{}, err
}
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
return dbxCPTX.CreatedAt, nil
}
// Update updates status and received for set of transactions.
func (db *coinPaymentsTransactions) Update(ctx context.Context, updates []stripecoinpayments.TransactionUpdate, applies coinpayments.TransactionIDList) (err error) {
defer mon.Task()(&ctx)(&err)
if len(updates) == 0 {
return nil
}
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
err = db.db.WithTx(ctx, func(ctx context.Context, tx *dbx.Tx) error {
for _, update := range updates {
_, err = tx.Update_CoinpaymentsTransaction_By_Id(ctx,
dbx.CoinpaymentsTransaction_Id(update.TransactionID.String()),
dbx.CoinpaymentsTransaction_Update_Fields{
ReceivedNumeric: dbx.CoinpaymentsTransaction_ReceivedNumeric(update.Received.BaseUnits()),
Status: dbx.CoinpaymentsTransaction_Status(update.Status.Int()),
},
)
if err != nil {
return err
}
}
for _, txID := range applies {
query := db.db.Rebind(`INSERT INTO stripecoinpayments_apply_balance_intents ( tx_id, state, created_at )
VALUES ( ?, ?, ? ) ON CONFLICT DO NOTHING`)
_, err = tx.Tx.ExecContext(ctx, query, txID.String(), applyBalanceIntentStateUnapplied.Int(), db.db.Hooks.Now().UTC())
if err != nil {
return err
}
}
return nil
})
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
return err
}
// Consume marks transaction as consumed, so it won't participate in apply account balance loop.
func (db *coinPaymentsTransactions) Consume(ctx context.Context, id coinpayments.TransactionID) (err error) {
defer mon.Task()(&ctx)(&err)
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
query := db.db.Rebind(`
WITH intent AS (
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
SELECT tx_id, state FROM stripecoinpayments_apply_balance_intents WHERE tx_id = ?
), updated AS (
UPDATE stripecoinpayments_apply_balance_intents AS ints
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
SET
state = ?
FROM intent
WHERE intent.tx_id = ints.tx_id AND ints.state = ?
RETURNING 1
)
SELECT EXISTS(SELECT 1 FROM intent) AS intent_exists, EXISTS(SELECT 1 FROM updated) AS intent_consumed;
`)
row := db.db.QueryRowContext(ctx, query, id, applyBalanceIntentStateConsumed, applyBalanceIntentStateUnapplied)
var exists, consumed bool
if err = row.Scan(&exists, &consumed); err != nil {
return err
}
if !exists {
return errs.New("can not consume transaction without apply balance intent")
}
if !consumed {
return stripecoinpayments.ErrTransactionConsumed
}
return err
}
// LockRate locks conversion rate for transaction.
func (db *coinPaymentsTransactions) LockRate(ctx context.Context, id coinpayments.TransactionID, rate decimal.Decimal) (err error) {
defer mon.Task()(&ctx)(&err)
rateFloat, exact := rate.Float64()
if !exact {
// It's not clear at the time of writing whether this
// inexactness will ever be something we need to worry about.
// According to the example in the API docs for
// coinpayments.net, exchange rates are given to 24 decimal
// places (!!), which is several digits more precision than we
// can represent exactly in IEEE754 double-precision floating
// point. However, that might not matter, since an exchange rate
// that is correct to ~15 decimal places multiplied by a precise
// monetary.Amount should give results that are correct to
// around 15 decimal places still. At current exchange rates,
// for example, a USD transaction would need to have a value of
// more than $1,000,000,000,000 USD before a calculation using
// this "inexact" rate would get the equivalent number of BTC
// wrong by a single satoshi (10^-8 BTC).
//
// We could avoid all of this by preserving the exact rates as
// given by our provider, but this would involve either (a)
// abuse of the SQL schema (e.g. storing rates as decimal values
// in VARCHAR), (b) storing rates in a way that is opaque to the
// db engine (e.g. gob-encoding, decimal coefficient with
// separate exponents), or (c) adding support for parameterized
// types like NUMERIC to dbx. None of those are very ideal
// either.
delta, _ := rate.Sub(decimal.NewFromFloat(rateFloat)).Float64()
mon.FloatVal("inexact-float64-exchange-rate-delta").Observe(delta)
}
_, err = db.db.Create_StripecoinpaymentsTxConversionRate(ctx,
dbx.StripecoinpaymentsTxConversionRate_TxId(id.String()),
dbx.StripecoinpaymentsTxConversionRate_RateNumeric(rateFloat),
)
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
return Error.Wrap(err)
}
// GetLockedRate returns locked conversion rate for transaction or error if non exists.
func (db *coinPaymentsTransactions) GetLockedRate(ctx context.Context, id coinpayments.TransactionID) (rate decimal.Decimal, err error) {
defer mon.Task()(&ctx)(&err)
dbxRate, err := db.db.Get_StripecoinpaymentsTxConversionRate_By_TxId(ctx,
dbx.StripecoinpaymentsTxConversionRate_TxId(id.String()),
)
if err != nil {
return decimal.Decimal{}, err
}
rate = decimal.NewFromFloat(dbxRate.RateNumeric)
return rate, nil
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
}
// ListAccount returns all transaction for specific user.
func (db *coinPaymentsTransactions) ListAccount(ctx context.Context, userID uuid.UUID) (_ []stripecoinpayments.Transaction, err error) {
defer mon.Task()(&ctx)(&err)
dbxTXs, err := db.db.All_CoinpaymentsTransaction_By_UserId_OrderBy_Desc_CreatedAt(ctx,
dbx.CoinpaymentsTransaction_UserId(userID[:]),
)
if err != nil {
return nil, err
}
var txs []stripecoinpayments.Transaction
for _, dbxTX := range dbxTXs {
tx, err := fromDBXCoinpaymentsTransaction(dbxTX)
if err != nil {
return nil, errs.Wrap(err)
}
txs = append(txs, *tx)
}
return txs, nil
}
// ListPending returns paginated list of pending transactions.
func (db *coinPaymentsTransactions) ListPending(ctx context.Context, offset int64, limit int, before time.Time) (_ stripecoinpayments.TransactionsPage, err error) {
defer mon.Task()(&ctx)(&err)
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
query := db.db.Rebind(`SELECT
id,
user_id,
address,
amount_numeric,
received_numeric,
status,
key,
created_at
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
FROM coinpayments_transactions
WHERE status IN (?,?)
AND created_at <= ?
ORDER by created_at DESC
LIMIT ? OFFSET ?`)
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
rows, err := db.db.QueryContext(ctx, query, coinpayments.StatusPending, coinpayments.StatusReceived, before, limit+1, offset)
if err != nil {
return stripecoinpayments.TransactionsPage{}, Error.Wrap(err)
}
defer func() {
err = errs.Combine(err, rows.Close())
}()
var page stripecoinpayments.TransactionsPage
for rows.Next() {
var id, address string
var userID uuid.UUID
var amount, received *int64
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
var status int
var key string
var createdAt time.Time
err := rows.Scan(&id, &userID, &address, &amount, &received, &status, &key, &createdAt)
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
if err != nil {
return stripecoinpayments.TransactionsPage{}, Error.Wrap(err)
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
}
// TODO: the currency here should be passed in to this function or stored
// in the database.
currency := monetary.StorjToken
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
page.Transactions = append(page.Transactions,
stripecoinpayments.Transaction{
ID: coinpayments.TransactionID(id),
AccountID: userID,
Address: address,
Amount: monetary.AmountFromBaseUnits(*amount, currency),
Received: monetary.AmountFromBaseUnits(*received, currency),
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
Status: coinpayments.Status(status),
Key: key,
CreatedAt: createdAt,
},
)
}
if err = rows.Err(); err != nil {
return stripecoinpayments.TransactionsPage{}, err
}
if len(page.Transactions) == limit+1 {
page.Next = true
page.NextOffset = offset + int64(limit)
page.Transactions = page.Transactions[:len(page.Transactions)-1]
}
return page, nil
}
// ListUnapplied returns TransactionsPage with a pending or completed status, that should be applied to account balance.
func (db *coinPaymentsTransactions) ListUnapplied(ctx context.Context, offset int64, limit int, before time.Time) (_ stripecoinpayments.TransactionsPage, err error) {
defer mon.Task()(&ctx)(&err)
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
query := db.db.Rebind(`SELECT
txs.id,
txs.user_id,
txs.address,
txs.amount_numeric,
txs.received_numeric,
txs.status,
txs.key,
txs.created_at
FROM coinpayments_transactions as txs
INNER JOIN stripecoinpayments_apply_balance_intents as ints
ON txs.id = ints.tx_id
WHERE txs.status >= ?
AND txs.created_at <= ?
AND ints.state = ?
ORDER by txs.created_at DESC
LIMIT ? OFFSET ?`)
rows, err := db.db.QueryContext(ctx, query, coinpayments.StatusReceived, before, applyBalanceIntentStateUnapplied, limit+1, offset)
if err != nil {
return stripecoinpayments.TransactionsPage{}, err
}
defer func() { err = errs.Combine(err, rows.Close()) }()
var page stripecoinpayments.TransactionsPage
for rows.Next() {
var id, address string
var userID uuid.UUID
var amountNumeric, receivedNumeric *int64
var status int
var key string
var createdAt time.Time
err := rows.Scan(&id, &userID, &address, &amountNumeric, &receivedNumeric, &status, &key, &createdAt)
satellite/satellitedb: prepare to remove big.Float from db Why: big.Float is not an ideal type for dealing with monetary amounts, because no matter how high the precision, some non-integer decimal values can not be represented exactly in base-2 floating point. Also, storing gob-encoded big.Float values in the database makes it very hard to use those values in meaningful queries, making it difficult to do any sort of analysis on billing. Now that we have amounts represented using monetary.Amount, we can simply store them in the database using integers (as given by the .BaseUnits() method on monetary.Amount). We should move toward storing the currency along with any monetary amount, wherever we are storing amounts, because satellites might want to deal with currencies other than STORJ and USD. Even better, it becomes much clearer what currency each monetary value is _supposed_ to be in (I had to dig through code to find that out for our current monetary columns). Deployment ---------- Getting rid of the big.Float columns will take multiple deployment steps. There does not seem to be any way to make the change in a way that lets existing queries continue to work on CockroachDB (it could be done with rules and triggers and a stored procedure that knows how to gob-decode big.Float objects, but CockroachDB doesn't have rules _or_ triggers _or_ stored procedures). Instead, in this first step, we make no changes to the database schema, but add code that knows how to deal with the planned changes to the schema when they are made in a future "step 2" deployment. All functions that deal with the coinbase_transactions table have been taught to recognize the "undefined column" error, and when it is seen, to call a separate "transition shim" function to accomplish the task. Once all the services are running this code, and the step 2 deployment makes breaking changes to the schema, any services that are still running and connected to the database will keep working correctly because of the fallback code included here. The step 2 deployment can be made without these transition shims included, because it will apply the database schema changes before any of its code runs. Step 1: No schema changes; just include code that recognizes the "undefined column" error when dealing with the coinbase_transactions or stripecoinpayments_tx_conversion_rates tables, and if found, assumes that the column changes from Step 2 have already been made. Step 2: In coinbase_transactions: * change the names of the 'amount' and 'received' columns to 'amount_gob' and 'received_gob' respectively * add new 'amount_numeric' and 'received_numeric' columns with INT8 type. In stripecoinpayments_tx_conversion_rates: * change the name of the 'rate' column to 'rate_gob' * add new 'rate_numeric' column with NUMERIC(8, 8) type Code reading from either of these tables must query both the X_gob and X_numeric columns. If X_numeric is not null, its value should be used; otherwise, the gob-encoded big.Float in X_gob should be used. A chore might be included in this step that transitions values from X_gob to X_numeric a few rows at a time. Step 3: Once all prod satellites have no values left in the _gob columns, we can drop those columns and add NOT NULL constraints to the _numeric columns. Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-08-10 23:30:23 +01:00
if err != nil {
return stripecoinpayments.TransactionsPage{}, err
}
// TODO: the currency here should be passed in to this function or stored
// in the database.
currency := monetary.StorjToken
page.Transactions = append(page.Transactions,
stripecoinpayments.Transaction{
ID: coinpayments.TransactionID(id),
AccountID: userID,
Address: address,
Amount: monetary.AmountFromBaseUnits(*amountNumeric, currency),
Received: monetary.AmountFromBaseUnits(*receivedNumeric, currency),
Status: coinpayments.Status(status),
Key: key,
CreatedAt: createdAt,
},
)
}
if err = rows.Err(); err != nil {
return stripecoinpayments.TransactionsPage{}, err
}
if len(page.Transactions) == limit+1 {
page.Next = true
page.NextOffset = offset + int64(limit)
page.Transactions = page.Transactions[:len(page.Transactions)-1]
}
return page, nil
}
// fromDBXCoinpaymentsTransaction converts *dbx.CoinpaymentsTransaction to *stripecoinpayments.Transaction.
func fromDBXCoinpaymentsTransaction(dbxCPTX *dbx.CoinpaymentsTransaction) (*stripecoinpayments.Transaction, error) {
userID, err := uuid.FromBytes(dbxCPTX.UserId)
if err != nil {
return nil, errs.Wrap(err)
}
// TODO: the currency here should be passed in to this function or stored
// in the database.
currency := monetary.StorjToken
return &stripecoinpayments.Transaction{
ID: coinpayments.TransactionID(dbxCPTX.Id),
AccountID: userID,
Address: dbxCPTX.Address,
Amount: monetary.AmountFromBaseUnits(dbxCPTX.AmountNumeric, currency),
Received: monetary.AmountFromBaseUnits(dbxCPTX.ReceivedNumeric, currency),
Status: coinpayments.Status(dbxCPTX.Status),
Key: dbxCPTX.Key,
Timeout: time.Second * time.Duration(dbxCPTX.Timeout),
CreatedAt: dbxCPTX.CreatedAt,
}, nil
}