storj/satellite/satellitedb/testdata
Cameron Ayer 0184d33e96 satellite/satellitedb: set default 0 on uptime columns
This is the first step in the removal of uptime columns on the
nodes table. These columns are no longer used:

uptime_success_count
total_uptime_count
uptime_reputation_alpha
uptime_reputation_beta

In order to avoid breaking backwards compatibility, we need to
remove all references to these columns before removing the columns
themselves from the database. However, since uptime_success_count
and total_uptime_count are NOT NULLABLE, we can't remove them from
the insert statements in the overlay. So we can't remove the columns
because of the references, and we can't remove the references because
the columns can't be null. What a pickle. To remedy this, we will set a
default on the columns. Then we should be able to remove them from the
insert statements

Change-Id: I75f6c56fb7897835bbf29869f86f39de1d9dd345
2021-01-12 17:44:37 +00:00
..
postgres.v103.sql satellite/satellitedb : remove migation steps 69-102 2020-08-27 07:36:05 +00:00
postgres.v104.sql satellite/satellitedb: add some docs and improve some snapshots 2020-05-22 21:27:36 +00:00
postgres.v105.sql satellite/satellitedb: add some docs and improve some snapshots 2020-05-22 21:27:36 +00:00
postgres.v106.sql satellite/satellitedb: add some docs and improve some snapshots 2020-05-22 21:27:36 +00:00
postgres.v107.sql satellite: migration to fix bad imported payment history 2020-05-28 12:59:08 -06:00
postgres.v108.sql satellite: migration to fix bad imported payment history 2020-05-28 12:59:08 -06:00
postgres.v109.sql satellite: migration to fix bad imported payment history 2020-05-28 12:59:08 -06:00
postgres.v110.sql satellite: migration to fix bad imported payment history 2020-05-28 12:59:08 -06:00
postgres.v111.sql satellite: migration to fix bad imported payment history 2020-05-28 12:59:08 -06:00
postgres.v112.sql satellite: migration to fix bad imported payment history 2020-05-28 12:59:08 -06:00
postgres.v113.sql satellite/satellitedb: drop project_id column from coupons table 2020-06-03 14:56:41 +00:00
postgres.v114.sql satellite/satellitedb: add new columns for offline suspension 2020-06-15 04:00:20 +00:00
postgres.v115.sql satellite: Check macaroon revocation 2020-06-22 13:50:07 -06:00
postgres.v116.sql satellite/satellitedb: add new DB table audit_histories 2020-07-01 21:14:35 +00:00
postgres.v117.sql satellite/nodeapiversion: new table for tracking node api usage 2020-07-09 15:02:25 +00:00
postgres.v118.sql satellite: prevents uplink from creating a bucket once it exceeds the max bucket allocation. 2020-07-15 17:27:05 +00:00
postgres.v119.sql satellite/{console,satellitedb}: add project_limit column to users table 2020-07-15 17:27:31 +00:00
postgres.v120.sql satellite/{console,satellitedb}: change project limiting based on new users field 2020-07-16 10:57:47 +00:00
postgres.v121.sql satellite/payments: delete credits and credits_spendings db tables 2020-07-30 12:19:57 +03:00
postgres.v122.sql satellite/db: drop project_invoice_stamps table 2020-08-10 13:22:10 +00:00
postgres.v123.sql satellite/satellitedb: Add online_score column to nodes table 2020-08-31 15:07:07 +00:00
postgres.v124.sql satellite: make limits be nullable 2020-09-21 19:34:19 +00:00
postgres.v125.sql satellite/satellitedb: makes limits nullable change backwards compatible 2020-09-23 17:54:42 +02:00
postgres.v126.sql satellite/satellitedb: makes limits nullable change backwards compatible 2020-09-23 17:54:42 +02:00
postgres.v127.sql satellite/satellitedb: enable multiple projects existing users 2020-09-23 18:17:38 +02:00
postgres.v128.sql satellite/satellitedb: make limits per default NULL 2020-10-14 20:28:16 +00:00
postgres.v129.sql satellite/satellitedb: make limits per default NULL 2020-10-14 20:28:16 +00:00
postgres.v130.sql Add index to graceful_exit_transfer_queue table 2020-10-26 14:47:48 +00:00
postgres.v131.sql satellite/orders: add storagenode_bw_phase2 table and dont delete tallies for longer 2020-11-13 17:15:24 +00:00
postgres.v132.sql satellite/{satellitedb, repair/{queue, checker}}: Use new column "segmentHealth" instead of "numHealthy" in injured segments queue 2020-11-16 21:18:09 +00:00
postgres.v133.sql satellite/accounting: account for old orders that can be submitted in satellite rollup 2020-11-18 14:46:00 -05:00
postgres.v134.sql satellite/satellitedb: drop num_healthy_pieces column from injuredsegments 2020-12-17 20:17:08 +00:00
postgres.v135.sql satellite/accounting: Performance improvements to getNodeIds used by GetBandwidthSince (#4009) 2020-12-21 16:37:01 +01:00
postgres.v136.sql satellite/accounting: Add index for project_id on bucket_storage_tallies (#4010) 2020-12-21 11:42:00 -05:00
postgres.v137.sql satellite/overlay: Add index on nodes table (#4012) 2020-12-21 12:48:48 -05:00
postgres.v138.sql satellite/satellitedb: Drop nodes_offline_times table. 2020-12-29 18:17:50 +00:00
postgres.v139.sql satellite/satellitedb: set default 0 on uptime columns 2021-01-12 17:44:37 +00:00
README.md satellite: make limits be nullable 2020-09-21 19:34:19 +00:00

Migrations and You

This document is best read in the voice of one of those 50s instructional videos. You know the ones, like this one. If you want a sound track to go with it, I would recommend playing this in the background.


Mechanics

In order to understand the right way to add a new migration with associated tests, a basic understanding of how the tests run and what they check is required. The migration tests work by having

  1. A single database kept for the duration of the test that the individual migrations are run on one at a time in order
  2. An expected snapshot for the databse for every single step
  3. A sql file for each snapshot to generate it

For a given step, what happens is

  1. The -- OLD DATA -- section is just run on the database being migrated
  2. The main section and -- NEW DATA -- sections are run on the snapshot
  3. The migration is run against the main database
  4. The -- NEW DATA -- section is run on the database

Then, the snapshot and current database state are compared for schema and data differences.

How to Create a Migration

With that basic overview out of the way, the steps to create a new migration are

  1. Create an empty postgres.vN.sql file in this folder.
  2. Copy the satellitedb.dbx.postgres.sql file from the satellitedb/dbx folder. This ensures that the snapshot does not drift from the dbx sql file. We bootstrap tests from the dbx output, so the correctness of our tests depends on them matching.
  3. Copy the INSERT statements from the end of the previous migration. These lines are after the CREATE INDEX lines but before any -- NEW DATA -- or -- OLD DATA -- section.
  4. Copy the -- NEW DATA -- statements from the end of the previous migration into the main section. They are no longer -- NEW DATA --. You should only need to copy INSERT statements.

Depending on what your migration is doing, you should then do one of these:

  1. If your migration is creating new tables, add an INSERT for it in the -- NEW DATA -- section. This ensures that future migrations don't break data in that table. Our test coverage depends on these INSERTs existing and being complete. Help avoid problems in production, now. Smokey the Bear says only you can prevent production fires.
  2. If you are updating data in old tables or changing the table schema a. and there is previous data (which there should be, by point 1), change the existing row in the main section to be the new updated value. b. and there is no previous data, add an insert into the -- OLD DATA -- section as well as a corresponding updated row like in point a. Unfortunately, someone got away with adding a table without adding a row, but no longer. The tests must forever grow.
  3. Anything more complicated, look at the mechanics section again to see if there's a way to solve your problem. If it is unclear (it probably is), just ask for help. Maybe the brains of multiple people can figure out something to do.

Best Practices and Common Mistakes

  1. Don't do in two migrations what can be done in one. The more migrations we have, the longer test times are, and the more often we may have to collapse them. Unless you have a good reason to do two migrations, just do one.

  2. There is almost no reason to have an UPDATE statement in a sql snapshot file. Each snapshot is run independently, so you should just adjust the INSERT statement to reflect the rows you expect to exist, and leave the UPDATE to the migration. This helps test that the migration runs and does what we expect. One exception is when the migration does not have deterministic output. For example,

    {
    	DB:          db.DB,
    	Description: "Backfill vetted_at with time.now for nodes that have been vetted already (aka nodes that have been audited 100 times)",
    	Version:     99,
    	Action: migrate.SQL{
    		`UPDATE nodes SET vetted_at = date_trunc('day', now() at time zone 'utc') at time zone 'utc' WHERE total_audit_count >= 100;`,
    	},
    },
    

    The above migration sets a value to the current day. This can be solved in one of two ways:

    1. Have each of the INSERT statements for the matching nodes in the main section use date_trunc('day', now() at time zone 'utc') at time zone 'utc' for the vetted_at column
    2. Have an UPDATE "nodes" SET vetted_at = 'fixed date' where id = 'expected id' in the -- NEW DATA -- section (so that it runs in both the snapshot, and after the migration has run) and update the main section's INSERT for that node. Future snapshot files do not need to retain the UPDATE in that case, and the INSERT statements can just use the fixed date for the future.

    See migration 99 for the specifics, where it chose option 2.

  3. Cockroach does schema changes asynchronously with regards to a transaction. This means if you need to add a column and fill it with some data, then these need to have them in separate migrations steps with using SeparateTx:

    {
    	DB:          db.DB,
    	Description: "Add project bandwidth limit",
    	Version:     999,
    	Action: migrate.SQL{
    		`ALTER TABLE projects ADD COLUMN bandwidth_limit`,
    		`UPDATE projects SET bandwidth_limit = usage_limit`,
    	},
    },
    

    Will fail with column "bandwidth_limit" is missing. To make it work, it needs to be written as:

    {
    	DB:          db.DB,
    	Description: "add separate bandwidth column",
    	Version:     107,
    	Action: migrate.SQL{
    		`ALTER TABLE projects ADD COLUMN bandwidth_limit bigint NOT NULL DEFAULT 0;`,
    	},
    },
    {
    	DB:          db.DB,
    	Description: "backfill bandwidth column with previous limits",
    	Version:     108,
    	SeparateTx:  true,
    	Action: migrate.SQL{
    			`UPDATE projects SET bandwidth_limit = usage_limit;`,
    	},
    },
    
  4. Removing a DEFAULT value for a column can be tricky. Old values inserted in the snapshot files may not specify every column and relying on those defaults. In the migration that removes the DEFAULT value, you must also change any INSERT statements to include any unspecified columns, setting them to the dropped DEFAULT. This is because the main database will have inserted them while they had the DEFAULT, but the new snapshot will not be inserting while the columns have the DEFAULT.

FAQ

As questions are asked and answered, and knowledge is gained, this section will include that knowledge so that we can keep it in a spot people can easily find and refer to when adding new migrations.