Commit Graph

256 Commits

Author SHA1 Message Date
Wilfred Asomani
26574fb2bd satellite/db: add account activation columns
This change adds a new column to the user table, which will hold a
random code for account activation. And another to hold the signup
request ID as another layer of verification.

Issue: #6428

Change-Id: Icd46cb5d8fc76102264d599aca27686cd8b2e84e
2023-11-22 09:57:52 +00:00
Clement Sam
f5164c78cd Revert "satellite/satellitedb: drop type column on nodes table"
This reverts commit 2b4f347c33.

Reason for revert: not a zero-downtime migration. We need to deploy the dbx changes before completely dropping the column.

Change-Id: Ic9215650242b7848d54f8a5f863a13bc18b60149
2023-11-20 13:36:44 +00:00
Clement Sam
2b4f347c33 satellite/satellitedb: drop type column on nodes table
Updates https://github.com/storj/storj/issues/5426

Change-Id: If554f30124f234d1c50b2e8059e3d1dfec389ebb
2023-11-20 10:09:17 +00:00
dlamarmorgan
fe9afad8cd
satellite/{buckets,satellitedb}: add versioning to buckets.DB
Add the DB schema changes and basic implementation methods to support
bucket level configuration of object versioning.

See bucket-level-versioning-configuration design doc for more details.

Change-Id: I4e920a20a403b3157970a34eb619d827a4007845
2023-10-24 11:16:31 -07:00
Moby von Briesen
e5fd061e70 satellite/satellitedb: Drop value_attributions.partner_id
Removes final instance of legacy "partner ID" column.

Resolves https://github.com/storj/storj/issues/5432

Change-Id: I89657c8f9f366c79a5439632f870a47db590ff99
2023-10-20 20:25:13 +00:00
Clement Sam
67bd36ae4d satellite/satellitedb: remove nodes.type from indices on nodes table
Since we no longer need the type column and hopefully,
there are no queries using it, we recreate all indices
that reference the nodes.type column.

Updates https://github.com/storj/storj/issues/5426

Change-Id: I610ccaf474a6f4031e166b79a6d649c4b138e338
2023-10-20 14:48:36 +00:00
Wilfred Asomani
be025a40cc satellite/{db,console}: add days column to account freeze table
This change adds a new column to the account_freeze_events table,
days_till_escalation, which is the number of days till the freeze event
is escalated. E.g.: 30 days from billing warning event to billing
freeze event.

Issue: #6382

Change-Id: I15cfd6f1208e641d8c380bef61717031befdad73
2023-10-19 18:18:44 +00:00
Michal Niewrzal
281edfa585 satellite/satellitedb: add index to bucket_storage_tallies table
Additional index on 'interval_start' to support data analysis.

Change-Id: I373787871da594aa1b4ebd66cb684484221eb4d5
2023-10-16 15:09:24 +00:00
Michal Niewrzal
2c76c1e6ae satellite/satellitedb: alter bucket_metainfos primary key
Current bucket_metainfos table schema was far from optimal. All
operations are using project_id and name to find bucket but primary key
was on id field which is completely not used. Turns out that secondary
index was not as fast as it could be because each select was also joining
primary index which was slowing down whole query.

Primary key is changed to project_id/name. Other indexes are dropped
and id column is now nullable which gives us space for dropping it
completely from schema and code later.

Change-Id: I477056a4243d39e39489f1473de01ded1ed24bbb
2023-10-03 12:23:00 +00:00
Márton Elek
afa5c54a35 satellite/satellitedb: add placement column to repair_queue
It makes it possible to run dedicated repair worker for different placement definitions.

https://github.com/storj/storj-private/issues/400

Change-Id: I376da867da5dbb4ab392d5f86c766f7543c32ee6
2023-09-12 11:29:18 +00:00
Márton Elek
500b6244f8
satellite/satellitedb: create table for node tags
Change-Id: I884bb740974e6b8241aa6b85faf266b85fe892d4
2023-07-05 09:38:53 +02:00
Jeremy Wharton
bb620e746b satellite/satellitedb: add index for project_members project_id column
An index has been added on the project_id column of the project_members
satellite database table so that we can retrieve members of a project
without performing a full table scan.

References #5855

Change-Id: I1cc30686f836c8fd1aa319247ce857a2392e7a52
2023-06-11 18:12:33 -05:00
Márton Elek
5c26661cd3 dbx/satellitedb: introduce new default_placement fields for users/projects
https://github.com/storj/storj/issues/5879

Change-Id: Ie6d50fd94c09f43c12f895444ed2a86f0820e01c
2023-06-08 10:02:10 +00:00
JT Olio
1437257dbf satellite: save and return which node features are enabled
current feature is if tcp fastopen was successfully enabled

Change-Id: Ide251863a9790b0fbebdf2e82dfd2afa8f25c408
2023-06-06 21:13:29 +00:00
Moby von Briesen
563239d555 satellite/satellitedb: Add value_attributions.partner_id back
There were some dbx-generated queries on the old satellite version that
still reference this column, so we need to add it back until the next
version.

Change-Id: I78b19336d9ca0384936d6cc11f5c50e579b4f2ab
2023-05-31 17:31:33 -04:00
Moby von Briesen
919278dbf9 satellite/satellitedb: drop partnerID columns
The last code referencing these columns was removed as of satellite
release v1.79, so it is safe to remove them now.

Resolves https://github.com/storj/storj/issues/5432

Change-Id: I2e9d641b2511a61e0b9482ef0f4955a73c290709
2023-05-24 14:38:57 +00:00
Jeremy Wharton
607b120116 satellite/satellitedb: add inviter ID column to project invites table
The project member invitations table has been modified to contain a
column for the ID of the user who sent the invitation. This ID is
required for us to return information about the inviter to the
satellite frontend.

References #5855

Change-Id: I928d987a8db2340f731ca65ce30173d4f90a9837
2023-05-17 15:46:40 +00:00
Michal Niewrzal
fa16d947c9 satellite/satellitedb: add index on owner_id column for projects table
We have method which is getting projects owned by specific user but it's
causing full table scan because we don't have index on owner_id column.

Change-Id: Icb71c9ac5b73104a52241ed8ba126c995c10811f
2023-05-09 08:57:32 +00:00
Moby von Briesen
432b12d1c4 satellite/satellitedb: make value_attributions.partner_id nullable
There is still a reference to partner_id in a query, which we cannot
move until dropping the "not null" constraint for it. This change adds
that migration.

Related to https://github.com/storj/storj/issues/5432

Change-Id: I98802a6e1bd59f3d9214de3db6688d9daf664a70
2023-05-05 23:42:50 +00:00
Jeremy Wharton
defb9eae82 satellite/satellitedb: add table for project invitations
A table for storing pending project member invitations has been added.
This table is required to satisfy our new project invitation UX revamp.

References #5766

Change-Id: I6f948de66ed5b4dc81532564958ff7f48533cad2
2023-04-26 11:26:12 -05:00
Cameron
549732228d satellite/satellitedb: add package info to customers db
Add columns package_plan and purchased_package_at to stripe_customers
table and add methods to update and select these values from console
service and payments accounts.

Change-Id: I1e89909055cc3054bfb7baa33c9dca3dfdc7336e
2023-03-31 15:51:30 +00:00
Wilfred Asomani
7c68b51c82 satellite/{console,db}: add onboardings columns
This change adds onboarding_start, onboarding_end and onboarding_step
columns to the user_settings table. the first two are used to determine
if a user should go through onboarding, the last will be used to as the
step a user got to before exiting onboarding without finishing.

Issue: https://github.com/storj/storj/issues/5660

Change-Id: I8070c880d0d2fc22086f24087c962f57c695cc50
2023-03-23 17:50:31 +00:00
JT Olio
2a63225b98 satellite/{contact,satellitedb}: preserve node message debounce support
Change-Id: I453ad35fda4e61d068db0c476dd86b50d7f2d705
2023-03-20 16:13:06 +00:00
Moby von Briesen
0980a2f9ce satellitedb: Add index to users table
Add a combined index on normalized_email,status to improve performance of
common "get user" query used for the satellite UI.

Change-Id: I24a20d7826e0a68a68c2f95b5847eb819921e7c0
2023-03-16 09:08:49 -04:00
Michal Niewrzal
d54ccfa92b satellite/satellitedb: change bucket_bandwidth_rollups primary key
In the past we set incorrect primary key for bucket_bandwidth_rollups
table and bucket name was first column of key instead of project id.
This change fixes this problem.

We will alter primary key manually for production satellites. That's
why migration code is checking if change was already applied.

https://github.com/storj/storj/issues/5332

Change-Id: I7dd555eb0c2f6fda1c90eaed08fa847b7f273bc7
2023-03-03 16:25:35 +00:00
Lizzy Thomson
54f0f47839 satellite/satellitedb: add passphrase_prompt column to user_settings
Add passphrase_prompt column to user_settings table to store a boolean
whether a user would like to be prompted to enter a passphrase when
entering a new project.

Issue https://github.com/storj/storj/issues/5613

Change-Id: I5d086df277aff458453343c7c38a379e7b1b8bf9
2023-03-01 14:53:18 -07:00
Lizzy Thomson
f733cb255b satellite/satellitedb: drop unused last_verification_reminder column
(updated to migration step 225) remove users table column
last_verification_reminder and update migrate_test.go

Issue: https://github.com/storj/storj/issues/4560

Change-Id: Icbd8b70a8243b75a11c481a3b87df0ebe9402c5f
2023-02-08 16:23:09 +00:00
Jeremy Wharton
49916b02f0 satellite/{console,satellitedb}: migration, methods for user settings
This change adds a database migration for a new table that stores
configurations for a user, the first of which is the session duration.
Database methods are implemented to interact with this table.

Resolves #5472

Change-Id: I01049265f385ea5de65907da1bc3bcf426d3c577
2023-02-07 21:48:29 +00:00
Michal Niewrzal
0b519fbd05 satellite/satellitedb: create index for interval_day for project_bandwidth_daily_rollup
To improve deletion of old entries in project_bandwidth_daily_rollup
we need index on `interval_day` column which is used to find those old
entries.

As an addition we are changing interval how often deletion is executed
from 7 to 1 day. We would like to have smaller portion of data to
delete.

Fixes https://github.com/storj/storj/issues/5465

Change-Id: Ie18ebe859887b93d6e4e6065a61fb9214c7ad27a
2023-02-03 10:02:58 +00:00
JT Olio
2753d5a32f satellite/overlay: keep track of noise info per node
Change-Id: Icef04c3e87dbf4bb57d3837274c323bf6dd2c81f
2023-02-01 23:03:35 -05:00
Michal Niewrzal
2876d5f20b satellite/satellitedb: drop project_bandwidth_rollups table
We are not using this table.

Change-Id: I3e4d5d0c2d8522b319caf48fea460b1cc35d5630
2023-01-12 10:23:48 +00:00
Moby von Briesen
e26654c2a9 satellite/satellitedb: Drop unused tables
The `coupons`, `offers`, `coupon_usages`, `coupon_codes`, and
`user_credits` tables are not used anymore. They exist due to
legacy billing code which has already been removed.

Change-Id: Ie93af90a06e5247a74015af2a78a223d69249565
2023-01-10 17:00:15 +00:00
Cameron
021c98c17a satellite/satellitedb: create table account_freeze_events
create table for account freeze events

github issue: https://github.com/storj/storj/issues/5396

Change-Id: Ia4f93ce5caacd02b161853e3261d237bc715f23d
2022-12-13 19:09:25 +00:00
Cameron
602f0cc132 satellite/satellitedb: add node events column last_attempted
Also add a method to read a node event by id.

Change-Id: I08ca010e18a08253015e0e1280f791d0cc04d239
2022-12-01 18:35:06 +00:00
Cameron
5f6b4900d2 satellite/satellitedb: add columns last_offline_email and last_software_update_email
Change-Id: I8b5023bbbba55de633ce5ac3faf3361352c202ef
2022-11-11 14:43:59 -05:00
Fadila Khadar
553c1558a5 satellite/satellitedb: add 'contained' column to nodes table
We want to be able to exclude contained nodes from nodes selection. For this, we add a 'contained' column to the nodes table to track the containment status.

Fixes https://github.com/storj/storj/issues/5231

Change-Id: Id78e645f172145adcb8664646e8ebf14e218b57b
2022-11-09 11:11:37 -06:00
paul cannon
1b5d953fd9 satellite/satellitedb: add queue for verifications
Since the auditor will be moving to a different process from the
metainfo loop, we need a way of communicating which segments have
been chosen for audit. This queue will be that communication, for now.

Contrast this with the queue for _re_verifications in commit 9c67f62f.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: I9a269c7ef21e6c5e9c6e5e1f3db298fe159a8a79
2022-11-09 15:38:56 +00:00
Cameron
80bfa55b8e satellite/satellitedb: edit node_events migration to have different PK
Change node_events schema to use an id column as primary key rather than
node id because there can be multiple events per node id.

Change-Id: I518d8ef9ea658764876483e282a4058d3c4910f4
2022-10-28 22:59:28 +00:00
Cameron
6bf54bfaef satellite/satellitedb: add new table node_events
Add new table for node events. We can use this to notify node
operators of certain node events. Further, we can squash events for
multiple nodes with the same email into a single notification.

Change-Id: Icea6dd939df8fe4a98806bd79c014e21d239c43e
2022-10-28 19:16:25 +00:00
paul cannon
9c67f62fe3 satellite/satellitedb: add table for reverify queue
This table will be used as a queue for pieces that need to be reverified
(a regular audit timed out on the owning node, so now that node is
contained and we need to validate the piece before un-containing it).

Refs: https://github.com/storj/storj/issues/5228

Change-Id: I5dcd26b6adced8674cbd81884c1543a61ea9d4c8
2022-10-27 15:28:47 +00:00
Lizzy Thomson
ef04eb8bea satellite/satellitedb: add columns in projects table
add user_specified_usage_limit and user_specified_bandwidth_limit columns in projects table

github issue https://github.com/storj/storj/issues/5185

Change-Id: Ia8c9b190a2bdb684ac9d43af8f2159027d65aa88
2022-10-06 18:20:39 -06:00
paul cannon
a57cbc4e0b satellite/satellitedb: drop _gob columns
This concludes the saga that began with commit c053bdbd, migrating away
from using gob-encoded big.Float values in the database to using
integers with an implied number of decimal places.

Nothing is using or accessing or expecting the _gob columns to exist
anymore. The transition code and the migration chore are gone. All that
remains is to drop the columns.

Change-Id: I9b15ee52f7781510a6dc91cf7c54f7f9022b1210
2022-08-26 19:45:22 +00:00
Wilfred Asomani
c59d0d415c satellitedb: add signup_captcha column to users table
The signup_captcha column will hold the captcha scores of new users.

see: https://github.com/storj/storj/issues/5067

Change-Id: Ia322af29a3b5b019b417843272506a3dbd1397e4
2022-08-17 14:31:52 -05:00
paul cannon
0dcc0a9ee0 satellite/reputation: reconfigure lambda and alpha
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.

For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at

    https://forum.storj.io/t/tuning-audit-scoring/14084

In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.

Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.

Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
2022-08-17 18:52:53 +00:00
dlamarmorgan
8eec61d315 satellitedb: add timestamp index to billing table
Adding an index to the timestamp field of the billing transaction table to improve query performance. This should prevent having to do a full table scan when we query for the last billing transaction of a particular source and/or type.

Change-Id: I581f09494cc8662a12efba4302022a07121ba309
2022-08-16 22:50:02 +00:00
dlamarmorgan
ac241501ca satellitedb: add wallet address index to storjscan_wallet table
Adding an index to the wallet address field of the storjscan wallet table to improve query performance. This should prevent having to do a full table scan when we query for one or more wallet addresses for a given user in our queries.

Change-Id: Ic1b5d06c2258489e5464d186fed5270172f8cba5
2022-08-15 18:05:06 +00:00
kimbotsao
c3e7b1409d satellitedb: add salt to projects table
Change-Id: I53691849efc169c495566648c12ae3d639614c3d
2022-08-11 12:16:53 -04:00
dlamarmorgan
c8b72c8f71 satellite/{payments/billing,satellitedb}: refactor billing DB
Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances

Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838

satellite/{payments/billing,satellitedb}: refactor billing DB

Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances

Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838
2022-08-03 17:52:32 +00:00
Clement Sam
f2f046af7a satellite/satellitedb: add interval_end_time column to accounting_rollups table
Adding an interval_end_time column to the accounting_rollups table
to keep the last interval_end_time for each daily storage tallies.

Updates https://github.com/storj/storj/issues/4178

Change-Id: If7a8210c5e9fe2fc9df84b137a8b6e3db2471c58
2022-07-27 17:53:27 +00:00
Cameron
023a1c6271 satellite/satellitedb: add public_id column and index to projects table
github issue: https://github.com/storj/storj/issues/4861

Change-Id: I3d1928ca51c825a4c1589db5ae55902e57f00c3b
2022-07-05 18:39:29 +00:00