Commit Graph

87 Commits

Author SHA1 Message Date
Wilfred Asomani
7c68b51c82 satellite/{console,db}: add onboardings columns
This change adds onboarding_start, onboarding_end and onboarding_step
columns to the user_settings table. the first two are used to determine
if a user should go through onboarding, the last will be used to as the
step a user got to before exiting onboarding without finishing.

Issue: https://github.com/storj/storj/issues/5660

Change-Id: I8070c880d0d2fc22086f24087c962f57c695cc50
2023-03-23 17:50:31 +00:00
JT Olio
2a63225b98 satellite/{contact,satellitedb}: preserve node message debounce support
Change-Id: I453ad35fda4e61d068db0c476dd86b50d7f2d705
2023-03-20 16:13:06 +00:00
Moby von Briesen
0980a2f9ce satellitedb: Add index to users table
Add a combined index on normalized_email,status to improve performance of
common "get user" query used for the satellite UI.

Change-Id: I24a20d7826e0a68a68c2f95b5847eb819921e7c0
2023-03-16 09:08:49 -04:00
Michal Niewrzal
d54ccfa92b satellite/satellitedb: change bucket_bandwidth_rollups primary key
In the past we set incorrect primary key for bucket_bandwidth_rollups
table and bucket name was first column of key instead of project id.
This change fixes this problem.

We will alter primary key manually for production satellites. That's
why migration code is checking if change was already applied.

https://github.com/storj/storj/issues/5332

Change-Id: I7dd555eb0c2f6fda1c90eaed08fa847b7f273bc7
2023-03-03 16:25:35 +00:00
Lizzy Thomson
54f0f47839 satellite/satellitedb: add passphrase_prompt column to user_settings
Add passphrase_prompt column to user_settings table to store a boolean
whether a user would like to be prompted to enter a passphrase when
entering a new project.

Issue https://github.com/storj/storj/issues/5613

Change-Id: I5d086df277aff458453343c7c38a379e7b1b8bf9
2023-03-01 14:53:18 -07:00
Lizzy Thomson
f733cb255b satellite/satellitedb: drop unused last_verification_reminder column
(updated to migration step 225) remove users table column
last_verification_reminder and update migrate_test.go

Issue: https://github.com/storj/storj/issues/4560

Change-Id: Icbd8b70a8243b75a11c481a3b87df0ebe9402c5f
2023-02-08 16:23:09 +00:00
Jeremy Wharton
49916b02f0 satellite/{console,satellitedb}: migration, methods for user settings
This change adds a database migration for a new table that stores
configurations for a user, the first of which is the session duration.
Database methods are implemented to interact with this table.

Resolves #5472

Change-Id: I01049265f385ea5de65907da1bc3bcf426d3c577
2023-02-07 21:48:29 +00:00
Michal Niewrzal
0b519fbd05 satellite/satellitedb: create index for interval_day for project_bandwidth_daily_rollup
To improve deletion of old entries in project_bandwidth_daily_rollup
we need index on `interval_day` column which is used to find those old
entries.

As an addition we are changing interval how often deletion is executed
from 7 to 1 day. We would like to have smaller portion of data to
delete.

Fixes https://github.com/storj/storj/issues/5465

Change-Id: Ie18ebe859887b93d6e4e6065a61fb9214c7ad27a
2023-02-03 10:02:58 +00:00
JT Olio
2753d5a32f satellite/overlay: keep track of noise info per node
Change-Id: Icef04c3e87dbf4bb57d3837274c323bf6dd2c81f
2023-02-01 23:03:35 -05:00
Michal Niewrzal
2876d5f20b satellite/satellitedb: drop project_bandwidth_rollups table
We are not using this table.

Change-Id: I3e4d5d0c2d8522b319caf48fea460b1cc35d5630
2023-01-12 10:23:48 +00:00
Moby von Briesen
e26654c2a9 satellite/satellitedb: Drop unused tables
The `coupons`, `offers`, `coupon_usages`, `coupon_codes`, and
`user_credits` tables are not used anymore. They exist due to
legacy billing code which has already been removed.

Change-Id: Ie93af90a06e5247a74015af2a78a223d69249565
2023-01-10 17:00:15 +00:00
Cameron
021c98c17a satellite/satellitedb: create table account_freeze_events
create table for account freeze events

github issue: https://github.com/storj/storj/issues/5396

Change-Id: Ia4f93ce5caacd02b161853e3261d237bc715f23d
2022-12-13 19:09:25 +00:00
Cameron
602f0cc132 satellite/satellitedb: add node events column last_attempted
Also add a method to read a node event by id.

Change-Id: I08ca010e18a08253015e0e1280f791d0cc04d239
2022-12-01 18:35:06 +00:00
Cameron
5f6b4900d2 satellite/satellitedb: add columns last_offline_email and last_software_update_email
Change-Id: I8b5023bbbba55de633ce5ac3faf3361352c202ef
2022-11-11 14:43:59 -05:00
Fadila Khadar
553c1558a5 satellite/satellitedb: add 'contained' column to nodes table
We want to be able to exclude contained nodes from nodes selection. For this, we add a 'contained' column to the nodes table to track the containment status.

Fixes https://github.com/storj/storj/issues/5231

Change-Id: Id78e645f172145adcb8664646e8ebf14e218b57b
2022-11-09 11:11:37 -06:00
paul cannon
1b5d953fd9 satellite/satellitedb: add queue for verifications
Since the auditor will be moving to a different process from the
metainfo loop, we need a way of communicating which segments have
been chosen for audit. This queue will be that communication, for now.

Contrast this with the queue for _re_verifications in commit 9c67f62f.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: I9a269c7ef21e6c5e9c6e5e1f3db298fe159a8a79
2022-11-09 15:38:56 +00:00
Cameron
80bfa55b8e satellite/satellitedb: edit node_events migration to have different PK
Change node_events schema to use an id column as primary key rather than
node id because there can be multiple events per node id.

Change-Id: I518d8ef9ea658764876483e282a4058d3c4910f4
2022-10-28 22:59:28 +00:00
Cameron
6bf54bfaef satellite/satellitedb: add new table node_events
Add new table for node events. We can use this to notify node
operators of certain node events. Further, we can squash events for
multiple nodes with the same email into a single notification.

Change-Id: Icea6dd939df8fe4a98806bd79c014e21d239c43e
2022-10-28 19:16:25 +00:00
paul cannon
9c67f62fe3 satellite/satellitedb: add table for reverify queue
This table will be used as a queue for pieces that need to be reverified
(a regular audit timed out on the owning node, so now that node is
contained and we need to validate the piece before un-containing it).

Refs: https://github.com/storj/storj/issues/5228

Change-Id: I5dcd26b6adced8674cbd81884c1543a61ea9d4c8
2022-10-27 15:28:47 +00:00
Lizzy Thomson
ef04eb8bea satellite/satellitedb: add columns in projects table
add user_specified_usage_limit and user_specified_bandwidth_limit columns in projects table

github issue https://github.com/storj/storj/issues/5185

Change-Id: Ia8c9b190a2bdb684ac9d43af8f2159027d65aa88
2022-10-06 18:20:39 -06:00
paul cannon
a57cbc4e0b satellite/satellitedb: drop _gob columns
This concludes the saga that began with commit c053bdbd, migrating away
from using gob-encoded big.Float values in the database to using
integers with an implied number of decimal places.

Nothing is using or accessing or expecting the _gob columns to exist
anymore. The transition code and the migration chore are gone. All that
remains is to drop the columns.

Change-Id: I9b15ee52f7781510a6dc91cf7c54f7f9022b1210
2022-08-26 19:45:22 +00:00
Wilfred Asomani
c59d0d415c satellitedb: add signup_captcha column to users table
The signup_captcha column will hold the captcha scores of new users.

see: https://github.com/storj/storj/issues/5067

Change-Id: Ia322af29a3b5b019b417843272506a3dbd1397e4
2022-08-17 14:31:52 -05:00
paul cannon
0dcc0a9ee0 satellite/reputation: reconfigure lambda and alpha
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.

For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at

    https://forum.storj.io/t/tuning-audit-scoring/14084

In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.

Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.

Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
2022-08-17 18:52:53 +00:00
dlamarmorgan
8eec61d315 satellitedb: add timestamp index to billing table
Adding an index to the timestamp field of the billing transaction table to improve query performance. This should prevent having to do a full table scan when we query for the last billing transaction of a particular source and/or type.

Change-Id: I581f09494cc8662a12efba4302022a07121ba309
2022-08-16 22:50:02 +00:00
dlamarmorgan
ac241501ca satellitedb: add wallet address index to storjscan_wallet table
Adding an index to the wallet address field of the storjscan wallet table to improve query performance. This should prevent having to do a full table scan when we query for one or more wallet addresses for a given user in our queries.

Change-Id: Ic1b5d06c2258489e5464d186fed5270172f8cba5
2022-08-15 18:05:06 +00:00
kimbotsao
c3e7b1409d satellitedb: add salt to projects table
Change-Id: I53691849efc169c495566648c12ae3d639614c3d
2022-08-11 12:16:53 -04:00
dlamarmorgan
c8b72c8f71 satellite/{payments/billing,satellitedb}: refactor billing DB
Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances

Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838

satellite/{payments/billing,satellitedb}: refactor billing DB

Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances

Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838
2022-08-03 17:52:32 +00:00
Clement Sam
f2f046af7a satellite/satellitedb: add interval_end_time column to accounting_rollups table
Adding an interval_end_time column to the accounting_rollups table
to keep the last interval_end_time for each daily storage tallies.

Updates https://github.com/storj/storj/issues/4178

Change-Id: If7a8210c5e9fe2fc9df84b137a8b6e3db2471c58
2022-07-27 17:53:27 +00:00
Cameron
023a1c6271 satellite/satellitedb: add public_id column and index to projects table
github issue: https://github.com/storj/storj/issues/4861

Change-Id: I3d1928ca51c825a4c1589db5ae55902e57f00c3b
2022-07-05 18:39:29 +00:00
Yaroslav Vorobiov
da97fafb0c satellite/payments/storjscan: add payments DB
Change-Id: Ia6f0f62912c9a60d2c513459959634179eb9506c
2022-06-10 13:44:27 +01:00
dlamarmorgan
007d4190c2 satellite/{payments/billing,satellitedb}: Add payment billing DB
Add billing DB to the satellite. This DB will hold all transactions on the users account and can be used to compute the users current usable account balance.

Change-Id: I056416efc169e5e5e30c9f30cd8bc766b7bc8073
2022-05-27 08:56:31 +00:00
Márton Elek
5e8266d1de satellite/{payments/storjscan,satellitedb}: Add storjscan wallet DB
Add storjscan wallets DB to the satellite. For now this DB is a one to one mapping of the users account to a storjscan wallet that can be used by the account holder to make payments on their Storj account.

relates to https://github.com/storj/storj/issues/4347

Change-Id: I6e65b15817b90ceb75641244f9bf173c3b4228a7
2022-05-24 08:42:02 +00:00
paul cannon
aa728bd6ea satellite/satellitedb: add reputations.disqualification_reason
We added nodes.disqualification_reason recently, but we didn't add a
corresponding column in the reputations table (despite having a
corresponding `disqualified` column there).

Without this change, the (very useful and informative) assignments to
updateFields.DisqualificationReason in reputations.go have no effect.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I77404902ca64b56aed72f1de76b303fe82b76aab
2022-05-17 10:09:36 -05:00
Cameron
bfad64a6f0 satellite/satellitedb: add verification_reminders column to users table
When a new user registers, we send a verification request to their
email. Currently, if they do not verify their email, we take no further
action. We want to send these users reminders: one after about one day
and one after about 5 days. To do this we will use this new
verification_reminders column.

It will look something like this:

```
SELECT email FROM users
WHERE status = 0
   AND (
          (verification_reminders = 0 AND created_at < now() - 'INTERVAL 1d')
          OR (verification_reminders = 1 AND created_at < now() - 'INTERVAL 5d')
   )
```

Change-Id: If0620e08c97e9e337c9563481d665c5bd462693b
2022-05-16 16:37:16 +00:00
Moby von Briesen
24bbb7f236 satellite/satellitedb: Add webapp sessions table
Enables us to better manage sessions for satellite UI users.

Change-Id: I6695f8f1174d2e5fb970efddb0a8b14725e2b57e
2022-05-04 20:37:56 +00:00
Yaroslav Vorobiov
553ef9d640 satellite/satellitedb: suspended column removal migration
Last part of backwards compatible db migration to remove "suspended" column.
Removes exeption which removes "suspended" column in tests from `migrate_test.go`.
Adds DB migration to remove "suspended" column from 'nodes' and 'reputations' tables.

Change-Id: I02051279f6f4181e966c567919af0e774583f165
2022-04-20 17:59:11 +00:00
Vitalii
513ad99756 satellite/satellitedb: migrate wrongly affected users to have default limits
Migrate free tier users to have default limits if their limits were set to 0.
They were affected by incorrect working of Update user query.

Change-Id: I4c49c8d99b12dba2b9b0ab61b2175085976dcc95
2022-04-13 13:33:03 +00:00
Vitalii Shpital
b4cec4fd0b satellite/satellitedb: add columns to users table to control failed login attempts
Added failed_login_count and login_lockout_expiration columns to users table to control users failed login attempts.
We want to prevent brute forcing of user login so this is the first step.

Change-Id: I06b0b9f5415a1922e08cd9908893b2fd3c26bca0
2022-04-12 08:37:07 +00:00
paul cannon
d8733ddd40 satellite/satellitedb: stop using _gob columns
This sets the corresponding _numeric columns to be NOT NULL (it has been
verified manually that there are no more NULL _numeric values on any
known satellites, and it should be impossible with current code to get
new NULL values in the _numeric columns.

We can't drop the _gob columns immediately, as there will still be code
running that expects them, but once this version is deployed we can
finally drop them and be totally done with this crazy 5-step migration.

Change-Id: I518302528d972090d56b3eedc815656610ac8e73
2022-03-30 04:13:13 +00:00
Yaroslav Vorobiov
ca09a06680 satellitedb: revert migration for suspension column removal change
Change-Id: Iec8bbed8dd82d32c68ee7f9e64eec71ab8469373
2022-03-23 13:42:11 +00:00
Yaroslav Vorobiov
ddbbb0038b satellite/satellitedb: remove suspension column from nodes and reputations
Remove redundant suspension timestamp column from nodes and reputation tables.
Suspended timestamp was moved to unknown_audit_suspended and suspended column is
no longer used so there is no point in keeping both.

Change-Id: Ieea3f12141b33ec9efe7594f4c9dbc7e10675b0e
2022-03-21 16:56:12 +00:00
Michał Niewrzał
39ac90835f satellite/satellitedb: change default segment limit to 100M for paying users
We decided that we want to have segment limit for paying users high
enough to not have to change it too often.

Fixes https://github.com/storj/storj/issues/4590

Change-Id: Ic1c38bf3e2fcc000548ff4c7e7004647b39fbecf
2022-03-04 13:18:14 +00:00
paul cannon
3540f9b3ad satellite/satellitedb: phase 2 getting rid of gob encodings in db
For a thorough explanation of the overall transition, see the message on
commit c053bdbd70.

This change will rename the columns containing gob-encoded big.Floats
and add new columns which will contain the equivalent data in a more
sql-friendly format.

The change should *not* break already-running satellite processes,
because all functionality touching these tables has already been taught
to work with these new columns if it sees any "undefined column" errors.

Change-Id: I229324376533e383c5d05064b8aedad149cf825b
2022-02-09 12:25:46 +00:00
Michał Niewrzał
3a7fb4588f satellite/satellitedb: users/projects segment limit migration
We have here two migrations in fact. One is for existing users,
we need to check if its paying user (paid_tier) and set 1M for
them and 150K for others.
Second migration is to set limits for projects depends on owner.
If owner is a paying user (paid_tier) then project should have
1M limit, otherwise it should be 150K. In this case to make
migration faster initially projects table segment_limit was set
to 1M by default. With migration we are selecting all paying
users and we are setting 150K limit for all projects which owners
are not in paying users set.

Initially we had a concern if that query wil lbe quick enough to
be executed during deployment but after investigation CRDB
team confirms that this should take seconds for out DBs.

Fixes https://github.com/storj/team-metainfo/issues/70

Change-Id: I8be06e9f949b68b993e043cc15525e8483bf49ea
2022-02-01 14:35:12 +00:00
Cameron Ayer
251e432e5b satellite/satellitedb: drop contained from nodes and reputations
A few months ago we removed all references to the contained
column in nodes and reputations

bb21551a9c
and
56fe636123

But we never did the migration to drop the columns.
This commit will finally do that.

Change-Id: I82aa2f257b1fb14a2f1c4c4a1589f80895360ae4
2022-01-20 19:46:48 +00:00
Mya
309ac41bfc satellite/satellitedb/dbx: add schema for oauth tokens
Change-Id: Ia980766a8f86b662545e7d8cae6ecf631e199809
2022-01-18 09:38:12 -06:00
Mya
83f7d3e045 satellite/satellitedb/dbx: add schema for oauth clients
Change-Id: Ib08c7074803805cca8ffaedbd89da54504842af7
2022-01-18 09:38:12 -06:00
Malcolm Bouzi
12cb733b16 satellite/satellitedb: add last_verifcation_reminder column to users table
We want to issue a reminder to users when they don't verify their email within 24hrs of registering. This change only adds a column to the users table.

Change-Id: I92e2baeabf179338ffec01574d4752c0ccdba88b
2022-01-04 19:38:04 +00:00
Michał Niewrzał
3f0babffe8 satellite/{console,satellitedb}: add project segment limit to user
All limits we have for projects have also parent limits stored
with user data. New created project is first taking limits from
owner (user) limits.
This change is extending users table with project_segment_limit
column and adds functionality to get and set value for this
column.

Change-Id: Iff5e36c62b517652390b649fc05992475916ecff
2021-12-16 08:46:01 +00:00
Michał Niewrzał
73730b23e7 satellite/satellitedb: add segment_limit colum to projects table
We want to set maximum number of segments per
project. This change adds only column to projects table.
Default value 1M is set to make later migration easier as
we need to set 1M for paid tier users and 140K for free
tier users.

Change-Id: I8e83712e08c5bd91dfa59f652d17e45c14240a36
2021-12-02 12:15:16 +00:00