Commit Graph

406 Commits

Author SHA1 Message Date
dlamarmorgan
fd13cf1058 dbx/satellitedb: add default_versioning field for projects
Change-Id: I294a85c244976493b490a58f17536f06f6c01306
2023-11-29 15:29:14 +00:00
Wilfred Asomani
26574fb2bd satellite/db: add account activation columns
This change adds a new column to the user table, which will hold a
random code for account activation. And another to hold the signup
request ID as another layer of verification.

Issue: #6428

Change-Id: Icd46cb5d8fc76102264d599aca27686cd8b2e84e
2023-11-22 09:57:52 +00:00
Michal Niewrzal
573ce712f2 satellite/bloomfilter: don't create BF for disqualified nodes
Currently we have large set of nodes that are already disqualified and
we are not sending bloom filters to them. The issue is that we are still
generating filters for them while garbage collection process. Even if
we have only segment with one piece which was stored on this node. This
consumes additional memory and processing powers.

This change is changing logic behind `AllPieceCounts` (renamed to
ActiveNodesPieceCounts) to return piece count for all nodes except disqualified one (even with piece count = 0). With this change we can
modify GC observer to skip nodes that where not returned by
ActiveNodesPieceCounts.

Change-Id: Ic75159135abe535084d8aeee560bb801a4a03e17
2023-11-22 07:07:51 +00:00
Clement Sam
b8b5afba73 satellite/satellitedb: remove dbx references to nodes.type column
Updates https://github.com/storj/storj/issues/5426

Change-Id: I36f12f354d9f2d575e4bd99b0520892f846bdc19
2023-11-20 18:08:37 +00:00
Clement Sam
f5164c78cd Revert "satellite/satellitedb: drop type column on nodes table"
This reverts commit 2b4f347c33.

Reason for revert: not a zero-downtime migration. We need to deploy the dbx changes before completely dropping the column.

Change-Id: Ic9215650242b7848d54f8a5f863a13bc18b60149
2023-11-20 13:36:44 +00:00
Clement Sam
2b4f347c33 satellite/satellitedb: drop type column on nodes table
Updates https://github.com/storj/storj/issues/5426

Change-Id: If554f30124f234d1c50b2e8059e3d1dfec389ebb
2023-11-20 10:09:17 +00:00
Wilfred Asomani
cd8e9bd044 satellite/{payment,console,analytics} extend freeze functionality for legal freeze
This change extends the account freeze functionality account for legal
freezes as well. This is a freeze event for accounts to be put on hold
for legal review. It also sets the LegalHold status on the affected
user.

Issue: storj/storj-private#492

Change-Id: I8c733269b5cfb647c840379a6bb033da120c8280
2023-11-14 11:09:25 +00:00
Wilfred Asomani
513c3cc632 satellite/admin: list users pending deletion
This change adds an endpoint to the admin API and UI to get a list of
users pending deletion and have no unpaid invoice.

Issue: #6410

Change-Id: I906dbf9eee9e7469e45f0c622a891867bf0cc201
2023-10-30 19:11:16 +00:00
dlamarmorgan
fe9afad8cd
satellite/{buckets,satellitedb}: add versioning to buckets.DB
Add the DB schema changes and basic implementation methods to support
bucket level configuration of object versioning.

See bucket-level-versioning-configuration design doc for more details.

Change-Id: I4e920a20a403b3157970a34eb619d827a4007845
2023-10-24 11:16:31 -07:00
Clement Sam
67bd36ae4d satellite/satellitedb: remove nodes.type from indices on nodes table
Since we no longer need the type column and hopefully,
there are no queries using it, we recreate all indices
that reference the nodes.type column.

Updates https://github.com/storj/storj/issues/5426

Change-Id: I610ccaf474a6f4031e166b79a6d649c4b138e338
2023-10-20 14:48:36 +00:00
Wilfred Asomani
be025a40cc satellite/{db,console}: add days column to account freeze table
This change adds a new column to the account_freeze_events table,
days_till_escalation, which is the number of days till the freeze event
is escalated. E.g.: 30 days from billing warning event to billing
freeze event.

Issue: #6382

Change-Id: I15cfd6f1208e641d8c380bef61717031befdad73
2023-10-19 18:18:44 +00:00
Michal Niewrzal
d7af97c919 satellite/satellitedb: move IterateBucketLocations sql to dbx
This is attempt to move query back to dbx.

It also removes one unused method.

Change-Id: I8182dd8ecf794cdf0cb3158c36cc00810fc683df
2023-10-17 10:30:51 +00:00
Michal Niewrzal
281edfa585 satellite/satellitedb: add index to bucket_storage_tallies table
Additional index on 'interval_start' to support data analysis.

Change-Id: I373787871da594aa1b4ebd66cb684484221eb4d5
2023-10-16 15:09:24 +00:00
Moby von Briesen
a2acf359ad satellite/satellitedb: Remove dbx references to partner_id
This column is no longer used, but it is referenced in dbx. This change
removes those references and adds an exception to the migration test to
disregard this column in schema comparison.

After this change is deployed, we will need to follow up with a
migration to remove the column in a later release, and remove the
exception in the migration test.

Related to https://github.com/storj/storj/issues/5432

Change-Id: I168fb57244b347901d1ed9c7813c1338554ee644
2023-10-04 15:33:44 +00:00
dlamarmorgan
8a1bedd367 satellite/payments/{billing,stripe}: handle pending invoice payments
Currently, pending invoice payments that are made using a users token
balance can get stuck in a pending state if the invoice is not able
to be paid appropriately in stripe. This change addresses these stuck
token invoice payments by attempting to transition them to failed
if the invoice cannot be paid.

Change-Id: I2b70a11c97ae5c733d05c918a1082e85bb7f73f3
2023-10-03 16:12:39 +00:00
Michal Niewrzal
2c76c1e6ae satellite/satellitedb: alter bucket_metainfos primary key
Current bucket_metainfos table schema was far from optimal. All
operations are using project_id and name to find bucket but primary key
was on id field which is completely not used. Turns out that secondary
index was not as fast as it could be because each select was also joining
primary index which was slowing down whole query.

Primary key is changed to project_id/name. Other indexes are dropped
and id column is now nullable which gives us space for dropping it
completely from schema and code later.

Change-Id: I477056a4243d39e39489f1473de01ded1ed24bbb
2023-10-03 12:23:00 +00:00
Clement Sam
89d682f49f satellite/metainfo: prevent internal DB errors in Public API
Resolves https://github.com/storj/storj/issues/6081

Change-Id: I0e530db39947138dcafc1b6bd1710ff1ca96b8c5
2023-09-12 15:12:44 +00:00
Márton Elek
afa5c54a35 satellite/satellitedb: add placement column to repair_queue
It makes it possible to run dedicated repair worker for different placement definitions.

https://github.com/storj/storj-private/issues/400

Change-Id: I376da867da5dbb4ab392d5f86c766f7543c32ee6
2023-09-12 11:29:18 +00:00
Egon Elbre
487f64e164 satellite/satellitedb,multinode/multinodedb: update to latest dbx
Change-Id: I500df6d0541706c3960d4560721c3783d0d049ff
2023-09-11 17:21:02 +00:00
Egon Elbre
87bfb3b02b satellite/satellitedb: don't generate rx
I'm not sure what Rx is, however, we aren't using it --
so let's remove it for now.

Change-Id: I9caacbc150479f93945477101528a4fd60ea865f
2023-09-11 16:00:15 +00:00
Márton Elek
8b4387a498 satellite/satellitedb: add tag information to nodes selected for upload/downloads
Change-Id: I0fa7daebcf83f7949726e5fffe68e0bdc6fd1d7a
2023-07-07 07:54:16 +00:00
Márton Elek
500b6244f8
satellite/satellitedb: create table for node tags
Change-Id: I884bb740974e6b8241aa6b85faf266b85fe892d4
2023-07-05 09:38:53 +02:00
Jeremy Wharton
22f8b029b9 satellite/console: fix transaction error when inviting project members
The SQL transaction that inserted project invitations relied on the
error result of one of its statements in order to determine whether an
invitation should be updated. This was inappropriate since any errors
returned from a transaction statement should end the transaction
immediately. This change resolves that issue.

Change-Id: I354e430df293054d8583fb4faa5dc1bcf9053836
2023-06-23 20:17:37 +00:00
Jeremy Wharton
d18f4f7d99 satellite/console: don't delete expired project invitations
This change removes instances of project invitation deletion due to
expiration because we now want such invitations to be accessible beyond
their expiration date. In the future, project members will be able to
view and resend expired invitations within the Team page in the
satellite frontend.

References #5752

Change-Id: If24a9637945874d719b894a66c06f6e0e9805dfa
2023-06-20 22:42:50 -05:00
Vitalii
1eee2fad69 satellite/{db, admin}: added endpoints to update user's and project's user_agent
Added backend (for now) implementation for updating user's and projects's user_agent using admin API.
Updating both user and project also updates bucket_metainfo and value_attribution tables.

Issue:
https://github.com/storj/storj-private/issues/297

Change-Id: I40244bbaa08b46834c1b1d0720e7d84d0c2a0330
2023-06-16 19:41:05 +00:00
Egon Elbre
edbea5efe1 go.mod: bump to pgx/v5
Change-Id: I31cf3bec1d7db94f0f612f6ed04b782f8b04d876
2023-06-14 18:32:54 +03:00
Jeremy Wharton
bb620e746b satellite/satellitedb: add index for project_members project_id column
An index has been added on the project_id column of the project_members
satellite database table so that we can retrieve members of a project
without performing a full table scan.

References #5855

Change-Id: I1cc30686f836c8fd1aa319247ce857a2392e7a52
2023-06-11 18:12:33 -05:00
Márton Elek
5c26661cd3 dbx/satellitedb: introduce new default_placement fields for users/projects
https://github.com/storj/storj/issues/5879

Change-Id: Ie6d50fd94c09f43c12f895444ed2a86f0820e01c
2023-06-08 10:02:10 +00:00
JT Olio
1437257dbf satellite: save and return which node features are enabled
current feature is if tcp fastopen was successfully enabled

Change-Id: Ide251863a9790b0fbebdf2e82dfd2afa8f25c408
2023-06-06 21:13:29 +00:00
Márton Elek
52598d1545 satellite/admin: introduce new placement rule
Change-Id: I582d38febd6187e06a46f14fd0f37531a2139c99
2023-06-05 13:34:26 +00:00
Moby von Briesen
563239d555 satellite/satellitedb: Add value_attributions.partner_id back
There were some dbx-generated queries on the old satellite version that
still reference this column, so we need to add it back until the next
version.

Change-Id: I78b19336d9ca0384936d6cc11f5c50e579b4f2ab
2023-05-31 17:31:33 -04:00
Wilfred Asomani
4791a6422d satellite/console: add STORJ bonus to transactions
This change includes STORJ bonuses to the list of transactions returned
by the /wallet/payments endpoint.

Issue: https://github.com/storj/storj/issues/5755

Change-Id: Icc95c2cb9dd9fc5ee7a373e68c1cf8a991e1aa58
2023-05-26 20:36:30 +00:00
Moby von Briesen
919278dbf9 satellite/satellitedb: drop partnerID columns
The last code referencing these columns was removed as of satellite
release v1.79, so it is safe to remove them now.

Resolves https://github.com/storj/storj/issues/5432

Change-Id: I2e9d641b2511a61e0b9482ef0f4955a73c290709
2023-05-24 14:38:57 +00:00
Jeremy Wharton
607b120116 satellite/satellitedb: add inviter ID column to project invites table
The project member invitations table has been modified to contain a
column for the ID of the user who sent the invitation. This ID is
required for us to return information about the inviter to the
satellite frontend.

References #5855

Change-Id: I928d987a8db2340f731ca65ce30173d4f90a9837
2023-05-17 15:46:40 +00:00
Wilfred Asomani
81b2b067e6 satellite/satellitedb/dbx/gen: update to work with updated dbx
This change updates the replacer in satellite/satellitedb/dbx/gen/main.go
to work with an updated dbx.

Change-Id: I08e89d6d27e6f1d435416105fe5f622009add7ad
2023-05-15 14:53:06 +00:00
Michal Niewrzal
87d0789691 satellite/payments/stripe: avoid full table scan while listing records
Stripe invoice project records while listing are causing full table scan
because of OFFSET caluse. This change is refactoring query to list using
cursor.

Change-Id: I6b73b9b2815173d7ef02cf615408778476eb3b7b
2023-05-09 14:16:55 +00:00
Michal Niewrzal
fa16d947c9 satellite/satellitedb: add index on owner_id column for projects table
We have method which is getting projects owned by specific user but it's
causing full table scan because we don't have index on owner_id column.

Change-Id: Icb71c9ac5b73104a52241ed8ba126c995c10811f
2023-05-09 08:57:32 +00:00
Moby von Briesen
432b12d1c4 satellite/satellitedb: make value_attributions.partner_id nullable
There is still a reference to partner_id in a query, which we cannot
move until dropping the "not null" constraint for it. This change adds
that migration.

Related to https://github.com/storj/storj/issues/5432

Change-Id: I98802a6e1bd59f3d9214de3db6688d9daf664a70
2023-05-05 23:42:50 +00:00
Jeremy Wharton
defb9eae82 satellite/satellitedb: add table for project invitations
A table for storing pending project member invitations has been added.
This table is required to satisfy our new project invitation UX revamp.

References #5766

Change-Id: I6f948de66ed5b4dc81532564958ff7f48533cad2
2023-04-26 11:26:12 -05:00
dlamarmorgan
5fc493b276 metainfo: per-project and user-agent eventkit-based usage tracking
Add eventkit events for each satellite RPC type, tagged with user agent and public project id.

Change-Id: I838401683298d4594495de18563c55031e82f881
2023-04-25 15:13:43 +00:00
Michal Niewrzal
a21afeddd1 satellite/payments/stripe: avoid full table scan while listing
Query to list (with pages) stripe customers were doing full table scan
because Offset clause was used. This refactoring changed listing to
use cursor instead Offset.

Change-Id: I14688e6c533bc932ba0d209a061562f080b4cf54
2023-04-13 12:36:31 +02:00
Cameron
549732228d satellite/satellitedb: add package info to customers db
Add columns package_plan and purchased_package_at to stripe_customers
table and add methods to update and select these values from console
service and payments accounts.

Change-Id: I1e89909055cc3054bfb7baa33c9dca3dfdc7336e
2023-03-31 15:51:30 +00:00
Clement Sam
e5c43722dc storagenode/pieces: introduce FileWalker
FileWalker implements methods to walk over pieces in
in a storage directory.

This is just a refactor to separate filewalker functions
from pieces.Store. This is needed to simplify the work
to create a separate filewalker subprocess and reduce the
number of config flags passed to the subprocess.

You might want to check https://review.dev.storj.io/c/storj/storj/+/9773

Change-Id: I4e9567024e54fc7c0bb21a7c27182ef745839fff
2023-03-30 18:33:52 +00:00
Wilfred Asomani
ed70a03844 satellite/{console,db,analytics}: better warning handling
This handles cases where a user is warned and triggers payment for their
account. Previously, only a frozen account will trigger this payment,
and will be unfrozen on successful payment. Now, accounts in warning
state trigger payments and are removed from that state on successful payment.

Issue: https://github.com/storj/storj/issues/5691

Change-Id: Icc2107f5d256657d176d8b0dd0a43a470eb01277
2023-03-28 14:30:38 +00:00
Wilfred Asomani
7c68b51c82 satellite/{console,db}: add onboardings columns
This change adds onboarding_start, onboarding_end and onboarding_step
columns to the user_settings table. the first two are used to determine
if a user should go through onboarding, the last will be used to as the
step a user got to before exiting onboarding without finishing.

Issue: https://github.com/storj/storj/issues/5660

Change-Id: I8070c880d0d2fc22086f24087c962f57c695cc50
2023-03-23 17:50:31 +00:00
JT Olio
2a63225b98 satellite/{contact,satellitedb}: preserve node message debounce support
Change-Id: I453ad35fda4e61d068db0c476dd86b50d7f2d705
2023-03-20 16:13:06 +00:00
Moby von Briesen
0980a2f9ce satellitedb: Add index to users table
Add a combined index on normalized_email,status to improve performance of
common "get user" query used for the satellite UI.

Change-Id: I24a20d7826e0a68a68c2f95b5847eb819921e7c0
2023-03-16 09:08:49 -04:00
Michal Niewrzal
ee720040c9 satellite/metainfo: use project limit cache with limiter
Metainfo needs to know rate and burst limit to be able to limit users
requests. We made cache for per project limiter but to make single
instance we need to know about limits. So far we were doing direct DB
call to get rate/burst limit for project but it's generating lots of
DB requests and can be easily cached as we even have project limit cache.

This change extends project limit cache with rate/burst limit and starts
using this change while creating project limiter instance for metainfo.

Because data size kept in project limit cache is quite small this change
also bumps a bit default capacity of the cache.

Fixes https://github.com/storj/storj/issues/5663

Change-Id: Icb42ec1632bfa0c9f74857b559083dcbd054d071
2023-03-14 08:11:11 +00:00
Michal Niewrzal
d54ccfa92b satellite/satellitedb: change bucket_bandwidth_rollups primary key
In the past we set incorrect primary key for bucket_bandwidth_rollups
table and bucket name was first column of key instead of project id.
This change fixes this problem.

We will alter primary key manually for production satellites. That's
why migration code is checking if change was already applied.

https://github.com/storj/storj/issues/5332

Change-Id: I7dd555eb0c2f6fda1c90eaed08fa847b7f273bc7
2023-03-03 16:25:35 +00:00
Lizzy Thomson
54f0f47839 satellite/satellitedb: add passphrase_prompt column to user_settings
Add passphrase_prompt column to user_settings table to store a boolean
whether a user would like to be prompted to enter a passphrase when
entering a new project.

Issue https://github.com/storj/storj/issues/5613

Change-Id: I5d086df277aff458453343c7c38a379e7b1b8bf9
2023-03-01 14:53:18 -07:00