Commit Graph

386 Commits

Author SHA1 Message Date
Márton Elek
8b4387a498 satellite/satellitedb: add tag information to nodes selected for upload/downloads
Change-Id: I0fa7daebcf83f7949726e5fffe68e0bdc6fd1d7a
2023-07-07 07:54:16 +00:00
Márton Elek
500b6244f8
satellite/satellitedb: create table for node tags
Change-Id: I884bb740974e6b8241aa6b85faf266b85fe892d4
2023-07-05 09:38:53 +02:00
Jeremy Wharton
22f8b029b9 satellite/console: fix transaction error when inviting project members
The SQL transaction that inserted project invitations relied on the
error result of one of its statements in order to determine whether an
invitation should be updated. This was inappropriate since any errors
returned from a transaction statement should end the transaction
immediately. This change resolves that issue.

Change-Id: I354e430df293054d8583fb4faa5dc1bcf9053836
2023-06-23 20:17:37 +00:00
Jeremy Wharton
d18f4f7d99 satellite/console: don't delete expired project invitations
This change removes instances of project invitation deletion due to
expiration because we now want such invitations to be accessible beyond
their expiration date. In the future, project members will be able to
view and resend expired invitations within the Team page in the
satellite frontend.

References #5752

Change-Id: If24a9637945874d719b894a66c06f6e0e9805dfa
2023-06-20 22:42:50 -05:00
Vitalii
1eee2fad69 satellite/{db, admin}: added endpoints to update user's and project's user_agent
Added backend (for now) implementation for updating user's and projects's user_agent using admin API.
Updating both user and project also updates bucket_metainfo and value_attribution tables.

Issue:
https://github.com/storj/storj-private/issues/297

Change-Id: I40244bbaa08b46834c1b1d0720e7d84d0c2a0330
2023-06-16 19:41:05 +00:00
Egon Elbre
edbea5efe1 go.mod: bump to pgx/v5
Change-Id: I31cf3bec1d7db94f0f612f6ed04b782f8b04d876
2023-06-14 18:32:54 +03:00
Jeremy Wharton
bb620e746b satellite/satellitedb: add index for project_members project_id column
An index has been added on the project_id column of the project_members
satellite database table so that we can retrieve members of a project
without performing a full table scan.

References #5855

Change-Id: I1cc30686f836c8fd1aa319247ce857a2392e7a52
2023-06-11 18:12:33 -05:00
Márton Elek
5c26661cd3 dbx/satellitedb: introduce new default_placement fields for users/projects
https://github.com/storj/storj/issues/5879

Change-Id: Ie6d50fd94c09f43c12f895444ed2a86f0820e01c
2023-06-08 10:02:10 +00:00
JT Olio
1437257dbf satellite: save and return which node features are enabled
current feature is if tcp fastopen was successfully enabled

Change-Id: Ide251863a9790b0fbebdf2e82dfd2afa8f25c408
2023-06-06 21:13:29 +00:00
Márton Elek
52598d1545 satellite/admin: introduce new placement rule
Change-Id: I582d38febd6187e06a46f14fd0f37531a2139c99
2023-06-05 13:34:26 +00:00
Moby von Briesen
563239d555 satellite/satellitedb: Add value_attributions.partner_id back
There were some dbx-generated queries on the old satellite version that
still reference this column, so we need to add it back until the next
version.

Change-Id: I78b19336d9ca0384936d6cc11f5c50e579b4f2ab
2023-05-31 17:31:33 -04:00
Wilfred Asomani
4791a6422d satellite/console: add STORJ bonus to transactions
This change includes STORJ bonuses to the list of transactions returned
by the /wallet/payments endpoint.

Issue: https://github.com/storj/storj/issues/5755

Change-Id: Icc95c2cb9dd9fc5ee7a373e68c1cf8a991e1aa58
2023-05-26 20:36:30 +00:00
Moby von Briesen
919278dbf9 satellite/satellitedb: drop partnerID columns
The last code referencing these columns was removed as of satellite
release v1.79, so it is safe to remove them now.

Resolves https://github.com/storj/storj/issues/5432

Change-Id: I2e9d641b2511a61e0b9482ef0f4955a73c290709
2023-05-24 14:38:57 +00:00
Jeremy Wharton
607b120116 satellite/satellitedb: add inviter ID column to project invites table
The project member invitations table has been modified to contain a
column for the ID of the user who sent the invitation. This ID is
required for us to return information about the inviter to the
satellite frontend.

References #5855

Change-Id: I928d987a8db2340f731ca65ce30173d4f90a9837
2023-05-17 15:46:40 +00:00
Wilfred Asomani
81b2b067e6 satellite/satellitedb/dbx/gen: update to work with updated dbx
This change updates the replacer in satellite/satellitedb/dbx/gen/main.go
to work with an updated dbx.

Change-Id: I08e89d6d27e6f1d435416105fe5f622009add7ad
2023-05-15 14:53:06 +00:00
Michal Niewrzal
87d0789691 satellite/payments/stripe: avoid full table scan while listing records
Stripe invoice project records while listing are causing full table scan
because of OFFSET caluse. This change is refactoring query to list using
cursor.

Change-Id: I6b73b9b2815173d7ef02cf615408778476eb3b7b
2023-05-09 14:16:55 +00:00
Michal Niewrzal
fa16d947c9 satellite/satellitedb: add index on owner_id column for projects table
We have method which is getting projects owned by specific user but it's
causing full table scan because we don't have index on owner_id column.

Change-Id: Icb71c9ac5b73104a52241ed8ba126c995c10811f
2023-05-09 08:57:32 +00:00
Moby von Briesen
432b12d1c4 satellite/satellitedb: make value_attributions.partner_id nullable
There is still a reference to partner_id in a query, which we cannot
move until dropping the "not null" constraint for it. This change adds
that migration.

Related to https://github.com/storj/storj/issues/5432

Change-Id: I98802a6e1bd59f3d9214de3db6688d9daf664a70
2023-05-05 23:42:50 +00:00
Jeremy Wharton
defb9eae82 satellite/satellitedb: add table for project invitations
A table for storing pending project member invitations has been added.
This table is required to satisfy our new project invitation UX revamp.

References #5766

Change-Id: I6f948de66ed5b4dc81532564958ff7f48533cad2
2023-04-26 11:26:12 -05:00
dlamarmorgan
5fc493b276 metainfo: per-project and user-agent eventkit-based usage tracking
Add eventkit events for each satellite RPC type, tagged with user agent and public project id.

Change-Id: I838401683298d4594495de18563c55031e82f881
2023-04-25 15:13:43 +00:00
Michal Niewrzal
a21afeddd1 satellite/payments/stripe: avoid full table scan while listing
Query to list (with pages) stripe customers were doing full table scan
because Offset clause was used. This refactoring changed listing to
use cursor instead Offset.

Change-Id: I14688e6c533bc932ba0d209a061562f080b4cf54
2023-04-13 12:36:31 +02:00
Cameron
549732228d satellite/satellitedb: add package info to customers db
Add columns package_plan and purchased_package_at to stripe_customers
table and add methods to update and select these values from console
service and payments accounts.

Change-Id: I1e89909055cc3054bfb7baa33c9dca3dfdc7336e
2023-03-31 15:51:30 +00:00
Clement Sam
e5c43722dc storagenode/pieces: introduce FileWalker
FileWalker implements methods to walk over pieces in
in a storage directory.

This is just a refactor to separate filewalker functions
from pieces.Store. This is needed to simplify the work
to create a separate filewalker subprocess and reduce the
number of config flags passed to the subprocess.

You might want to check https://review.dev.storj.io/c/storj/storj/+/9773

Change-Id: I4e9567024e54fc7c0bb21a7c27182ef745839fff
2023-03-30 18:33:52 +00:00
Wilfred Asomani
ed70a03844 satellite/{console,db,analytics}: better warning handling
This handles cases where a user is warned and triggers payment for their
account. Previously, only a frozen account will trigger this payment,
and will be unfrozen on successful payment. Now, accounts in warning
state trigger payments and are removed from that state on successful payment.

Issue: https://github.com/storj/storj/issues/5691

Change-Id: Icc2107f5d256657d176d8b0dd0a43a470eb01277
2023-03-28 14:30:38 +00:00
Wilfred Asomani
7c68b51c82 satellite/{console,db}: add onboardings columns
This change adds onboarding_start, onboarding_end and onboarding_step
columns to the user_settings table. the first two are used to determine
if a user should go through onboarding, the last will be used to as the
step a user got to before exiting onboarding without finishing.

Issue: https://github.com/storj/storj/issues/5660

Change-Id: I8070c880d0d2fc22086f24087c962f57c695cc50
2023-03-23 17:50:31 +00:00
JT Olio
2a63225b98 satellite/{contact,satellitedb}: preserve node message debounce support
Change-Id: I453ad35fda4e61d068db0c476dd86b50d7f2d705
2023-03-20 16:13:06 +00:00
Moby von Briesen
0980a2f9ce satellitedb: Add index to users table
Add a combined index on normalized_email,status to improve performance of
common "get user" query used for the satellite UI.

Change-Id: I24a20d7826e0a68a68c2f95b5847eb819921e7c0
2023-03-16 09:08:49 -04:00
Michal Niewrzal
ee720040c9 satellite/metainfo: use project limit cache with limiter
Metainfo needs to know rate and burst limit to be able to limit users
requests. We made cache for per project limiter but to make single
instance we need to know about limits. So far we were doing direct DB
call to get rate/burst limit for project but it's generating lots of
DB requests and can be easily cached as we even have project limit cache.

This change extends project limit cache with rate/burst limit and starts
using this change while creating project limiter instance for metainfo.

Because data size kept in project limit cache is quite small this change
also bumps a bit default capacity of the cache.

Fixes https://github.com/storj/storj/issues/5663

Change-Id: Icb42ec1632bfa0c9f74857b559083dcbd054d071
2023-03-14 08:11:11 +00:00
Michal Niewrzal
d54ccfa92b satellite/satellitedb: change bucket_bandwidth_rollups primary key
In the past we set incorrect primary key for bucket_bandwidth_rollups
table and bucket name was first column of key instead of project id.
This change fixes this problem.

We will alter primary key manually for production satellites. That's
why migration code is checking if change was already applied.

https://github.com/storj/storj/issues/5332

Change-Id: I7dd555eb0c2f6fda1c90eaed08fa847b7f273bc7
2023-03-03 16:25:35 +00:00
Lizzy Thomson
54f0f47839 satellite/satellitedb: add passphrase_prompt column to user_settings
Add passphrase_prompt column to user_settings table to store a boolean
whether a user would like to be prompted to enter a passphrase when
entering a new project.

Issue https://github.com/storj/storj/issues/5613

Change-Id: I5d086df277aff458453343c7c38a379e7b1b8bf9
2023-03-01 14:53:18 -07:00
Wilfred Asomani
faeea88265 satellite/{db,analytics,payments}: add chore for auto account freeze
This change adds a new chore that will check for failed invoices and
potentially freeze corresponding accounts.
It makes slight modifications to stripemock.go and invoices.go (adding
stripe CustomerID to the Invoice struct).

Issue: https://github.com/storj/storj-private/issues/140

Change-Id: I161f4037881222003bd231559c75f43360509894
2023-03-01 09:31:27 +00:00
Egon Elbre
f13d0f7df0 satellite/satellitedb/dbx: document oauth tables
Change-Id: Ide327699f4b7b35f46c9356d558221ece5bd77a9
2023-02-28 09:07:58 +00:00
Egon Elbre
1cef2334ff satellite/satellitedb/dbx: document billing
Change-Id: Ief90cc0a6bebaa8df9dd9255565eaa42dc0abc9e
2023-02-27 15:13:23 +00:00
Egon Elbre
4aa519a69f satellite/satellitedb/dbx: document node payments
Change-Id: Ie65ec960f1683f5735a2b5959fedbde326007d04
2023-02-27 14:13:35 +00:00
Egon Elbre
1f439f5ae9 satellite/satellitedb/dbx: document audit tables
Change-Id: Iae0f2d7eaddb30cf43a3a7cd678f1dcfb8256a45
2023-02-27 09:56:39 +00:00
Egon Elbre
4a6e34bb2c satellite/satellitedb/dbx: document peer_identity table
Change-Id: I112ef00cb80638a8168dd46a531522b0785b537a
2023-02-24 14:06:46 +00:00
Egon Elbre
cf324c5b6e satellite/satellitedb/dbx: document node tables
Also, fix nodevents iota usage. Type is directly used in the database
hence we should avoid iota, because it's easy to mess-up the constant
values.

Change-Id: I32bf6d0502fbf3d18660dab5b1dd0b5627b444df
2023-02-24 14:05:55 +00:00
Egon Elbre
e4914e9b7a satellite/satellitedb/dbx: document accounting tables
Change-Id: I94478892a37623882d306a811e272f51865e48dd
2023-02-24 16:04:29 +02:00
Egon Elbre
4e70ef83fb satellite/satellitedb/dbx: split accounting.dbx by project and node
Change-Id: Ia3dee52abf98a41ea62d94af0a155f4f09661224
2023-02-24 09:08:20 +00:00
Egon Elbre
1a2ad602d7 satellite/satellitedb/dbx: document gracefulexit tables
Change-Id: Ia76b67fa85e797c1d75438ae882bdfe83a6ee057
2023-02-23 19:14:51 +02:00
Egon Elbre
491fa78840 satellite/satellitedb/dbx: add comments for node_reputation.dbx
Change-Id: If9c2bb605a9c562c114a59c3f3cec0b01b765689
2023-02-10 17:10:34 +02:00
Egon Elbre
dd3d051170 satellite/satellitedb/dbx: add docs for user.dbx
Change-Id: I346ceaedf6963169bb9d6d1435454016239d4cb1
2023-02-08 17:30:29 +02:00
Egon Elbre
a1bd36bb64 satellite/satellitedb/dbx: add docs for repair.dbx
Change-Id: Icffcdebb7dcc1d10eb8c97212acc6ae309b2f92c
2023-02-08 17:30:27 +02:00
Jeremy Wharton
49916b02f0 satellite/{console,satellitedb}: migration, methods for user settings
This change adds a database migration for a new table that stores
configurations for a user, the first of which is the session duration.
Database methods are implemented to interact with this table.

Resolves #5472

Change-Id: I01049265f385ea5de65907da1bc3bcf426d3c577
2023-02-07 21:48:29 +00:00
Michal Niewrzal
0b519fbd05 satellite/satellitedb: create index for interval_day for project_bandwidth_daily_rollup
To improve deletion of old entries in project_bandwidth_daily_rollup
we need index on `interval_day` column which is used to find those old
entries.

As an addition we are changing interval how often deletion is executed
from 7 to 1 day. We would like to have smaller portion of data to
delete.

Fixes https://github.com/storj/storj/issues/5465

Change-Id: Ie18ebe859887b93d6e4e6065a61fb9214c7ad27a
2023-02-03 10:02:58 +00:00
JT Olio
2753d5a32f satellite/overlay: keep track of noise info per node
Change-Id: Icef04c3e87dbf4bb57d3837274c323bf6dd2c81f
2023-02-01 23:03:35 -05:00
Jeremy Wharton
091ed29935 satellite/payments/stripecoinpayments: make price overrides per-bucket
This change causes the bucket's partner info to be used rather than the
user's when calculating project usage prices. This ensures that users
who own differently-partnered buckets will be charged correctly for
usage based on the specific bucket they are utilizing.
according to the bucket's partner.

Related to storj/storj-private#90

Change-Id: Ieeedfcc5451e254216918dcc9f096758be6a8961
2023-02-01 23:17:25 +00:00
Yaroslav Vorobiov
5644fb1a7e satellite/accounting/nodetally: add ranged loop
Add node tally ranged loop observer and partial.
Add node tally randed observer to range loop peer.
Add config flag to select which loop to use for node tally.
Update satellite core to use segement/ranged loop based on a flag.
Duplicate existing node tally test but using ranged loop.

Change-Id: I6786f1a16933463fab5f79601bf438203a7a5f9e
2023-01-17 13:50:18 +01:00
Egon Elbre
dc6e752dff satellite/satellitedb/dbx: document project.dbx
Updates https://github.com/storj/team-delivery/issues/1

Change-Id: I28bafdc4af4149489439fb5ab6e5bbe1d8f5186b
2023-01-13 11:24:34 +02:00
Michal Niewrzal
2876d5f20b satellite/satellitedb: drop project_bandwidth_rollups table
We are not using this table.

Change-Id: I3e4d5d0c2d8522b319caf48fea460b1cc35d5630
2023-01-12 10:23:48 +00:00