Commit Graph

1084 Commits

Author SHA1 Message Date
paul cannon
7652a598d8 satellite/audit: add GetAllContainedNodes method to ReverifyQueue
We will be needing an infrequent chore to check which nodes are in the
reverify queue and synchronize that set with the 'contained' field in
the nodes db, since it is easily possible for them to get out of sync.
(We can't require that the reverification queue table be in the same
database as the nodes table, so maintaining consistency with SQL
transactions is out. Plus, even if they were in the same database, using
such SQL transactions to maintain consistency would be slow and
unwieldy.)

This commit adds a method to the class representing the reverify queue
in the database, allowing us to get the list of every node that has at
least one record in the reverification queue.

Refs: https://github.com/storj/storj/issues/5431
Change-Id: Idce2633b3d63f2645170365e5cdeb2ea749fa9cb
2023-02-02 00:39:29 +00:00
Jeremy Wharton
091ed29935 satellite/payments/stripecoinpayments: make price overrides per-bucket
This change causes the bucket's partner info to be used rather than the
user's when calculating project usage prices. This ensures that users
who own differently-partnered buckets will be charged correctly for
usage based on the specific bucket they are utilizing.
according to the bucket's partner.

Related to storj/storj-private#90

Change-Id: Ieeedfcc5451e254216918dcc9f096758be6a8961
2023-02-01 23:17:25 +00:00
Cameron
0596651580 satellite/satellitedb: fix updating nodes.last_software_update_email
The CASE expression used to determine which value to set
last_software_update_email to did not have an ELSE clause. Therefore,
when the node is both below the minimum version and did not receive a
version update email (no condition is true), the value would be set to
NULL.

Additionally, replace `time.Now()` with `timestamp` in the check to
determine if the email cooldown has passed.

Change-Id: I2e2e93f1a865e123ed8b665be9621cebfb72236f
2023-02-01 17:25:58 -05:00
Michal Niewrzal
3b6e1123b8 satellite/orders: fix sorting rollups before inserting
Sorting by primary key before inserting data into DB is fixed.
Earlier we were sorting input slice of BucketBandwidthRollup but then
we were putting all entries into map to rollup input data. Iteration
over map with a range loop doesn't guarantee any specific order so we
were loosing sorted order when we were creating with this map slices to
use with DB insert.

New code is also using map but when map is full its sorting map keys
separately and iterates over them to get data from map.

https://github.com/storj/storj/issues/5332

Change-Id: I5bf09489b0eecb6858bf854ab387b660124bf53f
2023-02-01 12:17:25 +00:00
paul cannon
b6bcb32ecf satellite/reputation: more accurate "reputation changes" list
`overlay.(*Service).UpdateReputation()` takes a "reputationChanges"
parameter, a slice of node events indicating whether we think the node's
disqualification or suspension status is changing. This is necessary so
that the overlay service can notify the nodeevents DB about these
changes.

In several cases, however, this list of events is not constructed
correctly, because of missing information about the previous state.
In most cases, this is because the node was offline, and the order limit
creation functions (which usually obtain and return the prior reputation
status) ignored that node.

This change makes it so that all callers to
`overlay.(*Service).UpdateReputation()` can be expected to provide a
correct list of change events (as correct as feasible, given that we
can't lock the node's information in the database during the entire
operation).

It ended up that there was only one caller we needed to worry about, and
that was reputation.(*Service).ApplyAudit(). So the bulk of this change
is teaching that function how to recognize when the prior reputation
status was not filled in, and fill it in.

Refs: https://github.com/storj/storj/issues/5464
Change-Id: I52ce385fc9c0ce3b283b998d517998e7f4ec8792
2023-01-31 18:39:40 +00:00
Wilfred Asomani
3a714cefcf satellite: remove rewards package
Affected packages admin,attribution,console,metainfo,satellitedb,web,payments
This change removes the satellite/rewards package and its related usages.
It removes references to APIKeyInfo/PartnerID, Project/PartnerID
 and User/PartnerID.

Issue: https://github.com/storj/storj/issues/5432

Change-Id: Ieaa352ee848db45e94f85556febdbcf1444d8c3e
2023-01-31 11:46:50 +00:00
Lizzy Thomson
1bff41e6b3 cmd/tools: add migration tool to update salt column when null
Add migration tool (and test) to update salt column in projects table
with the SHA-256 hash of the project ID when null

Issue https://github.com/storj/storj-private/issues/66

Change-Id: Ib8d484ac8d6ee25859064d803e2ac8fb46b45921
2023-01-27 22:04:07 +00:00
Egon Elbre
e4b325537e satellite/satellitedb: missing rows.Close in error case
Change-Id: Ib1cae87171c302653771507cdc31d0c11a68f504
2023-01-26 15:21:48 +00:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
Yaroslav Vorobiov
5644fb1a7e satellite/accounting/nodetally: add ranged loop
Add node tally ranged loop observer and partial.
Add node tally randed observer to range loop peer.
Add config flag to select which loop to use for node tally.
Update satellite core to use segement/ranged loop based on a flag.
Duplicate existing node tally test but using ranged loop.

Change-Id: I6786f1a16933463fab5f79601bf438203a7a5f9e
2023-01-17 13:50:18 +01:00
Egon Elbre
dc6e752dff satellite/satellitedb/dbx: document project.dbx
Updates https://github.com/storj/team-delivery/issues/1

Change-Id: I28bafdc4af4149489439fb5ab6e5bbe1d8f5186b
2023-01-13 11:24:34 +02:00
Michal Niewrzal
a2a9dafa33 satellite/orders: don't store allocated bandwidth in
bucket_bandwidth_rollups table

We have performance problems with updating bucket_bandwidth_rollups. To
improve situation we can stop storing allocated bandwidth in this table.
This should reduce large number of updates which are comming from
metainfo endpoints, repair workers and audit.

Next step will be to drop `allocated` column completely from
bucket_bandwidth_rollups.

Allocated GET bandwidth is all we need and we are keeping it in
bucket_bandwidth_rollups table.

Change-Id: Ifdd26a89ba8262acbca6d794a6c02883ad0c0c9b
2023-01-12 13:21:02 +00:00
Michal Niewrzal
2876d5f20b satellite/satellitedb: drop project_bandwidth_rollups table
We are not using this table.

Change-Id: I3e4d5d0c2d8522b319caf48fea460b1cc35d5630
2023-01-12 10:23:48 +00:00
Moby von Briesen
e26654c2a9 satellite/satellitedb: Drop unused tables
The `coupons`, `offers`, `coupon_usages`, `coupon_codes`, and
`user_credits` tables are not used anymore. They exist due to
legacy billing code which has already been removed.

Change-Id: Ie93af90a06e5247a74015af2a78a223d69249565
2023-01-10 17:00:15 +00:00
Moby von Briesen
6d1f2fe307 satellite/satellitedb/dbx/gen: use go instead of sh
The shell script has different behavior on different operating systems,
and now has enough logic that it makes sense to put it into a go script.
This change replaces dbx/gen.sh with dbx/gen/main.go, which accomplishes
the same thing.

Change-Id: Ibb53ebf9c2475377aae1165cfe245140d7162026
2023-01-10 16:30:39 +00:00
Egon Elbre
b3448606fa satellite/satellitedb/dbx: document project_bucket.dbx
Updates https://github.com/storj/team-delivery/issues/1

Change-Id: I2931af073b7e2e014ce3881e6e5b6e05618ec49b
2023-01-10 17:28:29 +02:00
Egon Elbre
1688e4965b satellite/satellitedb/dbx: split into multiple files
The current satellitedb.dbx has grown quite big over time.
Split the single file into multiple smaller ones.

Change-Id: I8d6ca851b834ac60820aff565906f04aab661875
2023-01-09 16:24:31 +02:00
Lizzy Thomson
591615b90b satellite/satellitedb: remove references to last_verification_reminder
Removing all references to column last_verification_reminder which is to be removed, due to new column verification_reminders

Issue: https://github.com/storj/storj/issues/4560

Change-Id: I7c9a426e946c7aed58e62c1eef80629daf6b1272
2023-01-05 08:58:05 -07:00
Cameron
5138bb1e4f satellite/console: add public ID field to graphql Project
Add public ID field to graphql Project so it can be used on the front
end. Additionally public_id needed to be added to the ListByOwnerID sql
query which is called by graphql OwnedProjectsQuery.

github issue: https://github.com/storj/storj/issues/5408

Change-Id: I2ec04363c20493dc0f9c70b6d1610f724f18ec2f
2023-01-04 15:00:20 -05:00
Jeremy Wharton
471f9e4e10 satellite/{console,satellitedb}: add account freeze service
This change adds an account freeze service with methods for checking
if a user is frozen, freezing a user, and unfreezing a user.
Furthermore, methods for altering the usage limits of a user or project
have been implemented for use by the account freeze service.

Change-Id: I77fecfac5c152f134bec90165acfe4f1dea957e7
2022-12-16 22:24:58 +00:00
paul cannon
fc905a15f7 satellite/audit: newContainment->containment
Now that all the reverification changes have been made and the old code
is out of the way, this commit renames the new things back to the old
names. Mostly, this involves renaming "newContainment" to "containment"
or "NewContainment" to "Containment", but there are a few other renames
that have been promised and are carried out here.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I34e2b857ea338acbb8421cdac18b17f2974f233c
2022-12-16 17:59:52 +00:00
paul cannon
0342ca1aa6 satellite/audit: delete now-unused code
Now that we are doing scalable piecewise reverifications, the code for
handling the old way of doing things (containment, pending audits,
reporting, testing) can now be removed.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ief1a75f423eff682e8f3d57804e343b3409a6631
2022-12-16 14:53:39 +00:00
Jeremy Wharton
d069045058 satellite/{console,satellitedb}: implement account freeze event DB
This change implements DB methods for interacting with the
account_freeze_event table and introduces structures related to
account freeze events.

Change-Id: Ib125b31dfb754b2428212c39b780e14cfc7f97bf
2022-12-15 14:16:24 +00:00
Fadila Khadar
5c3a148d6e satellite/overlaycache: fix typo in UpdateCheckIn request
- fix a malformed SQL query
- add test to be sure we don't have this problem again.

Change-Id: I3fde8c59ba01335411e51d964bec95bc26cfc961
2022-12-14 22:21:45 +01:00
paul cannon
ad82e74cb7 satellite/satellitedb: add reverifyqueue to safelyPartitionableDBs
Adding this entry means that the database accessed as "reverifyqueue"
(`(*satelliteDBCollection).ReverifyQueue()`) can be located on a
different database host from the other databases, and things should
still work. There aren't any queries that do a JOIN on tables from
reverifyQueue and other things in satellitedb, for example.

This should really have been put here earlier, when reverifyqueue was
first added, but it's ok. This won't have any bearing on things until we
need to deploy to prod.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I76f68de79cd645c869f3dbfbe3b2c9c4f9359e8f
2022-12-13 21:14:17 +00:00
Cameron
021c98c17a satellite/satellitedb: create table account_freeze_events
create table for account freeze events

github issue: https://github.com/storj/storj/issues/5396

Change-Id: Ia4f93ce5caacd02b161853e3261d237bc715f23d
2022-12-13 19:09:25 +00:00
Lizzy Thomson
0afd3938c7 sat/console update the updateProject to set user specified limits
update the updateProject function to set user specified bandwidth and storage limits

fixes https://github.com/storj/storj/issues/5185

Change-Id: Ib4132487f6b7ea0afa7c57acfc358857b3e852d1
2022-12-13 16:44:41 +00:00
paul cannon
1854351da6 satellite/audit: teach Reporter about piecewise audits
The Reporter is responsible for processing results from auditing
operations, logging the results, disqualifying nodes that reached
the maximum reverification count, and passing the results on to
the reputation system.

In this commit, we extend the Reporter so that it knows how to process
the results of piecewise reverification audits.

We also change most reporter-related tests so that reverifications
happen as piecewise reverification audits, exercising the new code.

Note that piecewise reverification audits are not yet being done outside
of tests. In a later commit, we will switch from doing segmentwise
reverifications to piecewise reverifications, as part of the
audit-scaling effort.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I9438164ce1ea4d9a1790d18d0e1046a8eb04d8e9
2022-12-12 11:28:02 +00:00
paul cannon
378b8915c4 satellite/{satellitedb,audit}: add NewContainment
NewContainment will replace Containment later in this commit chain, but
for now it is not yet being used.

NewContainment will allow a node to be contained for multiple pending
reverify jobs at a time. It is implemented by way of the reverify queue.

Refs: https://github.com/storj/storj/issues/5231
Change-Id: I126eda0b3dfc4710a88fe4a5f41780618ec19101
2022-12-07 18:03:37 +00:00
Michal Niewrzal
8e030f4e30 satellite/accounting/tally: fix looping over all buckets
We have a bug where if number of buckets in the system will be
multiplication of batch size (2500) then loop that is going over
all buckets can run indefinitely.

Fixes https://github.com/storj/storj/issues/5374

Change-Id: Idd4d97c638db83e46528acb9abf223c98ad46223
2022-12-07 10:32:23 +00:00
paul cannon
d8df75186b satellite/satellitedb: change reverifyQueue.Insert()
It helps for the (*reverifyQueue).Insert() method to be idempotent (it
does not make sense for the same node to be under containment for the
same piece multiple times). This change allows for that, by adding an
`ON CONFLICT DO NOTHING` clause to the database query.

Refs: https://github.com/storj/storj/issues/5231
Change-Id: Id2839ee185d5396c0bc2f84ffad610df9786f6c7
2022-12-05 17:15:11 +00:00
paul cannon
2617925f8d satellite/audit: Split Reverifier from Verifier
Adding a new worker comparable to Verifier, called Reverifier; as the
name suggests, it will be used for reverifications, whereas Verifier
will be used for verifications.

This allows distinct logging from the two classes, plus we can add some
configuration that is specific to the Reverifier.

There is a slight modification to GetNextJob that goes along with this.

This should have no impact on operational concerns.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: Ie60d2d833bc5db8660bb463dd93c764bb40fc49c
2022-12-05 09:55:49 -06:00
Cameron
f8917c49aa satellite/satellitedb: generate random salt for new projects
Change-Id: I52970e1f32c6d4e9840d741c9ce4fbe61ac52b29
2022-12-02 01:24:14 +00:00
Cameron
08c9d745f1 satellite/nodeevents: take last_attempted into account when selecting
Previously, the node events chore would select based on the earliest
created_at. However, if for some reason this batch fails, it would still
be the next item to select. If there is a consistent error, the chore
would be stuck retrying the same batch over and over. Now instead
GetNextBatch orders by `last_attempted NULLS FIRST ASC, created_at ASC`.
If a batch fails during Notify, last_attempted is updated so we can move
on to a new batch if one exists.

Change-Id: Ia8458e05ac358d85b2f2c6d690f3d607d631be61
2022-12-01 20:08:38 +00:00
Cameron
76c22fa572 satellite/nodeevents: add method UpdateLastAttempted
Add method to update last_attempted column for a group of rows.

Change-Id: Ib6eadc9efc21368fccb12d1e824c90612c26a8f7
2022-12-01 19:38:54 +00:00
Cameron
b27c6dd032 satellite/nodeevents: add method GetByID
Add method GetByID to get a node event by its ID. For easier testing.

Change-Id: Ic9e0d201ffd682a2ff3bd7b4d02b58b06d0f8282
2022-12-01 19:05:53 +00:00
Cameron
602f0cc132 satellite/satellitedb: add node events column last_attempted
Also add a method to read a node event by id.

Change-Id: I08ca010e18a08253015e0e1280f791d0cc04d239
2022-12-01 18:35:06 +00:00
paul cannon
ed0fa59f23 satellite/overlay: add SetNodeContained() method
SetNodeContained() will change the contained flag in the nodes table,
which will affect whether nodes are selected for new uploads. This flag
_should_ correlate with whether or not a given node has any entries in
the reverification queue. However, the reverification queue is intended
to be 'safely partitionable' from the nodes table, so we can't enforce
that characteristic transactionally. But this is ok; there are no dire
consequences if they are out of sync.

We will be adding a chore that updates the contained flag based on the
contents of the reverification queue periodically, if something fails
to set it directly when appropriate.

Refs: https://github.com/storj/storj/issues/5231
Change-Id: I26460d8718dee63fd55d00a44568b2065fc8fe30
2022-12-01 12:43:40 +00:00
paul cannon
fba39b72b8 satellite/audit: add GetByNodeID to ReverifyQueue
GetByNodeID will allow querying the reverification queue to see if there
are any pending jobs for a given node ID. And thus, to see if that node
ID should be contained or not.

Some parameters on the other methods of the ReverifyQueue interface have
been changed to accept pointers; this was done ahead of the rest of the
changes for the reverification queue to better match the signatures of
the methods that these will replace once ReverifyQueue is actually being
used (meaning fewer changes to tests).

Refs: https://github.com/storj/storj/issues/5251
Change-Id: Ic38ce6d2c650702b69f1c7244a224f00a34893a1
2022-12-01 12:14:49 +00:00
Yaroslav Vorobiov
bb1e86c790 satellite: remove unused coinpayments code and chores
issue: https://github.com/storj/storj/issues/4824

Change-Id: I2e3e63151d1def96270279719f6eceda0acba66c
2022-11-30 16:24:48 +00:00
paul cannon
b612ded9af satellite/audit: help performance of pushing to audit queue
The audit chore will be pushing a large number of segments to be
audited, and the db might choke on that large insert when under load.

This change divides the insert up into batches, which can be sized
however is optimal for the backing database. It also arranges for
segments to be inserted in the order of the primary key, which helps
performance on some systems.

Refs: https://github.com/storj/storj/issues/5228

Change-Id: I941f580f690d681b80c86faf4abca2995e37135d
2022-11-29 15:37:49 +00:00
paul cannon
8b494f3740 satellite/audit: use db for auditor queue
As part of the effort of splitting out the auditor workers to their own
process, we are transitioning the communication between the auditor
chore and the verification workers to a queue implemented in the
database, rather than the sequence of in-memory queues we used to use.

This logical database is safely partitionable from the rest of
satelliteDB.

Refs: https://github.com/storj/storj/issues/5251

Change-Id: I6cd31ac5265423271fbafe6127a86172c5cb53dc
2022-11-22 14:04:00 +00:00
Cameron
705dfa0875 satellite/nodeevents: add method Name to Type
Add method Name to Type for a more human-readable format

Change-Id: I10050b9ef3691922d9dfccdbb6f046769a649321
2022-11-18 12:31:02 +00:00
Cameron
c86340e6b9 satellite/satellitedb/dbx: change node_event read one to read first
read one is the wrong method when trying to select one row when there
are multiple. It returns TooManyRows error. Read first is the correct
method.

Change-Id: Ic6c92795486892ac041befd118b6945314bffeaa
2022-11-17 19:31:53 +00:00
Cameron
8681a36164 satellite/overlay: add ability to get offline nodes in need of email
Add LastOfflineEmail to overlay.NodeDossier. This is the last time a
node got an offline email. Add two new overlay db methods,
GetOfflineNodesForEmail and UpdateLastOfflineEmail. Edit db method
UpdateCheckIn to nullify last_offline_email if node is up.

Change-Id: I1ee60e7d98dd1b68348a57f9a4fb77c6c9895d6d
2022-11-17 19:03:04 +00:00
Jeremy Wharton
bc4cc4974d satellite/satellitedb: exclude end date when calculating project totals
This change modifies the method responsible for returning project
usage summaries such that the end date of the given time period
is excluded to prevent overlap.

Change-Id: If06155efff5c6fce3865f5f6e4344873abe3e432
2022-11-14 11:24:05 +00:00
Cameron
6e77e094ce satellite/nodeevents: add NodeEvents methods GetNextBatch, UpdateEmailSent
Change-Id: I0df3dd8f97d99da434fa7ef8049f13ae9a521af9
2022-11-11 22:55:41 +00:00
Cameron
2a25974261 satellite/overlay: insert node software update events into node events
When a node checks in and its version is below the minimum, insert
BelowMinVersion event into node events

Change-Id: I0e437ac34496778369515cbc40c15676da8b27ae
2022-11-11 20:43:56 +00:00
Cameron
5f6b4900d2 satellite/satellitedb: add columns last_offline_email and last_software_update_email
Change-Id: I8b5023bbbba55de633ce5ac3faf3361352c202ef
2022-11-11 14:43:59 -05:00
Fadila Khadar
553c1558a5 satellite/satellitedb: add 'contained' column to nodes table
We want to be able to exclude contained nodes from nodes selection. For this, we add a 'contained' column to the nodes table to track the containment status.

Fixes https://github.com/storj/storj/issues/5231

Change-Id: Id78e645f172145adcb8664646e8ebf14e218b57b
2022-11-09 11:11:37 -06:00
paul cannon
1b5d953fd9 satellite/satellitedb: add queue for verifications
Since the auditor will be moving to a different process from the
metainfo loop, we need a way of communicating which segments have
been chosen for audit. This queue will be that communication, for now.

Contrast this with the queue for _re_verifications in commit 9c67f62f.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: I9a269c7ef21e6c5e9c6e5e1f3db298fe159a8a79
2022-11-09 15:38:56 +00:00
Moby von Briesen
c07f016f57 satellite/satellitedb: Slight modifications to nodeevents table
* Mark node events table as "safely partitionable", meaning that it
  is/will not be queried relationally along with other tables. This way,
  we can safely use this table in Postgres rather than CockroachDB,
  where most of our other satellite tables are running.
* Add a dbx-generated delete function to the node events table, to allow
  us to easily delete entries created before a provided time. This
  allows us to keep the table clean, since there is no need to persist
  entries after emails have been sent.

Change-Id: I25e8a5c4092fe49dcfa6c8bb73f2043646bb611f
2022-11-07 17:16:59 +00:00
Cameron
cb0c359b81 satellite/overlay: insert DQ node events for stray nodes
Change-Id: I99da11e506ab7f6bcebdb08a5815078a3297c932
2022-11-04 15:48:17 +00:00
Cameron
74ddfab810 satellite/overlay: insert DQ event into node events in overlay.DisqualifyNode
Also, return node email from overlaycache db DisqualifyNode to be used
in node events insertion

Change-Id: I41534cf01351c1690c3966a8055c5fe6fcf0d6a6
2022-11-04 15:18:31 +00:00
Cameron
f42e5049c9 satellite/satellitedb: add log line for node events insertions
Change-Id: I3271885cfd130128c240c9e9993e835fedb1ab7f
2022-11-03 16:09:23 -04:00
Qweder93
5e5d6ecf6c satellite/{accounting, buckets}: added feature that allows to use custom SQL query instead of object loop for buckets accounting
since amount of objects is growing and looping through all of them
starts taking lot of time, we are switching for SQL query to do it
in chunks of tallies per bucket. 2nd part of issue fix.

Closes https://github.com/storj/team-metainfo/issues/125

Change-Id: Ia26bcac0a7e2c6503df9ebbf4817a636841d3284
2022-11-02 09:01:33 +00:00
Cameron
11e229cc7f satellite/nodeevents: implement node events DB
Implement node events DB with Insert and GetLatestByEmailAndEvent. Get
was changed to GetLatestByEmailAndEvent so we can verify items are being
inserted into the table without needing the ID, which is not available
to us in the tests.

Change-Id: I4abe63631c44774cd7e795fbab0cbab4d801db4c
2022-11-01 16:03:14 +00:00
Cameron
f76abb3b3d satellite/satellitedb: add dbx read method to node_events
Change-Id: I9cfcb467051921a9b54370e426e618a76ac4197e
2022-10-31 16:31:31 +00:00
Cameron
80bfa55b8e satellite/satellitedb: edit node_events migration to have different PK
Change node_events schema to use an id column as primary key rather than
node id because there can be multiple events per node id.

Change-Id: I518d8ef9ea658764876483e282a4058d3c4910f4
2022-10-28 22:59:28 +00:00
Cameron
6bf54bfaef satellite/satellitedb: add new table node_events
Add new table for node events. We can use this to notify node
operators of certain node events. Further, we can squash events for
multiple nodes with the same email into a single notification.

Change-Id: Icea6dd939df8fe4a98806bd79c014e21d239c43e
2022-10-28 19:16:25 +00:00
paul cannon
9c67f62fe3 satellite/satellitedb: add table for reverify queue
This table will be used as a queue for pieces that need to be reverified
(a regular audit timed out on the owning node, so now that node is
contained and we need to validate the piece before un-containing it).

Refs: https://github.com/storj/storj/issues/5228

Change-Id: I5dcd26b6adced8674cbd81884c1543a61ea9d4c8
2022-10-27 15:28:47 +00:00
Cameron
8c8688ca6b satellite/overlay: return email as part of NodeReputation
Make email accessible for email sending after reputation updates

Change-Id: I760feee0f6ca58b76a2955a04c0c366c618656bb
2022-10-26 17:33:22 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
Wilfred Asomani
903ea38c86 web/satellite: increment login failed_login_count in sql
This change increments users' failed_login_count in the database layer to avoid potential data race.
It also updates the login_lockout_expiration as well in one operation.

see: https://github.com/storj/storj/issues/4986

Change-Id: I74624f1bee31667b269cb205d74d16e79daabcb6
2022-10-07 16:46:29 +00:00
Lizzy Thomson
ef04eb8bea satellite/satellitedb: add columns in projects table
add user_specified_usage_limit and user_specified_bandwidth_limit columns in projects table

github issue https://github.com/storj/storj/issues/5185

Change-Id: Ia8c9b190a2bdb684ac9d43af8f2159027d65aa88
2022-10-06 18:20:39 -06:00
Márton Elek
9ddd20b72e satellite/projectaccounting: fix project usage right after the genesis
After you create a brand new cluster (with storj-up, for example) the project usage fails during the first 5 minutes.

The problem is the usage of `AS OF SYSTEM TIME` which points to a time where the master database didn't exist.

In this specific case the database not found error can be ignored to avoid such messages. (if the database is really missing, we will have problems way more earlier, eg. at the login)

Change-Id: I51ee78994d91fc2a14b56646402faaaa8154c934
2022-10-04 14:41:30 +00:00
Márton Elek
7f6f7e1246
satellite: optional migration for integration tests
This patch addresses the following issues:

 1. Running full migration in cockroachdb is quite slow. We already have an approach for unit tests to start from the latest snapshot. This patch makes it possible to use it for integrations tests.
 2. Migration requires executing a separated command which makes it hard to run application in containerized test environments (like storj-up) or from IDE. This patch introduces a hidden flag to run migration.
 3. Test user creation is painful. We do it with calling GraphQL + admin API. Providing an option with testuser makes the integration tests significant more simple (especially as the projectID -> access grant can be predictable)

Change-Id: I61010728727b490ea6aac32620f2da0484966727
2022-10-04 11:35:03 +02:00
Cameron
41c2bca096 satellite/{console,metainfo,satellitedb}: add projectdb method GetSalt
Add new project db method, GetSalt, to get project salt. If salt
column is empty, return the sha-256 hash of the project ID. This
new method is used in metainfo endpoint ProjectInfo to return the
project salt to the client. This is backwards compatible because
the salt column is not populated yet. The updated endpoint will
do the same thing as the current endpoint.

Change-Id: I7eba376c865e10995a5a916302feca7cd7c7efa2
2022-09-26 13:12:21 +00:00
Michal Niewrzal
a22e6bdf67 satellite/gc/bloomfilter: use int64 to count pieces
Pieces count in DB are stored as int64 and we would like to align bloom
filter processing with this type.

Change-Id: Iaec767e609a40d802077ae057520541805a7c44f
2022-09-22 09:39:53 +00:00
Yaroslav Vorobiov
ad7c5b1483 satellite/payments/monetary: remove pkg and all its references
Remove pkg satellite/payments/monetary as it moved to storj.io/common.
Update all code pkg references from monetary to common/currency.

Change-Id: If2519f4c80cf315a9299e6521a6b9bbc6c399156
2022-09-07 12:58:00 +00:00
Jeremy Wharton
1613d37466 satellite/console: add endpoints for listing and revoking api keys
This change adds the following endpoints:
- projects/apikeys/{id}: returns a paged list of API keys for the
	project specified by the given ID
- apikeys/delete/{id}: deletes the API key specified by the given ID

Additionally, the API Go code generator has been given the ability to
process unsigned integer parameters.

Change-Id: I5ff24e012da24a3f06bea1ebb62bae6ff62f951a
2022-09-01 14:24:00 -05:00
dlamarmorgan
335e11dacd satellitedb: add token balance to API endpoint
Add the users current wallet balance to the endpoints for claiming and listing storjscan wallets. Also prevent a user with a claimed wallet address from claiming a new wallet.

Change-Id: I0dbf1303699f924d05c8c52359038dc5ef6c42a1
2022-08-31 22:55:16 +00:00
dlamarmorgan
47301e5718 billing: add USDollarsMicro to billing DB
Adds USDollarsMicro currency to the billing DB which support fraction of a cent with decimal places for better billing amounts accuracy.

Change-Id: Id07dfae104d94e27c7b22ab8f5781010e16c4c8e
2022-08-30 14:48:51 -07:00
dlamarmorgan
afe58323f9 satellitedb: remove use of batch payment insert
Removed batch insert of payments since they do not guarantee order. Order of payments sent to the payments DB is important, because the billing chore will request new payments based on the last received payment. If the last payment inserted is not the last payment received, duplicate payments will be inserted into the billing table.

Change-Id: Ic3335c89fa8031f7bc16f417ca23ed83301ef8f6
2022-08-30 14:45:55 -07:00
paul cannon
a57cbc4e0b satellite/satellitedb: drop _gob columns
This concludes the saga that began with commit c053bdbd, migrating away
from using gob-encoded big.Float values in the database to using
integers with an implied number of decimal places.

Nothing is using or accessing or expecting the _gob columns to exist
anymore. The transition code and the migration chore are gone. All that
remains is to drop the columns.

Change-Id: I9b15ee52f7781510a6dc91cf7c54f7f9022b1210
2022-08-26 19:45:22 +00:00
JT Olio
12ef68bdf4 satellitedb: restore-trash shouldn't bother with nodes we've never talked to
Change-Id: I3c631e92a9a2670c52c9fa23b2b1baf67d3c9a9c
2022-08-25 16:14:29 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00
Wilfred Asomani
a4192acabb console/satellite: track signup captcha scores
This change tracks signup captcha scores in the signup_captcha column in the users table.
It slightly modifies the captcha verify method to return both the score and success.

see: https://github.com/storj/storj/issues/5067

Change-Id: I7b3993e44958cfcf179806c7df19d6887fe3eda9
2022-08-23 10:13:49 +00:00
dlamarmorgan
cd89e1e557 satellite/satellitedb: remove direct import of pgxerrcode
Use the provided ConstraintViolation method of the pgutil package rather than importing jackc pgxerrcode directly.

Change-Id: I4e86713000b3f5f0aadd54beee8ee239f0c8df8e
2022-08-22 11:31:48 -07:00
Clement Sam
31242f3820 satellite: use table aliases for query in QueryStorageNodeUsage endpoint
Following the changes made to fix the storage usage graph
on the storagenode dashboard, we added a new
interval_end_time column to the accounting_rollups table.

We noticed a two-day delay in the graph, turns out the sub-query
was wrong due to the conflicting interval_end_time column
in both tables so we have to explicitly state which table
column we are referring to.

Also, set the default for interval_end_time in the accounting
rollups to the start_time if the interval_end_time is null
which will be removed once we backfill the column and alter
it to be a non-nullable column.

Updates https://github.com/storj/storj/issues/4178

Change-Id: Iff32b261d07b6ee219d2b6b6542377f0c54633a1
2022-08-22 11:39:45 +00:00
Egon Elbre
8ddd5557f5 satellite/satellitedb: reduce migration tests
Database migration tests are rather slow, reduce them to
only the last 10 migrations, which should be sufficient.

Change-Id: Ib9d964fe6ec86ddeeef26c66b6ea9207b7868855
2022-08-22 09:04:49 +00:00
Yaroslav Vorobiov
9c77c762cb satellite/payments/storjscan: change USD currency to USDollarsMicro
Changes all usd amounts to use monetary.USDollarsMicro to have better prescision.

Change-Id: I391e2680f7fa11a0493e5af77488acff6dbf4999
2022-08-19 16:05:17 +00:00
Wilfred Asomani
c59d0d415c satellitedb: add signup_captcha column to users table
The signup_captcha column will hold the captcha scores of new users.

see: https://github.com/storj/storj/issues/5067

Change-Id: Ia322af29a3b5b019b417843272506a3dbd1397e4
2022-08-17 14:31:52 -05:00
paul cannon
0dcc0a9ee0 satellite/reputation: reconfigure lambda and alpha
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.

For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at

    https://forum.storj.io/t/tuning-audit-scoring/14084

In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.

Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.

Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
2022-08-17 18:52:53 +00:00
dlamarmorgan
8eec61d315 satellitedb: add timestamp index to billing table
Adding an index to the timestamp field of the billing transaction table to improve query performance. This should prevent having to do a full table scan when we query for the last billing transaction of a particular source and/or type.

Change-Id: I581f09494cc8662a12efba4302022a07121ba309
2022-08-16 22:50:02 +00:00
dlamarmorgan
f596f72f44 satellite/payments/storjscan: add billing update to chore
Update chore to add confirmed transactions to the billing table.

Change-Id: I3c9a98c2ddc96f8a8905250376a1f5490d810277
2022-08-16 11:18:15 -07:00
dlamarmorgan
ac241501ca satellitedb: add wallet address index to storjscan_wallet table
Adding an index to the wallet address field of the storjscan wallet table to improve query performance. This should prevent having to do a full table scan when we query for one or more wallet addresses for a given user in our queries.

Change-Id: Ic1b5d06c2258489e5464d186fed5270172f8cba5
2022-08-15 18:05:06 +00:00
kimbotsao
c3e7b1409d satellitedb: add salt to projects table
Change-Id: I53691849efc169c495566648c12ae3d639614c3d
2022-08-11 12:16:53 -04:00
wilfredasomani
d8b010f7bd satellite/{web,satellitedb}: fix project daily usage error
The new dashboard currently gets stuck on loading and displays an error when
it fails to get usage data. Failure happens on satelliteDb due to a cockroach transaction error
caused by reading data before using AS OF SYSTEM TIME in the same transaction.
This change reverses the order of daily usage queries to avoid this error.
And hides the loaders on the dashboard if/when an error occurs.

see: https://github.com/storj/storj/issues/5012

Change-Id: I06b6ee434f72242f9b7d21dec7aaf39d1d622f1e
2022-08-10 19:18:28 +00:00
paul cannon
37a4edbaff all: reformat comments as required by gofmt 1.19
I don't know why the go people thought this was a good idea, because
this automatic reformatting is bound to do the wrong thing sometimes,
which is very annoying. But I don't see a way to turn it off, so best to
get this change out of the way.

Change-Id: Ib5dbbca6a6f6fc944d76c9b511b8c904f796e4f3
2022-08-10 18:24:55 +00:00
Yaroslav Vorobiov
eca1689f17 satellite/console: add wallet payments API
Extends satelling console billing API with method to
retrieve payments for a particular wallet.

Change-Id: I2e82793c715db9353427e0a287baac57bf6a0f43
2022-08-09 14:53:54 +02:00
dlamarmorgan
174d2ec181 satellite/{payments/billing,satellitedb}: Add retry logic to insert
During an update to the billing DB, there is a special case failure that can occur if multiple updates to the table happen concurrently. In this case, the update would normally fail silently due to the balance constraint during update, and the subsequent insert for a new record fails because the user already exists in the table. The solution for this case, is to simply retry the insert with some limit to prevent infinite loops.

Change-Id: Ibe70fec2c386c25bd2484fe91f49a6a962357706
2022-08-08 14:42:01 -07:00
dlamarmorgan
92be1d878f satellite/payments/stripecoinpayments: storjscan invoice generation
Add line item with unclaimed Storjscan wallet balance during invoice generation.

Change-Id: I018bfa01abfcf7bfdffba0c5a1350a69188f63d5
2022-08-03 13:24:26 -07:00
dlamarmorgan
c8b72c8f71 satellite/{payments/billing,satellitedb}: refactor billing DB
Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances

Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838

satellite/{payments/billing,satellitedb}: refactor billing DB

Update the billing table to use generated IDs, to include the source and status fields, and to add metadata jsonb field that can be used for fields specific to a particular type of billing transaction. Additionally a new table was added to keep track of user balances

Change-Id: Ieb3a63aafd8fe21fc3386bafd43d52081b7d2838
2022-08-03 17:52:32 +00:00
Cameron
2a38740efa satellite/satellitedb: don't select public_id in ListByOwnerID because it will be NULL for old projects
Change-Id: I062fa617b622a55d9482206d3a2938cf8a080f2d
2022-08-03 11:37:12 -04:00
Yaroslav Vorobiov
2ba06aca87 satellitedb: storjscandb return payment with UTC time
Return UTC cached payment timestamp on converting from dbx
payment.

Change-Id: I693802d5ac3b786f9973e8299a95215a2e975982
2022-07-27 20:05:37 +00:00
Clement Sam
d1d15cbf4f satellite: return interval_end_time in DailyStorageUsage endpoint
Updates https://github.com/storj/storj/issues/4178

Change-Id: If0c6f66c58a258300bbf563eef56caba4f1dc626
2022-07-27 18:24:27 +00:00
Clement Sam
f2f046af7a satellite/satellitedb: add interval_end_time column to accounting_rollups table
Adding an interval_end_time column to the accounting_rollups table
to keep the last interval_end_time for each daily storage tallies.

Updates https://github.com/storj/storj/issues/4178

Change-Id: If7a8210c5e9fe2fc9df84b137a8b6e3db2471c58
2022-07-27 17:53:27 +00:00
Yaroslav Vorobiov
7194973b56 satellite/payments/storjscan: add usd value to cached payments
Add USD value to storjscan cached payment struct to store and
retrieve payments from payments DB.

Change-Id: Ic0b2b4a198e4ea8230bc9a01865b429343f52767
2022-07-27 14:55:45 +00:00
Cameron
4815cfc09b satellite/{console,satellitedb}: add PublicID to Project, db method GetByPublicID
github issue: https://github.com/storj/storj/issues/4861

Change-Id: Ia83635c0de751a77cd5a49d641da19ed76132c46
2022-07-21 19:49:26 +00:00
Egon Elbre
82fede2132 satellite/satellitedb: batch delete storage node tallies
Currently we have a significant number of tallies that need to be
deleted together. Add a limit (by default 10k) to how many will
be deleted at the same time.

Change-Id: If530383f19b4d3bb83ed5fe956610a2e52f130a1
2022-07-20 15:42:35 +03:00
paul cannon
2f20bbf4d8 satellite/reputation: add a reputation write cache
This should lower the amount of database load coming from
reputation updates.

Change-Id: Iaacfb81480075261da77c5cc93e08b24f69f8949
2022-07-14 21:40:16 +00:00
Jennifer Johnson
61649d2850 cmd/node-cleanup: command to delete uncontacted nodes
Some nodes were added to the nodes table due to a bug in quic
based storagenode contact code. This is a tool to clean up these nodes.

Delete with batch-size 1k seems to take ~400ms on local CockroachDB.

Change-Id: Ic0c1180528c27952e19c431fc9cc327292a10a5f
2022-07-12 12:15:49 +00:00
Yaroslav Vorobiov
6d95e34e39 satellite/payments: add payments method for deposit wallets
Add payments method to payments to DepositWallets interface.
Exposes payments retrieval API for a particular wallet to
other systems such as console billing API.

Change-Id: Ifcb3a35514aab50be00f6360007954980b5d8b38
2022-07-12 11:25:30 +00:00
Cameron
023a1c6271 satellite/satellitedb: add public_id column and index to projects table
github issue: https://github.com/storj/storj/issues/4861

Change-Id: I3d1928ca51c825a4c1589db5ae55902e57f00c3b
2022-07-05 18:39:29 +00:00
Egon Elbre
65b5a0fe82 satellite/{overlay,satellitedb}: add AS OF to download selection
This should reduce the load caused by download selection queries.

Change-Id: Ic1a89d9c3eb5418f0792eb20ec2aece18dc63f2c
2022-06-28 05:18:01 +00:00
Fadila Khadar
ffcd54719a satellite/satellitedb: use tx instead of db.db in transactions
Change-Id: Icee8871af76233651f007e03173660905eafc0e3
2022-06-14 17:35:44 +00:00
Cameron
240b70b828 satellite/console: use new type UpdateUserRequest as arg to db users.Update
The users.Update method in the satellitedb package takes a console.User
as an argument. It reads some of the fields on this struct and assigns
the value to dbx.User_Update_Fields. However, you cannot optionally
update only some of the fields. They all will always be updated. This means
that if you only want to update FullName, you still need to read the
user info from the DB to avoid updating the rest of the fields to zero.
This is not good because concurrent updates can overwrite each other.

This change introduces a new struct type, UpdateUserRequest, which
contains pointers for all the fields that are updated by satellite db
users.Update. Now the update method will check if a field is nil before
assigning the value to be updated in the db, so you only need to set the
field you want updated. For nullable columns, the respective field is a
double pointer. This allows us to update a column to NULL if the outer
pointer is not nil, but the inner pointer is.

Change-Id: I27f842d283c2711e24d51dcab622e57eeb9157f1
2022-06-14 09:28:03 -04:00
Yaroslav Vorobiov
da97fafb0c satellite/payments/storjscan: add payments DB
Change-Id: Ia6f0f62912c9a60d2c513459959634179eb9506c
2022-06-10 13:44:27 +01:00
Cameron
5cfa7ca460 satellite/console: add cutoff to email reminders
There are multiple entries in the users table with the same email
address. This is because in the past users were able to register
multiple times if the email was not verified. This is no longer
the case. If a user tries to register with an unverified email
already in the DB, we send a verification email instead of
creating another entry. However, since these old entries in the
table with duplicate emails were never cleaned up, the email
reminder chore will send out email verification reminders to them.
A single person will get one separate email per entry in the DB
with their email and where status = 0.

Since the multiple entries with the same email problem was solved
a while ago, just add a constraint to GetUnverifiedNeedingReminder
to only select users created after a cutoff. Once the DB is
migrated to remove the duplicate emails, we can remove the cutoff.

github issue: https://github.com/storj/storj/issues/4853

Change-Id: I07d77d43109bcacc8909df61d4fb49165a99527c
2022-06-09 16:45:35 +00:00
paul cannon
737d7c7dfc satellite/reputation: new ApplyUpdates() method
The ApplyUpdates() method on the reputation.DB interface acts like the
similar Update() method, but can allow for applying the changes from
multiple audit events, instead of only one.

This will be necessary for the reputation write cache, which will batch
up changes to each node's reputation in order to flush them
periodically.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I44cc47767ea2d9423166bb8fed080c8a11182041
2022-06-07 15:22:25 +00:00
Vitalii
a61f0f6be3 satellite/projectaccounting: make daily usage query return correct allocated traffic
Updated daily project usage query to return correct allocated traffic.
If allocated egress has expired then we return settled egress.
If not then we return allocated egress - dead egress.

Fix for this issue
https://github.com/storj/storj/issues/4563

Change-Id: Ia15a50d3bb8d8cb1106936e17dbe0f1f5a40fa87
2022-06-01 15:24:52 +00:00
Vitalii
07e65cd338 satellite/projectaccounting: sum up bucket usages for daily usage query
Fixed daily usage query returning single bucket usage.
We sum up bucket usages now.
Also fixed https://github.com/storj/storj/issues/4559.

Change-Id: I2eb6299f1ef500d68150879195011b6fbb5f37ed
2022-06-01 12:50:10 +00:00
dlamarmorgan
007d4190c2 satellite/{payments/billing,satellitedb}: Add payment billing DB
Add billing DB to the satellite. This DB will hold all transactions on the users account and can be used to compute the users current usable account balance.

Change-Id: I056416efc169e5e5e30c9f30cd8bc766b7bc8073
2022-05-27 08:56:31 +00:00
paul cannon
5bc4c254d4 satellite/reputation: move Config fields into a Config struct
This has been a cause of some confusion, even though the fields are
labeled as being copies of config values.

Having them be under a field explicitly named "Config" makes this
clearer, plus, allows the values to be passed in simply as a copy
of the Config struct from the satellite, rather than copying the fields
individually (which can be error-prone, particularly as the AuditCount
field in UpdateRequest is apparently not the same thing as the
AuditCount field in reputation.Config).

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I386953347b71068596618616934aa28e3245cdc1
2022-05-24 20:57:51 +00:00
Márton Elek
5e8266d1de satellite/{payments/storjscan,satellitedb}: Add storjscan wallet DB
Add storjscan wallets DB to the satellite. For now this DB is a one to one mapping of the users account to a storjscan wallet that can be used by the account holder to make payments on their Storj account.

relates to https://github.com/storj/storj/issues/4347

Change-Id: I6e65b15817b90ceb75641244f9bf173c3b4228a7
2022-05-24 08:42:02 +00:00
paul cannon
2ee9463e34 satellite/reputation: don't need 3 identical AuditHistory types
The two protobuf types are identical except that one is in our common/pb
package, and the other is in internalpb. Since the type is public
already, and there is no difference in the internal one, it seems better
to use the public one for all satellite needs.

There is also another type which is essentially identical, but which is
not a protobuf type, also called "AuditHistory". It looks like we don't
ever actually need to have a separate type from the protobuf one.

This change makes us use "storj/common/pb".AuditHistory for all of our
AuditHistory needs.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: If845fde21bb31c801db6d67ffc9a146d1617b991
2022-05-24 05:48:46 +00:00
paul cannon
951a842fd4 satellite/{reputation,satellitedb}: move addAudit
This functionality will be needed in both packages, so here we move it
into the more general reputation-code package and export it for use in
satellitedb.

This also removes the related UpdateAuditHistory() signature from the
reputation DB interface, since it doesn't have anything to do with the
db. It doesn't need to be a method, either.

Finally, this changes the test for addAudit to be a plain test function
instead of using testplanet.Run(). It didn't need a whole testplanet
setup or any databases.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I90f6a909e5404f03ad776b95cfa2f248308c57c1
2022-05-20 15:37:31 +00:00
Egon Elbre
5d6d5331f5 private/testplanet: don't cleanup cockroach
Márton found out that DROP DATABASE is rather slow on CRDB, and it makes
a significant impact when running the whole testsuite. In sum of test
times it's ~2.5h compared to ~2h. And the end-to-end ~20m to ~16m.

This adds a new flag STORJ_TEST_COCKROACH_NODROP for enabling this
behavior in the CI environment.

Fixes https://github.com/storj/dev-enablement/issues/6

Change-Id: I5a6616c32dc6596a96ba3d203f409368307d7438
2022-05-19 15:27:37 +03:00
paul cannon
ac7752ac3c satellite/reputation: return full Info from reputation update
This will let us update our reputation cache when writing through to the
db.

Since the information is already being fetched from the db and returned
to the application, the extra cpu load here should be minimal.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I2b8619f2c0d541893c7d3e7d33b1863b96775ebd
2022-05-18 17:22:06 +00:00
Malcolm Bouzi
087e57d037 web/satellite: create chore that will resend verification emails to unverified users
We want to remind unverified users to verify their emails:
once after 24 hours has passed and again after 5 days has passed.

Add mailservice.Service to satellite core because it is needed by the
chore for sending emails. To add the mailservice.Service to the core,
we create a helper function in satellite/peer.go to avoid duplicating
the code in both api.go and core.go. In addition to the chore, this
change adds methods to users.DB to get unverified users in need of
reminder.

Change-Id: I4e515bdf43f922788b4f965b2efb34fa32288bd1
2022-05-18 08:08:33 +00:00
paul cannon
aa728bd6ea satellite/satellitedb: add reputations.disqualification_reason
We added nodes.disqualification_reason recently, but we didn't add a
corresponding column in the reputations table (despite having a
corresponding `disqualified` column there).

Without this change, the (very useful and informative) assignments to
updateFields.DisqualificationReason in reputations.go have no effect.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I77404902ca64b56aed72f1de76b303fe82b76aab
2022-05-17 10:09:36 -05:00
Cameron
bfad64a6f0 satellite/satellitedb: add verification_reminders column to users table
When a new user registers, we send a verification request to their
email. Currently, if they do not verify their email, we take no further
action. We want to send these users reminders: one after about one day
and one after about 5 days. To do this we will use this new
verification_reminders column.

It will look something like this:

```
SELECT email FROM users
WHERE status = 0
   AND (
          (verification_reminders = 0 AND created_at < now() - 'INTERVAL 1d')
          OR (verification_reminders = 1 AND created_at < now() - 'INTERVAL 5d')
   )
```

Change-Id: If0620e08c97e9e337c9563481d665c5bd462693b
2022-05-16 16:37:16 +00:00
Cameron
772397172f satellite/console: implement WebappSessions db
Create WebappSessions interface in consoleauth package.
Interface implements the DB for webapp_sessions table.

https://github.com/storj/storj/blob/main/docs/blueprints/webapp-session-management.md

Change-Id: Ib56f238c20b58f2877046fc384232add253ee82b
2022-05-16 15:06:04 +00:00
Cameron
18eedaa62f satellite/satellitedb/dbx: add method to delete webapp sessions by user ID
Change-Id: I4552622fdf40c6ecd53046e597250bbe713f510c
2022-05-16 14:37:34 +00:00
Vitalii
95ae739d9c satellite/projectaccounting: fetch bucket usage since bucket creation instead of project creation
Fix for this customer issue
https://github.com/storj/customer-issues/issues/34

By this change we fetch bucket usage since its creation instead of using project's createdAt timestamp.

Change-Id: Ic0ea5d169056a5bd64ed143d13954d794da6e1d2
2022-05-13 13:27:42 +00:00
Egon Elbre
4791ba5d50 satellite/{mailservice,oidc}: improvements to debugging
Things that make debugging easier.
* Added logging to automatic link clicking to make it obvious, when it
  fails.
* Added monitoring to oidc.
* Made dbx create calls noreturn for oauth_*

Change-Id: I37397b4e84ce5bfd82954aed9c38fdfd52595f24
2022-05-11 19:59:42 +00:00
Erik van Velzen
26f495f717 satellite/repair: implementation of batch insert
Part of https://github.com/storj/storj/issues/4727

Change-Id: I44990a7614af26f8ee0be9c7aed496a1dd9e5df7
2022-05-09 12:41:22 +00:00
Erik van Velzen
10d71a8a3c satellite/satellitedb: outline for batch insert
Part of https://github.com/storj/storj/issues/4727

Change-Id: I1a9ad3b009f363e37f5e68e810074eecb7448db3
2022-05-09 11:39:52 +00:00
Moby von Briesen
24bbb7f236 satellite/satellitedb: Add webapp sessions table
Enables us to better manage sessions for satellite UI users.

Change-Id: I6695f8f1174d2e5fb970efddb0a8b14725e2b57e
2022-05-04 20:37:56 +00:00
Cameron
dd6ab25cbd {cmd/satellite/reports, satellite/attribution}: type and variable name adjustments
Change-Id: I553b7ed9cb702479911baaf54d77d8f2662faaa2
2022-04-26 20:12:38 +00:00
Michał Niewrzał
6e5a94698e satellite/metabase: add option to override metadata with CommitObject
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.

https://github.com/storj/team-metainfo/issues/105

Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
2022-04-26 08:03:52 +00:00
Qweder93
7e6618cfad satellite/accounting: test tally on copied object flaky fix
Resolves: https://github.com/storj/storj/issues/4732

Change-Id: I2abf50aafc62dd97be12b0c0573db2f95d560796
2022-04-21 10:30:17 +00:00
Yaroslav Vorobiov
553ef9d640 satellite/satellitedb: suspended column removal migration
Last part of backwards compatible db migration to remove "suspended" column.
Removes exeption which removes "suspended" column in tests from `migrate_test.go`.
Adds DB migration to remove "suspended" column from 'nodes' and 'reputations' tables.

Change-Id: I02051279f6f4181e966c567919af0e774583f165
2022-04-20 17:59:11 +00:00
Yaroslav Vorobiov
3f47d19aa6 satellite/overlay: add disqualification reason
Add disqualification reason to NodeDossier.
Extend DB.DisqualifyNode with disqualification reason.
Extend reputation Service.TestDisqualifyNode with disqualification reason.

Change-Id: I8611b6340c7f42ac1bb8bd0fd7f0648ad650ab2d
2022-04-20 13:29:31 +00:00
Yaroslav Vorobiov
4223fa01f8 satellite/reputation: add disqualification reason for status update
Set disqualification reason when reputations stats are updated on DB.Update.
Added tests for DisqualifyNode and for disqualification cases which happens during Update.

Change-Id: I00130ab5d9722422805159ad2f183c205de60f7e
2022-04-20 13:29:10 +00:00
Cameron
48fb3e947c cmd/satellite/reports: sum attribution data by user agent
Attribution is attached to bucket usage, but that's more granular than
necessary for the attribution report. This change iterates over the
bucket attributions, parses the user agent, converts the first entry
to lower case, and uses that as the key to a map which holds the
attribution totals for each unique user agent.

Change-Id: Ib2962ba0f57daa8a7298f11fcb1ac44a8bb97875
2022-04-18 13:23:37 +00:00
Vitalii
513ad99756 satellite/satellitedb: migrate wrongly affected users to have default limits
Migrate free tier users to have default limits if their limits were set to 0.
They were affected by incorrect working of Update user query.

Change-Id: I4c49c8d99b12dba2b9b0ab61b2175085976dcc95
2022-04-13 13:33:03 +00:00
Vitalii Shpital
b4cec4fd0b satellite/satellitedb: add columns to users table to control failed login attempts
Added failed_login_count and login_lockout_expiration columns to users table to control users failed login attempts.
We want to prevent brute forcing of user login so this is the first step.

Change-Id: I06b0b9f5415a1922e08cd9908893b2fd3c26bca0
2022-04-12 08:37:07 +00:00
Cameron
d4ad3a3ca6 satellite/satellitedb/attribution: update value attribution query to return byte-hours
Before, the VA query was summing the total and dividing by the number of
rows. This gives the average bytes stored per hour, but we charge for
usage with byte-hours. Why not do value attribution the same way?
To do that, we don't divide by the number of rows. We also have object
and segment fees so return segment-hours and object-hours too.

Change-Id: I1f18b7e1b2bae1d3fae1ca3b93bfc24db5b9b0e6
2022-04-08 16:22:21 -04:00
paul cannon
d8733ddd40 satellite/satellitedb: stop using _gob columns
This sets the corresponding _numeric columns to be NOT NULL (it has been
verified manually that there are no more NULL _numeric values on any
known satellites, and it should be impossible with current code to get
new NULL values in the _numeric columns.

We can't drop the _gob columns immediately, as there will still be code
running that expects them, but once this version is deployed we can
finally drop them and be totally done with this crazy 5-step migration.

Change-Id: I518302528d972090d56b3eedc815656610ac8e73
2022-03-30 04:13:13 +00:00
Cameron
84b522bc06 satellite/console: create account management api keys service
We are in the process of creating an api to allow users to manage their
accounts programmatically. We would like to use api keys for
authorization. We were originally going to create an entirely new table
for these api keys, but seeing as we already have 2 other tables for
keys/tokens, api_keys and oauth_tokens, we thought it might be better to
use one of these. We're using oauth_tokens.

We create a new oidc.OAuthTokenKind for account management api keys:
KindAccountManagementTokenV0. We made the key versioned because we
likely want to improve the implementation in the future, but we want to
get something functional out the door ASAP because the account management
api feature is highly desired.

Add a new method to oidc.OAuthTokens interface for revoking v0 account
management api keys, RevokeAccountManagementTokenV0. Add update method
to dbx implementation to allow updating the expiration. We will revoke
these keys by setting the expiration to 0 so they are expired.

Change-Id: Ideb8ae04b23aa55d5825b064b5e43e32eadc1fba
2022-03-23 17:02:20 +00:00
Yaroslav Vorobiov
ca09a06680 satellitedb: revert migration for suspension column removal change
Change-Id: Iec8bbed8dd82d32c68ee7f9e64eec71ab8469373
2022-03-23 13:42:11 +00:00
Michał Niewrzał
ba74cb17a9 satellite: adjust to latest CRDB version
We have an issue with latest CRDB. Single query cannot modify
the same table multiple times. Now build is blocked.
This change is unblocking build by:
* adjusting query for inserting into repair queue
* temporary removing code for deletion for server-side copy
* temporary disable backward compatibility tests for CRDB

Change-Id: Idd9744ebd228e5dc05bdaf65cfc8f779472a975d
2022-03-23 07:16:38 +00:00
Yaroslav Vorobiov
ddbbb0038b satellite/satellitedb: remove suspension column from nodes and reputations
Remove redundant suspension timestamp column from nodes and reputation tables.
Suspended timestamp was moved to unknown_audit_suspended and suspended column is
no longer used so there is no point in keeping both.

Change-Id: Ieea3f12141b33ec9efe7594f4c9dbc7e10675b0e
2022-03-21 16:56:12 +00:00
Egon Elbre
0d2d59f884 all: fix linting issues
Change-Id: Idfc93948e59a181321d79b365e638d63e256a16f
2022-03-21 15:26:42 +00:00
Mya
75be1c0a28 satellite/console: support reading apikeys by name and project id
When performing re-authorizations for OAuth, we need to pull up an
APIKey using it's project id and name. This change also updates the
APIKeyInfo struct to return the head value associated with an API
key.

Change-Id: I4b40f7f13fb9b58a1927dd283b42a39015ea550e
2022-03-14 13:21:21 -05:00
dlamarmorgan
715077d04a satellite/{console,satellitedb}: increase project limit on paid tier upgrade
Update the user to the default paid tier project limit, which is currently 3 projects, when the user upgrades to a paid account.

Change-Id: I95b19d62cebc7d878b716355f2ebcaf0b51ca3f7
2022-03-14 16:10:23 +00:00
Fadila Khadar
29fd36a20e satellite/repairer: handle excluded countries
For nodes in excluded areas, we don't necessarily want to remove them
from the pointer, but we do want to increase the number of pieces in the
segment in case those excluded area nodes go down. To do that, we
increase the number of pieces repaired by the number of pieces in
excluded areas.

Change-Id: I0424f1bcd7e93f33eb3eeeec79dbada3b3ea1f3a
2022-03-14 10:59:36 -04:00
Michał Niewrzał
39ac90835f satellite/satellitedb: change default segment limit to 100M for paying users
We decided that we want to have segment limit for paying users high
enough to not have to change it too often.

Fixes https://github.com/storj/storj/issues/4590

Change-Id: Ic1c38bf3e2fcc000548ff4c7e7004647b39fbecf
2022-03-04 13:18:14 +00:00