Commit Graph

675 Commits

Author SHA1 Message Date
Egon Elbre
63c7f8b7fc satellitedb/satellitedbtest: creating a database shouldn't auto-migrate
Some tests need to control the migration progress manually.

Change-Id: I776c69b6d56dc35c7cb88688c4b827d6bba4b7ac
2021-02-11 14:21:49 +02:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Yaroslav Vorobiov
966535e9de {storagenode,satellite}/nodeoperator: add wallet features
Change-Id: Iac7eb40a52b8fddcc573aebaad2e3a30a10cded9
2021-02-08 22:09:45 +02:00
Yingrong Zhao
3b49d3cddf satellite: remove referral program related code
This PR removes all back-end related referral program code including the
marketing portal.

We will have a separate PR for front-end code and database migration to
drop `offers` and `usercredits` table

Change-Id: If59f952cddfe0558a7dc03a0eac7cc1081517f88
2021-02-08 13:52:50 +00:00
Jeff Wendling
e114cfe86d satellite/satellitedb: fix broken query
Change-Id: I6d412a673d75264bf9751c6f15b1fb0ab94e1394
2021-02-04 16:00:51 -05:00
Malcolm Bouzi
db3a3088f9
satellite/satellitedb: add professional user fields to db interface (#4034) 2021-02-04 10:00:15 -05:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
JT Olio
c8e42df2f8 satellitedb: reorder migrations 140-144
These changes are independently tracked on
https://github.com/storj/storj/tree/jt/migration-reorder

The point of this is to make the distributed column
migration, needed for SNO invoice generation, the very
next one, so we can release it as a point release.

Change-Id: I26e1c03629c4f079b9ad12485e2b71a715d82b3b
2021-02-03 18:28:42 +00:00
Ethan
9506b67ca2 satellite/projectaccounting: Improve performance of ProjectAccounting.getBuckets
Limit bucket name lookup to date range of the calling methods since we only need distinct bucket names for that time period.
Adds new index and removes an index specific to project ID since it is no longer needed.

Change-Id: Ic07bbfb1c32280e0c0e39f8da020b284e1e5d974
2021-02-03 14:05:12 +00:00
Cameron Ayer
a17934cb51 satellite/satellitedb: remove reference to uptime counts
Change-Id: I26ac540b720a8ba5d6ca44526900228352dcaf4e
2021-02-02 14:51:27 -05:00
Jeff Wendling
759bdd6794 satellite/compensation: add total-paid and total-distributed to invoices
Change-Id: Id4414867917cbf8aad77795f764d6381e88d9a34
2021-02-02 18:14:31 +00:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Ivan Fraixedes
cc0d88f9c3
satellite/satellitedb: Fix GE flaky test
Fix an issue due to copy-paste problem that made that the Graceful Exit
test to be flaky.

The test uses a time created at the beginning of the test for avoiding
to get undeterministic time differences due to the fact of the response
time variation by the DB queries, however some part of the test were
using a current time rather than this base time, so they have been
addressed.

Change-Id: I4786f06209e041269875c07798a44c2850478438
2021-02-02 13:24:42 +01:00
Ivan Fraixedes
d93944c57b satellite/orders: Delete unused methods & DB tables
Delete satellite order methods and DB tables which aren't used anymore
after we have done a refactoring on the orders to stuck bucket
information in the orders' encrypted metadata.

There are also configuration parameters and a satellite chore that
aren't needed anymore after the orders refactoring.

Change-Id: Ida3682b95921df70792284b42c96d2508bf8ca9c
2021-02-01 18:01:29 +00:00
Ivan Fraixedes
076804eac9 cmd/satellite: Add command for GE data cleanup
Add a command to the satellite for cleaning up the Graceful Exit (a.k.a
GE) transfer queue items of nodes that have exited.

The commit adds to the GE satellite DB a couple of new methods, and its
corresponding test, for performing the operations of the new command.

Change-Id: I29a572a59689d63b24990ac13c52e76d65aaa917
2021-02-01 17:30:58 +00:00
Jeff Wendling
1cf3d89a56 satellite/satellitedb: add distributed column and migration
using redash i manually checked that the only times the sum of
the payments does not match the paid column is for 2020-12 and
if it does not match then there are no payments.

Change-Id: I71ce0571de7e38e21548d7d6757b25abc3bfa781
2021-02-01 16:33:14 +00:00
Natalie Villasana
91bd4191dd satellite/accounting: add rollup archiver chore
The rollup archiver chore moves bucket bandwidth rollups and
storagenode rollups that are older than a given duration
to two new archive tables.

Change-Id: I1626a3742ad4271bc744fbcefa6355a29d49c6a5
2021-02-01 09:29:54 -05:00
Kaloyan Raev
d0612199f0 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/config.go
	satellite/metainfo/metainfo_test.go

Change-Id: I95cf3c1d020a7918795b5eec63f36112fdb86749
2021-02-01 14:32:12 +02:00
Egon Elbre
54e01d37f9 satellite/overlay: add DownloadSelectionCache
Change-Id: Ic0779280172325f8d03f55a2e9673722f72bdd44
2021-01-29 16:47:06 +02:00
Kaloyan Raev
4d32bdaefb satellite/satellitedb: drop bucket_metainfos_name_project_id_key index
This index is obsolete and duplicates a similiar (project_id, name)
index on the same table.

Moreover, it might confuse CockroachDB which of the two index to use,
which may might affect DB performance.

Change-Id: If8d1df8347714942cea9dca82864ba5f4973bed3
2021-01-28 09:06:22 +02:00
Jeff Wendling
ca86820b8b satellite/snopayouts: use dbx + some refactorings
Change-Id: I8f3973d2377f071bcea2f61e0fc21d913ffa7ea8
2021-01-27 17:59:16 +00:00
Jeff Wendling
66e15fb7f1 satellite/compensation: remove ytd paid amounts
they aren't right and we aren't using them.

Change-Id: I5ca024e38d055696696886278863e941b5bc51bf
2021-01-27 17:31:01 +00:00
Malcolm Bouzi
24d60384c5
satellite/satellitedb: add columns for professional users (#4028)
Co-authored-by: Egon Elbre <egonelbre@gmail.com>
2021-01-26 11:38:53 -05:00
Isaac Hess
c92bda7e75 tally monkit: change location to monitor piecesize
When we observed the value for total piecesizes stored in the network,
we were doing it after converting them to byte-hours, rather than using
the actual piece sizes. This fixes that issue.

Change-Id: I1564d21b519f70eb59f298d97dbd777baf127723
2021-01-26 15:37:02 +00:00
Moby von Briesen
0a48071854 satellite/console: Add pagination fields for ListProjectsByOwnerID
Add ProjectsCursor type for pagination
Add PageCount, CurrentPage, and TotalCount ProjectsPage
This allows us to mimic the logic of GetBucketTotals and the
implementation of BucketUsages in graphql for the new ProjectsByOwnerID
functionality.

Change-Id: I4e1613859085db65971b44fcacd9813d9ddad8eb
2021-01-20 16:15:29 +00:00
Kaloyan Raev
c24ada7114 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum

Change-Id: Icf7c029e9d800e5f6a9fdd208c36f28e05468690
2021-01-20 17:35:57 +02:00
Cameron Ayer
d14607a5f7 satellite/{contact,nodestats,overlay,satellitedb}: remove references to total_uptime_count and uptime_success_count columns
Change-Id: I1f92022909bc564e9b1e31bf937fdfe7c16554de
2021-01-19 15:43:02 -05:00
Cameron Ayer
75d828200c private,satellite: add chore to dq stray nodes
Full scope:
private/testplanet,satellite/{overlay,satellitedb}

Description:
In most cases, downtime tracking with audits will eventually lead
to DQ for nodes who are unresponsive. However, if a stray node has no
pieces, it will not be audited and will thus never be disqualified.
This chore will check for nodes who have not successfully been contacted
in some set time and DQ them.

There are some new flags for toggling DQ of stray nodes and the timeframes
for running the chore and how long nodes can go without contact.

Change-Id: Ic9d41fdbf214736798925e728245180fb3c55615
2021-01-19 14:21:56 -05:00
Qweder93
6ba8f6c8a9 storanode, satellite: payout renamed to payouts, expected estimation payouts added, console api for audits reworked
Change-Id: I4aa5e99bffaa87d0a800a429a4c83aa498ad4b7b
2021-01-18 10:56:03 +00:00
Ivan Fraixedes
678b07b314
satellite: Fix typos & code formatting
Fix some typos in the doc comments and readdress some code formatting
applied automatically.

Change-Id: I605b4eff2e7c6c58227ecf16be4c1d26f5322eb6
2021-01-15 16:40:26 +01:00
Moby von Briesen
c24f84914c satellite/console: Add ability to list projects by owner ID
Listing projects by owner ID also includes the number of members in each
project.

Change-Id: I53a09674b60c199ef378943851bb0f164e92e4e2
2021-01-15 14:22:22 +00:00
Kaloyan Raev
6dff40f5c5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/metainfo.go

Change-Id: Ib5c49f3c911c58319855a171f9ce73657da976d9
2021-01-14 14:33:59 +02:00
Cameron Ayer
0184d33e96 satellite/satellitedb: set default 0 on uptime columns
This is the first step in the removal of uptime columns on the
nodes table. These columns are no longer used:

uptime_success_count
total_uptime_count
uptime_reputation_alpha
uptime_reputation_beta

In order to avoid breaking backwards compatibility, we need to
remove all references to these columns before removing the columns
themselves from the database. However, since uptime_success_count
and total_uptime_count are NOT NULLABLE, we can't remove them from
the insert statements in the overlay. So we can't remove the columns
because of the references, and we can't remove the references because
the columns can't be null. What a pickle. To remedy this, we will set a
default on the columns. Then we should be able to remove them from the
insert statements

Change-Id: I75f6c56fb7897835bbf29869f86f39de1d9dd345
2021-01-12 17:44:37 +00:00
Cameron Ayer
0403e99a5b satellite/{overlay,satellitedb}: remove unused methods for old downtime tracking
GetSuccessfulNodeNotCheckedInSince and GetOfflineNodesLimited are overlay methods
which were only used by the previous downtime tracking system which has been removed.
These methods should also be removed.

Change-Id: Idb829d742e1f987e095604423fff656fe581183e
2021-01-11 15:21:28 +00:00
Michał Niewrzał
ec88d21a3c Merge 'main' branch.
Change-Id: I6e8162d1a6caf75e89c9f9c9f9522730aebf83ae
2021-01-11 10:26:58 +01:00
Moby von Briesen
6e2ef3b9ee Revert "satellite/satellitedb: Do not consider nodes with offline_suspended as reputable."
This reverts commit e24262c2c9.

Change-Id: I287deb2e52d03bbd698ed055f0f216b0b5bf2798
2021-01-04 14:28:37 +00:00
Michał Niewrzał
ad3e3a38c5 Merge 'main' branch
Change-Id: Ia0db1b1f9ef3e0671d3f2208881b0abc3064e200
2021-01-04 12:13:45 +01:00
Moby von Briesen
825dc71227 satellite/{overlay, satellitedb}: Refactor audit history
* Separate audit history interface into its own file in the overlay
package
* Add overlay.AuditHistory struct so that internalpb.AuditHistory is
only used from within the database layer
* Add overlay.GetAuditHistory function for features that will require
access to detailed audit history information
* Do not return full audit history from UpdateAuditHistory - callers to
that function only need to know the online score and whether a full
tracking period has been completed
* Move audit history tests out of satellite/satellitedb, since they are
independent of database implementation

Change-Id: I35b0c4ac23bbaabd80624f8a9631c3cb1a1f33bd
2020-12-29 18:50:22 +00:00
Moby von Briesen
85ae13f11d satellite/satellitedb: Drop nodes_offline_times table.
Now that the deprecated downtime tracking service is removed
(3fc76f4ffe), we can safely remove
the nodes_offline_times table.

Change-Id: Ia7c6efe32ba104dff5a830af5f2beee3337eefe5
2020-12-29 18:17:50 +00:00
Moby von Briesen
e24262c2c9 satellite/satellitedb: Do not consider nodes with offline_suspended as reputable.
Nodes which are offline_suspended will no longer be considered for new
uploads. The current threshold that enters a node into offline
suspension is 0.6. Disqualification for offline suspension is still
disabled.

Change-Id: I0da9abf47167dd5bf6bb21e0bc2186e003e38d1a
2020-12-29 17:59:09 +00:00
Ethan Adams
6070018021
satellite/overlay: use AS OF SYSTEM TIME with Cockroach
Query nodes table using AS OF SYSTEM TIME '-10s' (by default) when on CRDB to alleviate contention on the nodes table and minimize CRDB retries. Queries for standard uploads are already cached, and node lookups for graceful exit uploads has retry logic so it isn't necessary for the nodes returned to be current.
2020-12-22 21:07:07 +02:00
Michal Niewrzal
9a8959d429 Merge 'master' branch
Change-Id: Iba69ea73ca4d3f1cd4ae94243eaaae033c5324e8
2020-12-22 14:55:57 +01:00
Ethan Adams
563197c628
satellite/overlay: Add index on nodes table (#4012)
satellite/accounting: Add index for project_id on bucket_storage_tallies
2020-12-21 12:48:48 -05:00
Ethan Adams
9b52283570
satellite/accounting: Add index for project_id on bucket_storage_tallies (#4010)
Change-Id: I47ab2d1e24f94307c3383c497cffe2a150fa8ab7
2020-12-21 11:42:00 -05:00
Ethan Adams
6e501898c3
satellite/accounting: Performance improvements to getNodeIds used by GetBandwidthSince (#4009) 2020-12-21 16:37:01 +01:00
Jessica Grebenschikov
da0327c9b7 satellite/dbcleanup: remove expired serial chore
Change-Id: Ib71d41eb6679d6435e5bc10b6244dac66380a74e
2020-12-18 09:36:28 -08:00
Michal Niewrzal
2111740236 Merge 'master' branch
Change-Id: Ib73af0ff3ce0e9a1547b0b9fc55bf88704f6f394
2020-12-18 09:13:24 +01:00
Cameron Ayer
28eaae66af satellite/satellitedb: drop num_healthy_pieces column from injuredsegments
This column is no longer used as it has been replaced by the segment_health
column.

Change-Id: I6b4df89cd4f994d8418976f88e8c5f57615f8115
2020-12-17 20:17:08 +00:00
Michal Niewrzal
70ba4deea9 satellite/repair/checker: adjust irreparable part of repair checker
Change-Id: I0732104a97ba18a5359de3966cd692677a0ff790
2020-12-17 14:11:22 +00:00
Kaloyan Raev
9aa61245d0 satellite/audits: migrate to metabase
Change-Id: I480c941820c5b0bd3af0539d92b548189211acb2
2020-12-17 14:38:48 +02:00