Commit Graph

1289 Commits

Author SHA1 Message Date
Jessica Grebenschikov
41497569cd satellite/orders: add settled amount to rollups write cache
https://storjlabs.atlassian.net/browse/SM-1109

Change-Id: Ic5859b141c1384157b33df0d7fb6c8b43cc8a6b1
2020-07-07 14:46:06 +00:00
Kaloyan Raev
12a15e5a6a satellite/payments: remove migare-credits billing command
This was a one-time command and it has been already executed on all
production satellites.

Change-Id: Ic58a151c3e5da9c139df875f6517f37275a1ebf5
2020-07-07 11:47:53 +03:00
Isaac Hess
fd740295ec satellite/satellitedb: Add comment to revocation
Change-Id: I1b65b7e46439c4788835ea5bfd4df3d32a713b44
2020-07-06 21:51:35 +00:00
Caleb Case
db55386fbc satellite/rewards: Add Rclone to partners database.
Change-Id: Ieab098f7fb411230b72a88533ae632bac8967266
2020-07-06 16:13:15 +00:00
Bill Thorp
00ae5ebbab satellite/payemnts: Credit coin payments earlier
Apply the coin payments when CoinPayments.net recieves the funds
Instead of the when STORJ gets them from CoinPayments.net

Based on 7/1/20 User Growth standup guidance by JG
Relates to: https://storjlabs.atlassian.net/browse/USR-801

Change-Id: I174ca23a585010f39464c45525e1dfe0179b7c1a
2020-07-06 13:24:26 +00:00
littleskunk
e1a18ce974
satellite/accounting: bugfix infinit storage usage increase (#3916)
* satellite/accounting: bugfix infinit storage usage increase

* support int64 values

Co-authored-by: JT Olio <hello@jtolio.com>
2020-07-03 12:34:13 -06:00
Cameron Ayer
e3088d9ad5 satellite/satellitedb: add new DB table audit_histories
Change-Id: I5f854514994cab9a68cf978f2dabfb588df695f5
2020-07-01 21:14:35 +00:00
Qweder93
b639ec08d4 satellite/heldamount: payments added, endpoind for payments added
Change-Id: Ia2b9580bc353ef614680230c6f82c5bf6ded49c4
2020-07-01 18:15:01 +03:00
Bill Thorp
66661c7486 satellite/coinpayments: query status of >25 coinpayments
Allow more than 25 coin payment statuses to be queries from coinpayments.net
Add some slight logging, a coinpayments duration metric, and a disabled test

These changes are small changes in support of https://storjlabs.atlassian.net/browse/USR-801

Change-Id: I5b176cdd5e513d8bd88b92f9b22a8bd2456bbdd5
2020-07-01 09:04:16 +00:00
Cameron Ayer
cadb435d25 {satellite/audit, private/testplanet}: remove ErrAlreadyExists, run 2 audit workers in testplanet
Since we increased the number of concurrent audit workers to two, there are going
to be instances of a single node being audited simultaneously for different segments.
If the node times out for both, we will try to write them both to the pending audits
table, and the second will return an error since the path is not the same as what
already exists. Since with concurrent workers this is expected, we will log the
occurrence rather than return an error.

Since the release default audit concurrency is 2, update testplanet default to run with
concurrent workers as well.

Change-Id: I4e657693fa3e825713a219af3835ae287bb062cb
2020-06-30 18:00:07 +00:00
Egon Elbre
735dc6e163 satellite/orders: add encryption keys
This adds EncryptionKey definition that can be used as a flag.
These order.EncryptionKey-s will be used to encrypt data in
order limits.

This helps to avoid storing lots of transient data in the
main database.

This code doesn't yet contain encryption itself.

Change-Id: I2efae102a89b851d33342a0106f8d8b3f35119bb
2020-06-30 15:03:14 +00:00
Egon Elbre
3567b49ef4 satellite/metainfo/piecedeletion: fix int to string conversion
Change-Id: I8d0cdec7cee182ade479c0cfb1d38f7f97af2ab7
2020-06-29 17:22:52 +03:00
paul cannon
4997fd55d0 satellite/repair: remove healthy from irreparabledb
Change-Id: Ia9d300d0359883f03734d0bdf204d56d6642ce34
2020-06-26 21:26:00 +00:00
Rafael Gomes
bdaabd611d satellite/metainfo/piecedeletion: add metrics for deletion requests
Change-Id: I48ad96e78dab84b9238c63d62bda679fc65c2072
2020-06-26 11:12:33 -03:00
Egon Elbre
d91cf5f4de satellite/satellitedb: add missing SeparateTx
Change-Id: I3ba5a4e0632a1e0e5e77c30e515953eadf05bc45
2020-06-26 12:27:05 +03:00
Egon Elbre
13a5854535 satellite/satellitedb: clarify test migration merging
Use a field to distinguish migration steps that need to use a
different transaction from previous steps. This is clearer than
using a func.

Change-Id: I2147369d05413f3e8ddb50c71a46ab1ba3ab5114
2020-06-25 14:32:45 +00:00
Egon Elbre
df8cf8f58a satellite/orders: delete unused code
Change-Id: I431c8cc2f23e538c676d6f742fb1faef7cc1d73e
2020-06-25 16:48:26 +03:00
Kaloyan Raev
091b49b921 cmd/satellite: command to move unspent bonuses to Stripe balance
Jira: https://storjlabs.atlassian.net/browse/USR-821

The `migrate-credits` billing command checks the available credits
balance for all users and moves it to the Stripe balance by creating a
new credit balance transaction.

Change-Id: Iafc7b95a4edad47f7c145a22e210f8c821ac183d
2020-06-24 13:20:24 +00:00
Cameron Ayer
3b4b5f45c7 satellite: replace references to Suspended with UnknownAuditSuspended
Change-Id: I3d2d00c95954c0546ad077702617895f262926ef
2020-06-23 14:19:22 +00:00
VitaliiShpital
5b3c8b2f1a web/satellite: google tag manager for signup pages
WHAT:
GTM added for partnered satellites sign up pages
csp values were extended to make GTM work at all:
1. googletagmanager.com for GTM script
2. google-analytics.com for GA script
3. hash was added to avoid using 'unsafe-inline' value in 'script-src' directive

Also config flag for GTM id was added

WHY:
Marketing team needs GTM and GA for their campaigns

Change-Id: Ibb2ace737feb971dda6c191599d479fe4a7af332
2020-06-23 10:45:04 +00:00
Isaac Hess
4ce1c099b1 satellite/metainfo: Add RevokeAPIKey to batch
Change-Id: I53b233a5f9f1c8ce7d0bd30e88998f2cdee37762
2020-06-22 13:50:07 -06:00
Isaac Hess
34b019d16a satellite/metainfo: Handle revocation request
Logic to handle a request to revoke a macaroon.

Change-Id: I5d5c93fcc2e026b0aaf82bfdfacc822185b10f9f
2020-06-22 13:50:07 -06:00
Isaac Hess
2d727bb14e satellite: Check macaroon revocation
When a request comes in on the satellite api and we validate the
macaroon, we now also check if any of the macaroon's tails have been
revoked.

Change-Id: I80ce4312602baf431cfa1b1285f79bed88bb4497
2020-06-22 13:50:07 -06:00
Egon Elbre
6cc7fd5f31 satellite/metainfo: remove old endpoints
Change-Id: Ic85c27caeea08879714e8303b285b6136f9a7fd1
2020-06-16 16:12:14 +03:00
Isaac Hess
05a981bc13 satellite/metainfo: Add RevokeAPIKey stub endpoint
Change-Id: I07fb997dda48ea204aab11f656a1a37762684339
2020-06-16 07:03:02 -06:00
Egon Elbre
f68e7b3fde satellite/overlay: replace pb.InfoResponse
pb.InfoResponse wasn't used for protocol buffer communication, but
instead as a satellite type.

Change-Id: I755619f2deec5b76c4fe488591b7d8c1b9fcdafb
2020-06-16 15:16:55 +03:00
Egon Elbre
6673125c03 satellite/metainfo: remove code for handling partner uuid
Change-Id: I2e0fdc6a23288b3d8e55548f1d97598e48588e4c
2020-06-16 11:31:41 +00:00
Egon Elbre
19d431ff38 satellite/metainfo: disable old endpoints
First step in removing endpoints that were used by pre 1.0 clients.

Change-Id: Id861c73ffac97139aa9540f4973ea849e572f906
2020-06-16 11:22:44 +00:00
Ivan Fraixedes
b72bf4a6d8 satellite/admin: Make minor improvements in README
Make some minimal improvements in the README about the API documentation
regarding the required requests and successful response bodies.

Change-Id: If7832f3c40166a55d9baefdb2395211ff9e8dc04
2020-06-15 10:23:20 +00:00
Cameron Ayer
0885ba5646 satellite/satellitedb: add new columns for offline suspension
add new columns `offline_suspended` and `under_review` to nodes table.
`unknown_audit_suspended` is a new column which will replace `suspended`

Change-Id: I22ddeb338ea0ff63f14332a7ebd0f3e9e4c06cdc
2020-06-15 04:00:20 +00:00
paul cannon
7b8e91ff28 satellite/satellitedb: no orders for exited nodes
We should not be sending any type of orders to nodes that have completed
graceful exit with the current satellite. In particular, we should not
be trying to audit them, because that would be silly.

Change-Id: Ie2153e5739914ab696feefcdef28545ed70f84e4
2020-06-13 13:49:33 +00:00
Rafael Gomes
958ea1b9df satellite/accounting: add download limit cache
Change-Id: I722930cab8bd5d240f4878dc6997e9bc7637311f
2020-06-12 16:33:46 -03:00
VitaliiShpital
8ecf01ece8 web/satellite: credit history page implemented
WHAT:
credit history page implemented.
can be visited by clicking specific button in a free credits dropdown.

WHY:
UI didn't display remaining coupon value.
coupons and referral items (in future) are displayed in the same place.

Change-Id: I495fd7a99f2ea5117152aaf8f495bd5322f02588
2020-06-12 14:31:02 +00:00
Michal Niewrzal
e48177ce81 satellite/payments/stripecoinpayments: add basic summary for invoice
generation commands

https://storjlabs.atlassian.net/browse/USR-823

Change-Id: I24a7dec56973abcbbb66e70842e3aa2b7c01a4b3
2020-06-12 09:52:41 +00:00
Egon Elbre
410d897840 satellite: fix string(int) conversions
Change-Id: I54c6ca8c2dad3c321175f72271b7536cc2a4df09
2020-06-12 06:41:34 +00:00
Egon Elbre
d88394c3d2 satellite/metainfo: verify pointer in service
Change-Id: Ib489a177c93739f48d3a9669a56d9892640c74ef
2020-06-11 13:45:58 +00:00
Egon Elbre
1ed5a1bac5 satellite/satellitedb/satellitedbtest: skip omitted database
The first implementation missed some changes.

Change-Id: I7ae696175e0a9ea46954970ba8547638a05ed5a9
2020-06-11 13:28:16 +00:00
Qweder93
0826c8d87f satellite/heldamount: fix dimension of usage_at_rest
Change-Id: If1518ad41736912d15fb2c882c9e236c16f85a51
2020-06-11 13:07:51 +00:00
stefanbenten
e8ef9b76bb satellite/rewards: add CloudBloq
Change-Id: Ic0b7554c68d9055acbde2745a445bfa883c237af
2020-06-10 21:14:45 +02:00
stefanbenten
c6c8b923af satellite/dbcleanup: run cleanup more frequently
As the tables that get cleaned up by this job get a lot of inserts and deletes over the course of a day, the autovacuum process on PostgreSQL struggles fairly easily/quickly.
Due to its limitation, it can only delete 180,000,000 tuples in one go, before it has to rescan the entire table/index.

With the current load, the most busy satellites accumulate about 1,000,000,000 tuples per day (consumed_serials). With our current 24h interval that results in ~6-7 scans, slowing the entire database down for a quite long time.
This PR reduces the interval to 4 hours, which under a constant load, results in less than 180,000,000 entries per run.
That way, we do not scan twice for only a small gain over said amount. Reducing the interval further would also increase the DB load unnecessary, as each run scans the entire tables at least once.

For future reference, we might need to adjust the interval, if the load is significantly changing.

Change-Id: I18fdd45d93d468cff126e719c8380c29a49f43dd
2020-06-10 18:32:15 +00:00
Cameron Ayer
26fce54b11 satellite/audit: increase MinDownloadTimeout in TestReverifySlowDownload
This test failed due to a timeout on a download which is supposed to
succeed. The testplanet default for the value is 5 seconds, but here
it is 500 milliseconds.

It looks like this is due to the fact that later in the test we need to
wait for a slow node to timeout, so we cut the timeout shorter to reduce
test time.

This PR increases the timeout to 1 second. Still not too long to wait, but
gives us twice as much time to download, decreasing the likelihood that we
see the timeout error.

Change-Id: I504db39ab5dc4d3c505520337b258265d6da7020
2020-06-10 17:43:50 +00:00
Cameron Ayer
bad299b541 satellite/satellitedb: serialize UpdateStats and BatchUpdateStats transactions
Since we increased the number of audit workers from 1 to 2, we need to make sure
concurrent updates do not trample each other. We can do this by serializing the
transactions.

Change-Id: If1b2f71cabe3c779c12ffa33c0c3271778ac3ae0
2020-06-10 17:11:28 +00:00
stefanbenten
dfeb4dc9c2 satellite/payments: add finalize invoice command
Change-Id: Ib74dc8fd352e9576a302f5923439f5e50f69c9f0
2020-06-10 14:48:05 +00:00
Qweder93
e52809d53e cmd/storagenode: add check if satellites available to gracefulexit
Change-Id: I8747507593d810bbdec0d140de0600ee147011c3
2020-06-10 13:38:36 +00:00
VitaliiShpital
47a766328f web/satellite: deposit and billing history splitted to be shown separately
WHAT:
1. Deposit & Billing history view was divided to be shown separately as Deposit History and Billing History
2. Datepicker was removed from billing page

WHY:
billing UX enhancements

Change-Id: Ie183849ef0965169997674ce37b71db38a562fc2
2020-06-09 12:32:02 +00:00
Michal Niewrzal
90e2e3c8c4 satellite/payments/stripecoinpayments: display more details about
invoice item generation

Part of https://storjlabs.atlassian.net/browse/USR-823

Change-Id: If0ffa56b68bb0a6b9b4e145fa21468006e7507f6
2020-06-08 23:33:38 +00:00
Egon Elbre
36c461bd59 private/tagsql: track proper closing of rows and statements
This ensures that rows are closed to avoid leaks.
Also verifies that Err() is called, to ensure that no
error is left behind.

Change-Id: Idd1bec9bf479f40021da67b2c80ce83033149469
2020-06-05 18:25:43 +00:00
Jeff Wendling
943eb872d3 satellite/audit: depend less on details of some error message
the error message changed when we removed spacemonkeygo/errors

Change-Id: I4904a072cfd84e4c39c881b58669325bcf51df46
2020-06-05 10:39:05 -06:00
Egon Elbre
34db4a80fd ci: fix staticcheck failures
Change-Id: I176fb24214755a1940a0a1a4e9cc8e39f184870b
2020-06-05 13:15:34 +00:00
Michal Niewrzal
2b2efcc662 satellite/payments/stripecoinpayments: move Coupons expiration date
sorting directly to listing method

Change-Id: I58d8a6ea1feba9ff2d19f21a1dbc87bfb8b49801
2020-06-04 09:47:42 +00:00