Commit Graph

1243 Commits

Author SHA1 Message Date
Egon Elbre
36c461bd59 private/tagsql: track proper closing of rows and statements
This ensures that rows are closed to avoid leaks.
Also verifies that Err() is called, to ensure that no
error is left behind.

Change-Id: Idd1bec9bf479f40021da67b2c80ce83033149469
2020-06-05 18:25:43 +00:00
Jeff Wendling
943eb872d3 satellite/audit: depend less on details of some error message
the error message changed when we removed spacemonkeygo/errors

Change-Id: I4904a072cfd84e4c39c881b58669325bcf51df46
2020-06-05 10:39:05 -06:00
Egon Elbre
34db4a80fd ci: fix staticcheck failures
Change-Id: I176fb24214755a1940a0a1a4e9cc8e39f184870b
2020-06-05 13:15:34 +00:00
Michal Niewrzal
2b2efcc662 satellite/payments/stripecoinpayments: move Coupons expiration date
sorting directly to listing method

Change-Id: I58d8a6ea1feba9ff2d19f21a1dbc87bfb8b49801
2020-06-04 09:47:42 +00:00
Michal Niewrzal
96286fde47 satellite/payments/stripecoinpayments: list projects by owner, not
by project members

This is a fix for listing the same project twice because project has
more than one member.

Change-Id: I3f6fe3456a6753d6d091a64436c22027dcbe2520
2020-06-04 08:42:01 +00:00
Jeff Wendling
254b42ff65 satellite/satellitedb: fix leaked rows from repairQueue.Insert
Change-Id: If5e62c49770f591ebe3f4d2dd4dd2658c229a022
2020-06-03 14:31:21 -06:00
Michal Niewrzal
b20ced9519 satellite/satellitedb: drop project_id column from coupons table
This is last part of https://storjlabs.atlassian.net/browse/USR-818

Change-Id: I053d11b37df962c12e46645bae2fc2dad49c9755
2020-06-03 14:56:41 +00:00
Kaloyan Raev
fbfc3e5d18 satellite/payments: adjust label for bonus
This change adjusts the label for STORJ deposit bonuses in billing
history to be more consistent with other labels.

Change-Id: I5e7179ae3ac52dafb0dcef084e9a7c4742491f9e
2020-06-03 13:15:14 +00:00
Cameron Ayer
6a60e1e96b satellite/satellitedb: inclusive interval_start in GetAllocatedBandwidthTotal
The DB query in GetAllocatedBandwidthTotal uses an exclusive range:

'WHERE interval_start > ?'

The value that is used for this condition is the first day of current the month,
00:00:00 UTC.

By using the exclusive '>', we exclude the entire first hour of the month from the
result set.

Change-Id: I3ed300f5230c7514dc9495a85e8166213cd0842e
2020-06-02 13:06:45 -04:00
VitaliiShpital
c272872d5d satellite/payments: available coupon value feature
Change-Id: Ieae9385fbd917230298afff91a6be2838ad9b313
2020-06-02 10:00:45 +00:00
Michal Niewrzal
e8c4010311 satellite/metainfo: add missing error check
Change-Id: I4043633505a3eaee8304f71f5ddea17de2b3d10f
2020-06-02 09:38:55 +00:00
Jeff Wendling
2b3545c0c3 satellite/satellitedb: use delete returning to query pending_serial_queue
this way we don't have to do 2 steps, and by using the ctid, postgres
is going to do two very efficient prefix scans.

Change-Id: Ia9d0546cdf0a1af67ceec9cd508d336a5fdcbdb9
2020-06-01 15:43:33 -06:00
Moby von Briesen
b82d04e618 satellite/metainfo: limit size of uplink-provided metadata to 2KiB
Change-Id: Id44a46046ddb4a12102525531f4502fcff2b6252
2020-06-01 16:51:29 -04:00
Jeff Wendling
44433f38be satellite/satellitedb: remove ORDER BY when reading from queue
also remove the continuation support from the queue, otherwise
we may end up sequential scanning the entire table to get
a few rows at the end.

then, in the core, instead of looping both to get a big enough
batch inside of the queue, as well as outside of it to ensure
we consume the whole queue, just get a single batch at a time.

also, make the queue size configurable because we'll need to
do some tuning in production.

Change-Id: If1a997c6012898056ace89366a847c4cb141a025
2020-06-01 18:31:14 +00:00
Yingrong Zhao
163c027a6d satellite/satellitedb: remove monkit trace from convertDBNode
In jaeger, it shows that this function gets called repetitively in
a single request. Most of the time, it's less than 1ms. Therefore, it
doesn't add much value in our trace but create noises.

Change-Id: I20234f36bbcf0fc22f91e5e1a5634c0cad577ed0
2020-06-01 17:58:43 +00:00
Kaloyan Raev
d66e646b57 satellite/payments: add deposit bonus to stripe balance
Jira issue: https://storjlabs.atlassian.net/browse/USR-820

The bonus for depositing STORJ tokens is now added as to the Stripe
balance instead of the to `credits` DB table on the satellite.

Existing unspent bonuses in the `credits` DB table are still processed
as usual when generating invoices. They will be migrated to the Stripe
balance with a separate change.

The bonus is added to the Stripe balance with a separate Credit
transaction. The balance transactions for the deposit and the bonus can
be differentiate by their different description.

The billing history is modified to list the bonus from the Stripe
transactions list.

The workflow for depositing STORJ tokens to the Stripe balance is
improved to survive failures in the middle of the process.

Change-Id: I6a1017984eae34e97c580f9093f7e51ca417b962
2020-06-01 17:41:10 +00:00
Kaloyan Raev
948cc7333d satellite/payments: mark coupon as expired on its last month
Jira issue: https://storjlabs.atlassian.net/browse/USR-110

When generating invoices, if a coupon is not fully spent, but this is
its last valid month, it will be marked as expired in the database.

Change-Id: Ib1b2375f6703f9514d1ae089a387aa029b62d1fb
2020-06-01 17:10:00 +00:00
Michal Niewrzal
a9f6489663 satellite/payments/stripecoinpayments: remove ProjectID from Coupon
struct

This change is removing ProjectID from code. Next change will be about
dropping this colum from DB table.

Change-Id: Idb949e2829e2c304a2b6b011259c7cc7667082e1
2020-06-01 11:37:20 +00:00
Michal Niewrzal
1b69d20eb3 satellite/payments/stripecoinpayments: apply coupons in the order of
their expiration date

Change-Id: Ib013899d82715adf7b3002f2067d6481ceb59de2
2020-06-01 11:01:04 +00:00
VitaliiShpital
ee9bb0d689 satellite/payments: coupon expiration bug fixed
Change-Id: Icc89e9ee6a1dd91109d34909490ee446b716e2ed
2020-06-01 12:20:08 +03:00
Jeff Wendling
839a6f15d1 stripecoinpayments: use fixed date for test
the test as written fails if it's the last day of the month because
the two tallies that are added span two different periods.

Change-Id: I294397278508035ce640500fc3b1b0ffd82b9b71
2020-06-01 09:02:52 +02:00
Michal Niewrzal
f6452e9c09 satellite/payments/stripecoinpayments: process customers instead of
projects

Currently, to generate invoices we are listing all projects. This is
problematic because we decided that elements like e.g. coupons should be
calculated per user. We want to have
more user oriented processing. With this change to prepare invoice
elements we are:
* listing Stripe customers
* listing projects for each customer
* processing projects to create project records
* calculating coupons per customer
* calculating credit spending per customer

Change-Id: I48b0063e44e4d301f97c87ecffb342649359968a
2020-05-30 13:16:07 +00:00
littleskunk
801a3ab90d
satellite/coinpayments: Reduce update interval to 2 minutes (#3897)
* satellite/coinpayments: Reduce update interval to 2 minutes

* satellite/coinpayments: Reduce balance update

Co-authored-by: paul cannon <thepaul@users.noreply.github.com>
2020-05-29 22:21:27 +02:00
VitaliiShpital
c9b9c686fc web/satellite: logic for new signup/login flow
WHAT:
1. updated verification page URL in config
2. added list of partnered satellites to config
3. added logic for satellites dropdown on new signup/login pages

WHY:
1. signup/login flow was reworked in tardigrade.io repo (iframe removed, new pages etc.)
2. new config flag was added to check if satellite name matches at least one member of partnered satellites list to redirect user to verification page
3. new pages will have dropdown with partnered satellites list. Appropriate logic was added.

Change-Id: I33399ab66ca31f07b297a433f6b1f41da4cb6e66
2020-05-29 17:11:44 +00:00
Moby von Briesen
e7e69f383a satellite/repair/repairer/ec.go: Add monkit tracing for ec repairer
Adds monkit tracing for ecrepairer.downloadAndVerifyPiece and
ecrepairer.putPiece so we can get more accurate estimates of node
performance during repair.

Change-Id: Ic05025bf3c493bb3d6f5d325d090c5b7c9e5465d
2020-05-29 14:00:45 +00:00
Egon Elbre
d6c90b7ab5 pkg/macaroon: remove aliased package
Change-Id: I8b06e9a794d00752948f190623006e1afffdb031
2020-05-29 13:34:20 +00:00
Egon Elbre
07050eea26 all: use common/storj
Change-Id: Id1e36d52f9807b5ffbb72ce73f4b60cb21b68a78
2020-05-29 11:57:32 +03:00
Moby von Briesen
acf8b72cd0 satellite/repair/repairer: cut off long tail when minimum number of required uploads is met
This will speed up the Put step of repair by not waiting to time out for
a handful of slow nodes, at the expense of a slightly less durable
pointer. It will still repair to the optimal threshold, but not every
node that is selected will end up in the pointer.

Change-Id: I02a0658e3fe6fc0383f26af0f50a065b8b11a651
2020-05-28 16:25:28 -04:00
Jeff Wendling
1e065fb450 satellite: migration to fix bad imported payment history
the initial calculations for the historical values of comp_at_rest
were wrong. because our historical data only included total amounts
as well as compensation for bandwidth, the at rest value was
calculated as

    at_rest = total - bandwidth

unfortunately, that calculation did not take surge pricing into
account correctly. the at rest and bandwidth values do not
include surge pricing, but the total that was used did. so what
we actually calculated was

    no_surge_at_rest = surge_total - no_surge_bandwidth

which will create a value that is too large. this migration
fixes the calculation for imports that are old enough and
of a non-negligable difference.

Change-Id: I61eb0b670510f6d7fb8fc3de39ba79150fac10eb
2020-05-28 12:59:08 -06:00
paul cannon
7395dd1e6e storagenode/gracefulexit: revalidate existing pieces
..before they are transferred to another node and submitted to the
satellite as successful piece transfers, because if we submit an invalid
signature, the node will be marked as a cheater and disqualified
immediately.

These signatures should have been validated when the piece was
originally stored, but bitrot does happen and needn't be cause for an
immediate DQ.

Change-Id: I8b0ebd5812ea8a2e60766005b7251fbb74ef7857
2020-05-28 09:50:14 -05:00
paul cannon
0c9a4a5e56 satellite/gracefulexit: better validation error messages
Change-Id: I14168dbeaf17302e5e853854956f8fbcb82a0900
2020-05-28 09:41:15 -05:00
Michal Niewrzal
84892631c8 private/testplanet: remove old libuplink from testplanet
Change-Id: Ib1553f84d0b3ae12a5b00382f0f53357b6a273e2
2020-05-28 13:50:23 +00:00
Ethan
d5dfeca84d satellite/payments: Return empty iterator instead of nil from StripeMock List methods
Some satellites do not have payment configured (ex. Salt Lake, Europe North). In this case the StripeMock is used, which returns nil on invoice and charges List methods. This causes a panic.

https://storjlabs.atlassian.net/browse/SM-978

Change-Id: Iec1b0bfd9b383e6f793d03dd63a3dec60e0fd9f3
2020-05-27 16:20:18 +00:00
Kaloyan Raev
ee7de0424b satellite/payments: use quantities in invoices
Jira issue: https://storjlabs.atlassian.net/browse/USR-719

Invoices now show the amount of used resources and their unit price.

This change also makes proper rounding to the nearest cent in few places
to resolve the "off-by-one-cent" issue observed in few invoices.

Change-Id: I2d70d6916b5cf4a9a9138c99c422f5f4d2deb35b
2020-05-27 09:33:12 +00:00
Michal Niewrzal
75b3db5426 satellite/payments/stripecoinpayments: test invoice user with more than
1 project

https://storjlabs.atlassian.net/browse/USR-291

Change-Id: I98286e40254e8868de9eb675a9c9a8cd0bf5f3b1
2020-05-27 09:12:23 +00:00
Moby von Briesen
290c006a10 satellite/repair/{checker,queue}: add metric for new segments added to repair queue
* add monkit stat new_remote_segments_needing_repair, which reports the
number of new unhealthy segments in the repair queue since the previous
checker iteration

Change-Id: I2f10266006fdd6406ece50f4759b91382059dcc3
2020-05-27 06:23:47 +00:00
Stefan Benten
ed857f0fb4
satellite/admin: Add the proper documentation for bandwidth limits (#3900) 2020-05-27 09:22:45 +03:00
Egon Elbre
c5d4a13158 satellite/nodeselection: use NodeURL
Change-Id: I2ebd4dbf993ff5c7864f3a3a665b5c8fc48aa7d1
2020-05-27 05:46:11 +00:00
Cameron Ayer
9edb363558 satellite/metainfo: log if replacing pieces in pointer goes wrong
Every now and then we see the repair error, "piece to add already exists".
With these new logs we should be able to verify if it is due to a change in
the pointer. These logs are only temporary

Change-Id: I029390cc4816668707546df14ed2cfe7ca192b0b
2020-05-26 21:48:40 +00:00
Michal Niewrzal
88a2561317 satellite/payments/stripecoinpayments: don't request anything from
Coinpayments if credentials are missing

Workaround for hitting Coinpayment API while tests.

Change-Id: I27114c820d4d7e17d05b066f55a13a709dac1711
2020-05-26 19:38:12 +02:00
Ethan
7014cf2083 satellite/gracefulexit: Change handleFailed to return nil if we can't get the pending transfer
https://storjlabs.atlassian.net/browse/SM-975

Change-Id: Ia1746b5e274e5011b6cd9f5a52f9a5faf703be51
2020-05-26 13:07:38 +00:00
littleskunk
2fbb34c3ea
nodeselection: Increase minimum free space to 500MB (#3898) 2020-05-25 12:13:28 +02:00
Jeff Wendling
074649835b satellite/satellitedb: add some docs and improve some snapshots
This attempts to add a README.md to help create consistent migrations
that maximize our test coverage and do not include unnecessary
statements.

It also adds a feature to have an `-- OLD DATA --` section as well
as a `-- NEW DATA --` section so that we can fix mistakes made in
previous snapshots (like a row that was forgotten to be added when a
table was created) without editing them going forward.

Change-Id: I28a786f8ef163cae1de1bb08f61af1e1104b0a88
2020-05-22 21:27:36 +00:00
Michal Niewrzal
5c10964040 satellite/payments/stripecoinpayments: add test for listing issues while
invoice generation

https://review.dev.storj.io/c/storj/storj/+/1853
https://review.dev.storj.io/c/storj/storj/+/1882

Change-Id: Ie71363b819866dd60dbe7117b42cfa8348479310
2020-05-22 17:24:16 +00:00
Michal Niewrzal
83cc80ef06 satellite/payments/stripecoinpayments: fix listing issue for coupon
usages and credit spendings

Test will be added as a separate change.

Change-Id: I9341d2ecb736fdbd45beb96f778f04448df68bd7
2020-05-21 07:48:04 +00:00
Egon Elbre
bef84a5f9d storagenode: remove dependency to overlay.NodeDossier
This is the last dependency from storage node to satellite.

Change-Id: I12f7abb91e84f823ba5af126c6e2979519838612
2020-05-21 08:37:13 +03:00
Egon Elbre
b42778c42e private/testplanet: remove some additional Local-s
Change-Id: I49701c41efb92efca27cc18d0a3f6d6b44d3cf8b
2020-05-21 08:37:13 +03:00
Egon Elbre
7f323754a4 metainfo/piecedeletion: use NodeURL-s
Change-Id: I247dbfe03e7864e940e4cd1d0f343f38e84099e0
2020-05-21 08:37:13 +03:00
Jennifer Johnson
03e5f922c3 satellite/overlay: updates node with a vetted_at timestamp if they meet the vetting criteria
What: As soon as a node passes the vetting criteria (total_audit_count and total_uptime_count
are greater than the configured thresholds), we set vetted_at to the current timestamp.

Why: We may want to use this timestamp in future development to select new vs vetted nodes.
It also allows flexibility in node vetting experiments and allows for better metrics around
vetting times.

Please describe the tests: satellitedb_test: TestUpdateStats and TestBatchUpdateStats make sure vetted_at is set appropriately
Please describe the performance impact: This change does add extra logic to BatchUpdateStats and UpdateStats and
commits another variable to the db (vetted_at), but this should be negligible.

Change-Id: I3de804549b5f1bc359da4935bc859758ceac261d
2020-05-20 16:30:26 -04:00
Michal Niewrzal
fe6a6f063f private/testplanet: cleanup predefined data generation
Use Console service to create user and project instead direct DB
modifications.

Change-Id: Ib0074b38313b3dc43b7d8d63ab2775d29028fb7b
2020-05-20 12:38:43 +00:00