Commit Graph

1302 Commits

Author SHA1 Message Date
stefanbenten
1149417615 satellite/admin: cleanup parameter handling
We passed in revocationDB and metainfoDB for no reason.
Lets remove it from the dependency list to further reduce the footprint.

Change-Id: Ic0317bb92670fbd305d4a8b0ed1cb82858e2f6d3
2020-07-14 13:53:09 +02:00
stefanbenten
133fda4bef satellite/admin: add user deletion
Change-Id: Ideea978698183e25f5e0d73128c4f93c2caa1577
2020-07-14 11:37:00 +00:00
stefanbenten
0a32ba0e6b satellite/admin: add project rename functionality
Change-Id: I4c0f42d4c2c26859279f247f94cef97a8ff630a9
2020-07-14 11:36:49 +00:00
stefanbenten
f768302c91 satellite/admin: harden project deletion requirements
Change-Id: Ia7ea469f87469b16e464dc22af24b98a6ef1873d
2020-07-14 11:36:29 +00:00
Jessica Grebenschikov
8abb907010 satellite/orders: add settle orders with window
Why: We need a way to cut down on database traffic due to bandwidth
measurement and tracking.

What: This changeset is the Satellite side of settling orders in 1 hr windows.
See design doc for more details: https://review.dev.storj.io/c/storj/storj/+/1732

Change-Id: I2e1c151e2e65516ebe1b7f47b7c5f83a3a220b31
2020-07-13 15:41:29 -07:00
Egon Elbre
06a3510ae4 satellite/orders: add EncryptionKey encryption
Add encryption using nacl/secretbox.

Change-Id: I0add31a9fd2359be1bf9d3e43b6c4b9ff3b6fb03
2020-07-13 16:10:19 +00:00
paul cannon
bbdb351e5e all: use jackc/pgx in place of lib/pq
What:

Use the github.com/jackc/pgx postgresql driver in place of
github.com/lib/pq.

Why:

github.com/lib/pq has some problems with error handling and context
cancellations (i.e. it might even issue queries or DML statements more
than once! see https://github.com/lib/pq/issues/939). The
github.com/jackx/pgx library appears not to have these problems, and
also appears to be better engineered and implemented (in particular, it
doesn't use "exceptions by panic"). It should also give us some
performance improvements in some cases, and even more so if we can use
it directly instead of going through the database/sql layer.

Change-Id: Ia696d220f340a097dee9550a312d37de14ed2044
2020-07-13 15:54:41 +00:00
Michal Niewrzal
f532d4b25c satellite/payments/stripecoinpayments: add test for invoice items
values

Change-Id: I924b02e70d7ed36b0a769f1072410dd811390abb
2020-07-13 08:28:46 +00:00
Egon Elbre
9dc9cd8a17 tests: allow STORJ_TEST_POSTGRES
STORJ_POSTGRES_TEST naming was not consistent with STORJ_SIM_POSTGRES.

This allows to use STORJ_TEST_POSTGRES for clarity, it still has a
fallback to STORJ_POSTGRES_TEST.

Change-Id: I6f294c66c80fcfd6750fea2a89795f3b7f5dd691
2020-07-10 16:43:49 +03:00
Egon Elbre
5bdcd86fa7 ci: test benchmarks
This runs each benchmark for one iteration to ensure that they are
valid. Unfortunately, it does not give any useful metrics as output.

Change-Id: I68940398c8dd849aed656bd12656f48d5df10128
2020-07-10 13:26:49 +00:00
Egon Elbre
4869cfc9a4 satellite/vouchers: remove deprecated endpoint
Change-Id: I0a754217d9424253e448126face6594bc143f412
2020-07-10 12:38:46 +00:00
JT Olio
cdb0200280 satellite: collect hardware stats on satellites
Change-Id: Icc4d9920bb3d23be4d51058237da1baa0e1f3291
2020-07-09 19:17:20 +00:00
Jeff Wendling
885ef70c58 satellite/nodeapiversion: new table for tracking node api usage
This system tracks an abstract "api version" from nodes based on
their usage, allowing us to have latching behavior where if a node
ever uses a new api, it can be blocked from using the old api.

This is better than using self-reported semver version information
because the node cannot lie, there's no confusion about what semver
version implies which features, no questions about dev and ci
environments, and no dependencies between reporting the version
and using the new api.

Change-Id: Ifeced5c9ae8e0a16102d79635e176a7d3bdd8ed4
2020-07-09 15:02:25 +00:00
Jessica Grebenschikov
41497569cd satellite/orders: add settled amount to rollups write cache
https://storjlabs.atlassian.net/browse/SM-1109

Change-Id: Ic5859b141c1384157b33df0d7fb6c8b43cc8a6b1
2020-07-07 14:46:06 +00:00
Kaloyan Raev
12a15e5a6a satellite/payments: remove migare-credits billing command
This was a one-time command and it has been already executed on all
production satellites.

Change-Id: Ic58a151c3e5da9c139df875f6517f37275a1ebf5
2020-07-07 11:47:53 +03:00
Isaac Hess
fd740295ec satellite/satellitedb: Add comment to revocation
Change-Id: I1b65b7e46439c4788835ea5bfd4df3d32a713b44
2020-07-06 21:51:35 +00:00
Caleb Case
db55386fbc satellite/rewards: Add Rclone to partners database.
Change-Id: Ieab098f7fb411230b72a88533ae632bac8967266
2020-07-06 16:13:15 +00:00
Bill Thorp
00ae5ebbab satellite/payemnts: Credit coin payments earlier
Apply the coin payments when CoinPayments.net recieves the funds
Instead of the when STORJ gets them from CoinPayments.net

Based on 7/1/20 User Growth standup guidance by JG
Relates to: https://storjlabs.atlassian.net/browse/USR-801

Change-Id: I174ca23a585010f39464c45525e1dfe0179b7c1a
2020-07-06 13:24:26 +00:00
littleskunk
e1a18ce974
satellite/accounting: bugfix infinit storage usage increase (#3916)
* satellite/accounting: bugfix infinit storage usage increase

* support int64 values

Co-authored-by: JT Olio <hello@jtolio.com>
2020-07-03 12:34:13 -06:00
Cameron Ayer
e3088d9ad5 satellite/satellitedb: add new DB table audit_histories
Change-Id: I5f854514994cab9a68cf978f2dabfb588df695f5
2020-07-01 21:14:35 +00:00
Qweder93
b639ec08d4 satellite/heldamount: payments added, endpoind for payments added
Change-Id: Ia2b9580bc353ef614680230c6f82c5bf6ded49c4
2020-07-01 18:15:01 +03:00
Bill Thorp
66661c7486 satellite/coinpayments: query status of >25 coinpayments
Allow more than 25 coin payment statuses to be queries from coinpayments.net
Add some slight logging, a coinpayments duration metric, and a disabled test

These changes are small changes in support of https://storjlabs.atlassian.net/browse/USR-801

Change-Id: I5b176cdd5e513d8bd88b92f9b22a8bd2456bbdd5
2020-07-01 09:04:16 +00:00
Cameron Ayer
cadb435d25 {satellite/audit, private/testplanet}: remove ErrAlreadyExists, run 2 audit workers in testplanet
Since we increased the number of concurrent audit workers to two, there are going
to be instances of a single node being audited simultaneously for different segments.
If the node times out for both, we will try to write them both to the pending audits
table, and the second will return an error since the path is not the same as what
already exists. Since with concurrent workers this is expected, we will log the
occurrence rather than return an error.

Since the release default audit concurrency is 2, update testplanet default to run with
concurrent workers as well.

Change-Id: I4e657693fa3e825713a219af3835ae287bb062cb
2020-06-30 18:00:07 +00:00
Egon Elbre
735dc6e163 satellite/orders: add encryption keys
This adds EncryptionKey definition that can be used as a flag.
These order.EncryptionKey-s will be used to encrypt data in
order limits.

This helps to avoid storing lots of transient data in the
main database.

This code doesn't yet contain encryption itself.

Change-Id: I2efae102a89b851d33342a0106f8d8b3f35119bb
2020-06-30 15:03:14 +00:00
Egon Elbre
3567b49ef4 satellite/metainfo/piecedeletion: fix int to string conversion
Change-Id: I8d0cdec7cee182ade479c0cfb1d38f7f97af2ab7
2020-06-29 17:22:52 +03:00
paul cannon
4997fd55d0 satellite/repair: remove healthy from irreparabledb
Change-Id: Ia9d300d0359883f03734d0bdf204d56d6642ce34
2020-06-26 21:26:00 +00:00
Rafael Gomes
bdaabd611d satellite/metainfo/piecedeletion: add metrics for deletion requests
Change-Id: I48ad96e78dab84b9238c63d62bda679fc65c2072
2020-06-26 11:12:33 -03:00
Egon Elbre
d91cf5f4de satellite/satellitedb: add missing SeparateTx
Change-Id: I3ba5a4e0632a1e0e5e77c30e515953eadf05bc45
2020-06-26 12:27:05 +03:00
Egon Elbre
13a5854535 satellite/satellitedb: clarify test migration merging
Use a field to distinguish migration steps that need to use a
different transaction from previous steps. This is clearer than
using a func.

Change-Id: I2147369d05413f3e8ddb50c71a46ab1ba3ab5114
2020-06-25 14:32:45 +00:00
Egon Elbre
df8cf8f58a satellite/orders: delete unused code
Change-Id: I431c8cc2f23e538c676d6f742fb1faef7cc1d73e
2020-06-25 16:48:26 +03:00
Kaloyan Raev
091b49b921 cmd/satellite: command to move unspent bonuses to Stripe balance
Jira: https://storjlabs.atlassian.net/browse/USR-821

The `migrate-credits` billing command checks the available credits
balance for all users and moves it to the Stripe balance by creating a
new credit balance transaction.

Change-Id: Iafc7b95a4edad47f7c145a22e210f8c821ac183d
2020-06-24 13:20:24 +00:00
Cameron Ayer
3b4b5f45c7 satellite: replace references to Suspended with UnknownAuditSuspended
Change-Id: I3d2d00c95954c0546ad077702617895f262926ef
2020-06-23 14:19:22 +00:00
VitaliiShpital
5b3c8b2f1a web/satellite: google tag manager for signup pages
WHAT:
GTM added for partnered satellites sign up pages
csp values were extended to make GTM work at all:
1. googletagmanager.com for GTM script
2. google-analytics.com for GA script
3. hash was added to avoid using 'unsafe-inline' value in 'script-src' directive

Also config flag for GTM id was added

WHY:
Marketing team needs GTM and GA for their campaigns

Change-Id: Ibb2ace737feb971dda6c191599d479fe4a7af332
2020-06-23 10:45:04 +00:00
Isaac Hess
4ce1c099b1 satellite/metainfo: Add RevokeAPIKey to batch
Change-Id: I53b233a5f9f1c8ce7d0bd30e88998f2cdee37762
2020-06-22 13:50:07 -06:00
Isaac Hess
34b019d16a satellite/metainfo: Handle revocation request
Logic to handle a request to revoke a macaroon.

Change-Id: I5d5c93fcc2e026b0aaf82bfdfacc822185b10f9f
2020-06-22 13:50:07 -06:00
Isaac Hess
2d727bb14e satellite: Check macaroon revocation
When a request comes in on the satellite api and we validate the
macaroon, we now also check if any of the macaroon's tails have been
revoked.

Change-Id: I80ce4312602baf431cfa1b1285f79bed88bb4497
2020-06-22 13:50:07 -06:00
Egon Elbre
6cc7fd5f31 satellite/metainfo: remove old endpoints
Change-Id: Ic85c27caeea08879714e8303b285b6136f9a7fd1
2020-06-16 16:12:14 +03:00
Isaac Hess
05a981bc13 satellite/metainfo: Add RevokeAPIKey stub endpoint
Change-Id: I07fb997dda48ea204aab11f656a1a37762684339
2020-06-16 07:03:02 -06:00
Egon Elbre
f68e7b3fde satellite/overlay: replace pb.InfoResponse
pb.InfoResponse wasn't used for protocol buffer communication, but
instead as a satellite type.

Change-Id: I755619f2deec5b76c4fe488591b7d8c1b9fcdafb
2020-06-16 15:16:55 +03:00
Egon Elbre
6673125c03 satellite/metainfo: remove code for handling partner uuid
Change-Id: I2e0fdc6a23288b3d8e55548f1d97598e48588e4c
2020-06-16 11:31:41 +00:00
Egon Elbre
19d431ff38 satellite/metainfo: disable old endpoints
First step in removing endpoints that were used by pre 1.0 clients.

Change-Id: Id861c73ffac97139aa9540f4973ea849e572f906
2020-06-16 11:22:44 +00:00
Ivan Fraixedes
b72bf4a6d8 satellite/admin: Make minor improvements in README
Make some minimal improvements in the README about the API documentation
regarding the required requests and successful response bodies.

Change-Id: If7832f3c40166a55d9baefdb2395211ff9e8dc04
2020-06-15 10:23:20 +00:00
Cameron Ayer
0885ba5646 satellite/satellitedb: add new columns for offline suspension
add new columns `offline_suspended` and `under_review` to nodes table.
`unknown_audit_suspended` is a new column which will replace `suspended`

Change-Id: I22ddeb338ea0ff63f14332a7ebd0f3e9e4c06cdc
2020-06-15 04:00:20 +00:00
paul cannon
7b8e91ff28 satellite/satellitedb: no orders for exited nodes
We should not be sending any type of orders to nodes that have completed
graceful exit with the current satellite. In particular, we should not
be trying to audit them, because that would be silly.

Change-Id: Ie2153e5739914ab696feefcdef28545ed70f84e4
2020-06-13 13:49:33 +00:00
Rafael Gomes
958ea1b9df satellite/accounting: add download limit cache
Change-Id: I722930cab8bd5d240f4878dc6997e9bc7637311f
2020-06-12 16:33:46 -03:00
VitaliiShpital
8ecf01ece8 web/satellite: credit history page implemented
WHAT:
credit history page implemented.
can be visited by clicking specific button in a free credits dropdown.

WHY:
UI didn't display remaining coupon value.
coupons and referral items (in future) are displayed in the same place.

Change-Id: I495fd7a99f2ea5117152aaf8f495bd5322f02588
2020-06-12 14:31:02 +00:00
Michal Niewrzal
e48177ce81 satellite/payments/stripecoinpayments: add basic summary for invoice
generation commands

https://storjlabs.atlassian.net/browse/USR-823

Change-Id: I24a7dec56973abcbbb66e70842e3aa2b7c01a4b3
2020-06-12 09:52:41 +00:00
Egon Elbre
410d897840 satellite: fix string(int) conversions
Change-Id: I54c6ca8c2dad3c321175f72271b7536cc2a4df09
2020-06-12 06:41:34 +00:00
Egon Elbre
d88394c3d2 satellite/metainfo: verify pointer in service
Change-Id: Ib489a177c93739f48d3a9669a56d9892640c74ef
2020-06-11 13:45:58 +00:00
Egon Elbre
1ed5a1bac5 satellite/satellitedb/satellitedbtest: skip omitted database
The first implementation missed some changes.

Change-Id: I7ae696175e0a9ea46954970ba8547638a05ed5a9
2020-06-11 13:28:16 +00:00