Commit Graph

4343 Commits

Author SHA1 Message Date
Jeff Wendling
1944d734ef satellite/orders: check and enforce node api version
Change-Id: Ibdeb1a85dfed8b534bfed32a7cdaae5c3dc8b420
2020-07-16 10:38:12 +00:00
Moby von Briesen
1b807761bd storagenode/orders: Update orders filestore to be compatible with new satellite endpoint
* Instead of archiving a list of orders and deleting an "unsent" file in
separate steps, archival simply moves the old unsent file to a new
archived file
* Add maxInFlightTime to be used along with grace period for sending
buffer
* Create unsent/archival directories in constructor
* Code cleanup

Change-Id: Ia3bc2aaf60cced6c6d413465423d78c7d5151188
2020-07-15 14:21:56 -04:00
stefanbenten
0209a2095f satellite/{console,satellitedb}: add project_limit column to users table
Change-Id: I603f085f17ca5b413dd1c6837c2081f9e7e791a1
2020-07-15 17:27:31 +00:00
stefanbenten
2c2d284f3d satellite/admin: add bucket limit handling endpoint
Change-Id: I4b199277cff30f11f4a9fff3b0ac4017b694f2e8
2020-07-15 17:27:23 +00:00
Jennifer Johnson
784a156eea satellite: prevents uplink from creating a bucket once it exceeds the max bucket allocation.
Change-Id: I4b3822ed723c03dbbc0df136b2201027e19ba0cd
2020-07-15 17:27:05 +00:00
Qweder93
62fec25104 storagenode/heldamount: returns usage_at_rest in tbm instead of tbh
Change-Id: I183a56460ea76a53680ca6861d02cecebe3576ec
2020-07-15 15:46:13 +03:00
Jeff Wendling
3a8766936b satellite/orders: remove race condition in new endpoint tests
the flush batch size was set to 1 which means that a flush was
async scheduled after the first write. the explicit trigger wait
was then always flushing nothing, and the test would only
pass if the async flush was scheduled before the read.

remove that async flush and pause the flush loop so that we are
in full control of when the flushes happen so there are no races.

the tests are still disabled but that's because the endpoint is
still disabled.

Change-Id: I2b7b07fd5525388c30be8efbf4af7105087228da
2020-07-14 15:12:33 -04:00
Caleb Case
581579e80a
satellite/rewards: Add Duplicati to partners database.
Change-Id: I1d702504adc21cc7a243fe67f1856bd13928c369
2020-07-14 12:49:48 -04:00
stefanbenten
257855b5de all: replace == comparison with errors.Is
Change-Id: I05d9a369c7c6f144b94a4c524e8aea18eb9cb714
2020-07-14 15:50:25 +00:00
Qweder93
7b4a8c4d6d storagenode/heldamount: payoutHistory added
Change-Id: I93dd3d024085d19ecff76075e52bf66796207fd6
2020-07-14 17:35:03 +03:00
Jessica Grebenschikov
1f1e3f0604 satellite/orders: disable settlementwithwindow, skip flaky tests
Change-Id: Ia60d7e0f2d383919650cdc736ba4569bb26ff2d7
2020-07-14 07:17:18 -07:00
Qweder93
7d6973b5a2 satellite: heldamount and nodestats not returning error node not found by rpc
Change-Id: Ifb00b16a4a04603251de60da6a6612fd5e98d597
2020-07-14 16:31:02 +03:00
stefanbenten
1149417615 satellite/admin: cleanup parameter handling
We passed in revocationDB and metainfoDB for no reason.
Lets remove it from the dependency list to further reduce the footprint.

Change-Id: Ic0317bb92670fbd305d4a8b0ed1cb82858e2f6d3
2020-07-14 13:53:09 +02:00
stefanbenten
133fda4bef satellite/admin: add user deletion
Change-Id: Ideea978698183e25f5e0d73128c4f93c2caa1577
2020-07-14 11:37:00 +00:00
stefanbenten
0a32ba0e6b satellite/admin: add project rename functionality
Change-Id: I4c0f42d4c2c26859279f247f94cef97a8ff630a9
2020-07-14 11:36:49 +00:00
stefanbenten
f768302c91 satellite/admin: harden project deletion requirements
Change-Id: Ia7ea469f87469b16e464dc22af24b98a6ef1873d
2020-07-14 11:36:29 +00:00
Yaroslav Vorobiov
75ab3507dd storagenode-updater: process storagenode config
Change-Id: I452dd4422ae660ca46d7f4525358c8e809ba0d75
2020-07-14 13:30:14 +03:00
Jessica Grebenschikov
8abb907010 satellite/orders: add settle orders with window
Why: We need a way to cut down on database traffic due to bandwidth
measurement and tracking.

What: This changeset is the Satellite side of settling orders in 1 hr windows.
See design doc for more details: https://review.dev.storj.io/c/storj/storj/+/1732

Change-Id: I2e1c151e2e65516ebe1b7f47b7c5f83a3a220b31
2020-07-13 15:41:29 -07:00
VitaliiShpital
d4bde7848e web/satellite: coupon status replaced with expiration date
WHAT:
coupon status view replaced with month and year of coupon expiration date

WHY:
better User experience

Change-Id: I67abe14418aeb876a524de5e3f3c027407343100
2020-07-13 18:02:44 +00:00
VitaliiShpital
02a39354c6 web/satellite: rel='noopener noreferrer' added to blank links
WHAT:
rels added to target="_blank" links

WHY:
to avoid tabnapping

Change-Id: Iada8ad4e1e1be5d4efd33f5d8c5b14816c2d85e0
2020-07-13 17:21:32 +00:00
Egon Elbre
06a3510ae4 satellite/orders: add EncryptionKey encryption
Add encryption using nacl/secretbox.

Change-Id: I0add31a9fd2359be1bf9d3e43b6c4b9ff3b6fb03
2020-07-13 16:10:19 +00:00
paul cannon
bbdb351e5e all: use jackc/pgx in place of lib/pq
What:

Use the github.com/jackc/pgx postgresql driver in place of
github.com/lib/pq.

Why:

github.com/lib/pq has some problems with error handling and context
cancellations (i.e. it might even issue queries or DML statements more
than once! see https://github.com/lib/pq/issues/939). The
github.com/jackx/pgx library appears not to have these problems, and
also appears to be better engineered and implemented (in particular, it
doesn't use "exceptions by panic"). It should also give us some
performance improvements in some cases, and even more so if we can use
it directly instead of going through the database/sql layer.

Change-Id: Ia696d220f340a097dee9550a312d37de14ed2044
2020-07-13 15:54:41 +00:00
Egon Elbre
262da14359 storagenode/console/consoleapi: disable flaky TestStorageNodeApi
Change-Id: I076c9a46fece86d34eae117ab84f94f99e7e64e0
2020-07-13 18:35:38 +03:00
Isaac Hess
78f5755d46 storagenode/nodestats: Add sat to heldamount error
Nodes are receiving an error that heldamount rpc doesn't exist on a
satellite. This simply adds which satellite to the error.

Change-Id: I7708e0511b55fdd2425969db2a545645339bad81
2020-07-13 14:29:09 +00:00
Qweder93
facde770de storagenode/heldadmount: removed logging errors node not found during getAllPaystubs/Payments from trusted satellites
Change-Id: I87f6c697d98546812450fcfb090623c76dec4bbc
2020-07-13 16:45:23 +03:00
Qweder93
f73e92c268 storagenode/gracefulexit: added blobs clean
on node's start checks if any of trusted satellites has GE status "Exited successfully"
if so - trying to delete blobs/satellite folder, so no trash left on SNO.

Change-Id: I566266c84f2a872df54cd01bc2f15a9934f138ed
2020-07-13 11:49:18 +00:00
Egon Elbre
29ccda5d47 cmd/uplink/cmd: remove spf13/cast dependency
Change-Id: I8013014b0b37966128e673155d851705ee1d43db
2020-07-13 11:10:48 +00:00
Michal Niewrzal
f532d4b25c satellite/payments/stripecoinpayments: add test for invoice items
values

Change-Id: I924b02e70d7ed36b0a769f1072410dd811390abb
2020-07-13 08:28:46 +00:00
Rafael Gomes
24a1eac16c storage/postgreskv: Sort storage keys before delete (postgres)
Change-Id: I63599e142b387ded25d110458ae10c2c96cd8ea6
2020-07-10 20:43:45 +00:00
Rafael Gomes
569b49768e storage/cockroachkv: Sort storage keys before delete (cockroach)
Change-Id: Iee8e56ff66e2760e933f3860d5fc75230a507558
2020-07-10 16:27:26 -04:00
Egon Elbre
9dc9cd8a17 tests: allow STORJ_TEST_POSTGRES
STORJ_POSTGRES_TEST naming was not consistent with STORJ_SIM_POSTGRES.

This allows to use STORJ_TEST_POSTGRES for clarity, it still has a
fallback to STORJ_POSTGRES_TEST.

Change-Id: I6f294c66c80fcfd6750fea2a89795f3b7f5dd691
2020-07-10 16:43:49 +03:00
Egon Elbre
5bdcd86fa7 ci: test benchmarks
This runs each benchmark for one iteration to ensure that they are
valid. Unfortunately, it does not give any useful metrics as output.

Change-Id: I68940398c8dd849aed656bd12656f48d5df10128
2020-07-10 13:26:49 +00:00
Egon Elbre
4869cfc9a4 satellite/vouchers: remove deprecated endpoint
Change-Id: I0a754217d9424253e448126face6594bc143f412
2020-07-10 12:38:46 +00:00
Qweder93
e17243fcd7 storagenode/console: estimation payour for current and previous month reworked
Change-Id: I937d5d8f7c17949b539dcd6e36af27400a5043e2
2020-07-10 12:18:53 +00:00
JT Olio
cdb0200280 satellite: collect hardware stats on satellites
Change-Id: Icc4d9920bb3d23be4d51058237da1baa0e1f3291
2020-07-09 19:17:20 +00:00
Jeff Wendling
885ef70c58 satellite/nodeapiversion: new table for tracking node api usage
This system tracks an abstract "api version" from nodes based on
their usage, allowing us to have latching behavior where if a node
ever uses a new api, it can be blocked from using the old api.

This is better than using self-reported semver version information
because the node cannot lie, there's no confusion about what semver
version implies which features, no questions about dev and ci
environments, and no dependencies between reporting the version
and using the new api.

Change-Id: Ifeced5c9ae8e0a16102d79635e176a7d3bdd8ed4
2020-07-09 15:02:25 +00:00
NickolaiYurchenko
6c5d948b82 web/storagenode: loading screen added
Change-Id: I7d966d2dfba1c275d98259eb7b368297559f9f2d
2020-07-09 11:16:53 +00:00
Stefan Benten
9dbd511396
private/dbutil: reduce db connection defaults (#3920) 2020-07-08 19:59:42 +02:00
Qweder93
0521435e08 storagenode/gracefulexit: added deletion of all files left in storage/blobs/satellite after successful GE
https://storjlabs.atlassian.net/browse/SG-368

Change-Id: I29a978fe0d0153aedf2be91dc7f45b4ef386d447
2020-07-08 14:38:31 +03:00
VitaliiShpital
dd9eaafceb web/satellite: api keys component slightly refactored
Change-Id: I8d2bde0a7d2c2778ba3c906232a1a38aaf94cff3
2020-07-08 10:13:39 +00:00
VitaliiShpital
a26fab2553 web/satellite: balance history dropdown implemented on billing page
WHAT:
1. STORJ Balance renamed to Available Balance
2. Balance history dropdown added

WHY:
better User experience

Change-Id: I4e7186fd285f6d9d1dd4b1ad93142fcf9d3367f3
2020-07-08 09:59:05 +00:00
VitaliiShpital
2ec9de94ba web/satellite: onboarding tour's payment wall updated
WHAT:
1. payment step added to overview screen
2. pricing modal added to payment screen
3. info bar with payment info added to be shown before adding first payment method
4. message on verifying coin payment transaction screen was updated
5. coin payment transaction checkout button added to verifying screen

WHY:
better User experience

Change-Id: Ia3b12a3f7f6c874f19a9414b61afc6ba5a29970b
2020-07-08 09:44:25 +00:00
Bill Thorp
a3c902ab84 storagenode/pieces: hours in a month should be 720
Per https://documentation.tardigrade.io/pricing/billing-and-payment:
"The calculation of per object fees is based on a standard 720-hour month."

On most years, the average value is 730 (365*24/12), except leap years.
However, we want to have ours be 720 (30*24) so its lines up with days.

Change-Id: Ifb9691878f1a7ea81ed36c92b37985493295fe31
2020-07-07 15:26:15 -04:00
Moby von Briesen
e9dd5b2845 storagenode/piecestore: Properly log/send metrics for all successful pieces
When an uplink or repair work finishes uploading a piece to a
storagenode, it has no reason to wait another round trip after the piece
is committed to gracefully close the connection - in many (most?) cases,
the connection is simply canceled once the upload is complete. This has
the unintended side effect of producing a lot of "piece canceled" logs
and metrics on the storagenode side, when the reality is that the piece
uploads were successful, and not really canceled. This commit fixes
that.

Change-Id: Icbc1f7857d380134560219c1c19c186df2783cd0
2020-07-07 15:19:17 +00:00
Jessica Grebenschikov
41497569cd satellite/orders: add settled amount to rollups write cache
https://storjlabs.atlassian.net/browse/SM-1109

Change-Id: Ic5859b141c1384157b33df0d7fb6c8b43cc8a6b1
2020-07-07 14:46:06 +00:00
Kaloyan Raev
12a15e5a6a satellite/payments: remove migare-credits billing command
This was a one-time command and it has been already executed on all
production satellites.

Change-Id: Ic58a151c3e5da9c139df875f6517f37275a1ebf5
2020-07-07 11:47:53 +03:00
Isaac Hess
fd740295ec satellite/satellitedb: Add comment to revocation
Change-Id: I1b65b7e46439c4788835ea5bfd4df3d32a713b44
2020-07-06 21:51:35 +00:00
Caleb Case
db55386fbc satellite/rewards: Add Rclone to partners database.
Change-Id: Ieab098f7fb411230b72a88533ae632bac8967266
2020-07-06 16:13:15 +00:00
JT Olio
78c0d9352d storage: support monkit traces of limit exceeded
errors.New errors will not show up in monkit tracing
as a useful error type. this change fixes a test (!)
and makes it so monkit will tell us what the error
type is, if we have this failure

Change-Id: Ic9933704e4095495c7ee286d9df3eb7eb94b25c9
2020-07-06 15:14:02 +00:00
Bill Thorp
00ae5ebbab satellite/payemnts: Credit coin payments earlier
Apply the coin payments when CoinPayments.net recieves the funds
Instead of the when STORJ gets them from CoinPayments.net

Based on 7/1/20 User Growth standup guidance by JG
Relates to: https://storjlabs.atlassian.net/browse/USR-801

Change-Id: I174ca23a585010f39464c45525e1dfe0179b7c1a
2020-07-06 13:24:26 +00:00