Commit Graph

4131 Commits

Author SHA1 Message Date
Ivan Fraixedes
dc5502cb81 private: Prepare pkg for enabling gosec
Prepare package for enabling gosec linter.

Change-Id: I0cce91d83969385f95e5bf82269d6c23629e04a0
2020-06-11 12:00:52 +00:00
Moby von Briesen
be59727790 storagenode/orders: Add archival functionality to orders filestore
* Allow orders to be archived after being settled successfully with the
satellite.
* Allow for cleanup of orders that were archived before a certain time.
* Rewrite other parts of the orders file store to work better with new
design.

Change-Id: I39bea96d80e66a324ec522745169bd6d8b351751
2020-06-11 08:47:37 +00:00
stefanbenten
e8ef9b76bb satellite/rewards: add CloudBloq
Change-Id: Ic0b7554c68d9055acbde2745a445bfa883c237af
2020-06-10 21:14:45 +02:00
stefanbenten
c6c8b923af satellite/dbcleanup: run cleanup more frequently
As the tables that get cleaned up by this job get a lot of inserts and deletes over the course of a day, the autovacuum process on PostgreSQL struggles fairly easily/quickly.
Due to its limitation, it can only delete 180,000,000 tuples in one go, before it has to rescan the entire table/index.

With the current load, the most busy satellites accumulate about 1,000,000,000 tuples per day (consumed_serials). With our current 24h interval that results in ~6-7 scans, slowing the entire database down for a quite long time.
This PR reduces the interval to 4 hours, which under a constant load, results in less than 180,000,000 entries per run.
That way, we do not scan twice for only a small gain over said amount. Reducing the interval further would also increase the DB load unnecessary, as each run scans the entire tables at least once.

For future reference, we might need to adjust the interval, if the load is significantly changing.

Change-Id: I18fdd45d93d468cff126e719c8380c29a49f43dd
2020-06-10 18:32:15 +00:00
Cameron Ayer
26fce54b11 satellite/audit: increase MinDownloadTimeout in TestReverifySlowDownload
This test failed due to a timeout on a download which is supposed to
succeed. The testplanet default for the value is 5 seconds, but here
it is 500 milliseconds.

It looks like this is due to the fact that later in the test we need to
wait for a slow node to timeout, so we cut the timeout shorter to reduce
test time.

This PR increases the timeout to 1 second. Still not too long to wait, but
gives us twice as much time to download, decreasing the likelihood that we
see the timeout error.

Change-Id: I504db39ab5dc4d3c505520337b258265d6da7020
2020-06-10 17:43:50 +00:00
Cameron Ayer
bad299b541 satellite/satellitedb: serialize UpdateStats and BatchUpdateStats transactions
Since we increased the number of audit workers from 1 to 2, we need to make sure
concurrent updates do not trample each other. We can do this by serializing the
transactions.

Change-Id: If1b2f71cabe3c779c12ffa33c0c3271778ac3ae0
2020-06-10 17:11:28 +00:00
NickolaiYurchenko
95a7403802 web/storagenode: uptime section removed
Change-Id: I0918d2fdf71e40e70d0b7ed0045de3067195e843
2020-06-10 15:26:20 +00:00
stefanbenten
44792f7b45 cmd/satellite: add finalize invoices command
Change-Id: Ie57dedc9b9c8fe133dede573c921dfee0883e5ff
2020-06-10 14:48:15 +00:00
stefanbenten
dfeb4dc9c2 satellite/payments: add finalize invoice command
Change-Id: Ib74dc8fd352e9576a302f5923439f5e50f69c9f0
2020-06-10 14:48:05 +00:00
stefanbenten
fdbb2c3ed6 cmd/satellite: organize subcommands
Change-Id: If45575656e5e16642bdee4b6e6867e4f1d824f9a
2020-06-10 14:35:21 +00:00
NickolaiYurchenko
dd1fabe2b0 web/satellite: added no payout data state reset
Change-Id: I950e88d957366e9bd058ea60fe22478d69f5dae6
2020-06-10 14:14:19 +00:00
Qweder93
e52809d53e cmd/storagenode: add check if satellites available to gracefulexit
Change-Id: I8747507593d810bbdec0d140de0600ee147011c3
2020-06-10 13:38:36 +00:00
Moby von Briesen
0b109c32e4 storagenode/piecestore/usedserials: add monkit metric for serials that are randomly deleted
This will give storagenode operators a better idea of whether the memory
allocated to the usedserials store is sufficient.

Change-Id: I5c30f2e39473a573f43409511ad9e2e32680479c
2020-06-09 17:04:37 -04:00
VitaliiShpital
a7211badc2 web/satellite: validate methods replaced into class Validator
Change-Id: Ib52b7209bab3fc1cb2f713421ef6bac2125c5f04
2020-06-09 13:05:24 +00:00
VitaliiShpital
cc0fca850e web/satellite: charges container on billing page reworked
WHAT:
charges container view was reworked to be more user-friendly

WHY:
billing UI/UX enhancements

Change-Id: Ice663544a1e69742a5c5fcc7123eeaa38bfbf255
2020-06-09 12:46:23 +00:00
VitaliiShpital
47a766328f web/satellite: deposit and billing history splitted to be shown separately
WHAT:
1. Deposit & Billing history view was divided to be shown separately as Deposit History and Billing History
2. Datepicker was removed from billing page

WHY:
billing UX enhancements

Change-Id: Ie183849ef0965169997674ce37b71db38a562fc2
2020-06-09 12:32:02 +00:00
Egon Elbre
1c30efd3a1 private/testplanet: allow setting "omit" as database to reduce output
Change-Id: I7af90fdefe2ff2df1340aa2b17f40806d889ca18
2020-06-09 12:41:58 +03:00
Michal Niewrzal
90e2e3c8c4 satellite/payments/stripecoinpayments: display more details about
invoice item generation

Part of https://storjlabs.atlassian.net/browse/USR-823

Change-Id: If0ffa56b68bb0a6b9b4e145fa21468006e7507f6
2020-06-08 23:33:38 +00:00
JT Olio
d9b1c4520e storagenode-updater: fix logic to use minimum version correctly
See:
  https://review.dev.storj.io/c/storj/storj/+/2052
  https://review.dev.storj.io/c/storj/private/+/2053

Change-Id: I30952c6e3893194ca876c12ecb7e53944e66aa2e
2020-06-08 17:43:10 +00:00
Michal Niewrzal
4dd47a4601 storj: add bump-dependencies target
One target to update all dependencies in main module and in testsuite

Change-Id: I2d8512f5fd7f5bb814451988a3feef366b39eee1
2020-06-08 15:51:37 +00:00
JT Olio
a974db7eec Update automatic updater blueprint to catch missing requirement
from https://github.com/storj/storj/pull/2677#pullrequestreview-270882629

this bit evidently never made it in.

See:
  https://review.dev.storj.io/c/storj/private/+/2053
  https://review.dev.storj.io/c/storj/storj/+/2054

Change-Id: Ie1435867cefa6d7e8b9f71bca13e24a161a619ae
2020-06-05 16:24:23 -06:00
Egon Elbre
36c461bd59 private/tagsql: track proper closing of rows and statements
This ensures that rows are closed to avoid leaks.
Also verifies that Err() is called, to ensure that no
error is left behind.

Change-Id: Idd1bec9bf479f40021da67b2c80ce83033149469
2020-06-05 18:25:43 +00:00
Yingrong Zhao
c78f88beb6 scripts/tests: use the same version for storj-sim as satellites
Change-Id: If85b14a0a5af334b9547d73bc4dbe65cdd856984
2020-06-05 13:17:17 -04:00
Jeff Wendling
943eb872d3 satellite/audit: depend less on details of some error message
the error message changed when we removed spacemonkeygo/errors

Change-Id: I4904a072cfd84e4c39c881b58669325bcf51df46
2020-06-05 10:39:05 -06:00
Egon Elbre
10f8b5492c Revert "private/tagsql: add finalizer based leak checks during dev"
This reverts commit c6310b34d2.

The change was causing data-races that are hard to deal with.

Change-Id: I0d29d85af70dce7ee2e967b9d7854719b32cf216
2020-06-05 17:52:46 +03:00
Yingrong Zhao
175f048aa7 scripts;cmd/storj-sim: include satellite node id in satellite address
Expose satellite ID from storj-sim so we can have access to it when
changing satellite address.

Change-Id: Ife816f4d35eae2d0bc9f9ad592fe75d73d93d9ff
2020-06-05 09:44:19 -04:00
Egon Elbre
34db4a80fd ci: fix staticcheck failures
Change-Id: I176fb24214755a1940a0a1a4e9cc8e39f184870b
2020-06-05 13:15:34 +00:00
Egon Elbre
0de1b5fa9f ci: fix golangci-lint failures
Change-Id: Ia8c4dd03439ce4e3f85890d45255b8ea1c94b50a
2020-06-05 12:55:35 +00:00
Yaroslav Vorobiov
09ca382abf storagenode/db: preflight improve index discovery
Change-Id: I876b321f6cd4e91dfced87aa4d39f2cf9a8e63d0
2020-06-05 14:03:25 +03:00
Yingrong Zhao
9a04ca0527 scripts/tests/rollingupgrade: skip v1.6.3
We removed the ability to change satellite address in for uplink cli v1.6.3. Therefore we will skip that version for now

Change-Id: I0f78cfe27b51fd6cad571636ba38266f5f672d58
2020-06-05 09:00:36 +00:00
paul cannon
5ed1985b22 docs/blueprints: trusted-delegated-repair.md
Trusted Delegated Repair is performing repair on trusted servers outside
the network boundary of the satellite.

Change-Id: I13b0e6f53100cefcb89c0c77bd1421c549551ab7
2020-06-04 18:22:25 +00:00
Qweder93
7f8e553022 console/dashboard: added pieces headers size to calculations
Change-Id: I0ee8d6bcb9ce9f69d49ebac2b95579166389668e
2020-06-04 16:39:02 +00:00
Michal Niewrzal
056ae7ffa8 scripts/test/rollingupgrade: set correct satellite address for imported
access

Addition: use always latest gateway release

Change-Id: I5e5231e6da4b6f7900cb71bb6e227901474270ea
2020-06-04 15:04:10 +00:00
Caleb Case
34ee3ef970
bump uplink to v1.0.7
Change-Id: Iab4ebded301b00bcf53c6dea012c35ce68eee8ba
2020-06-04 09:35:55 -04:00
Michal Niewrzal
2b2efcc662 satellite/payments/stripecoinpayments: move Coupons expiration date
sorting directly to listing method

Change-Id: I58d8a6ea1feba9ff2d19f21a1dbc87bfb8b49801
2020-06-04 09:47:42 +00:00
Michal Niewrzal
96286fde47 satellite/payments/stripecoinpayments: list projects by owner, not
by project members

This is a fix for listing the same project twice because project has
more than one member.

Change-Id: I3f6fe3456a6753d6d091a64436c22027dcbe2520
2020-06-04 08:42:01 +00:00
Yingrong Zhao
9d7713cdd0 script/testdata: update tracing agent default address
Change-Id: I730994f16f135c4b8643a52f4cf499487e4af326
2020-06-03 23:46:18 +00:00
Moby von Briesen
c8c0a42269 storagenode/orders: begin implementation of file store for order limits
* Will replace order limits database.
* This change adds functionality for storing and listing unsent orders.
* The next change will add functionality for order archival after
submission.

Change-Id: Ic5e2abc63991513245b6851a968ff2f2e18ce48d
2020-06-03 22:47:04 +00:00
NickolaiYurchenko
6b447a415b web/storagenode: free space calculation for chart fix
Change-Id: I4fb19b6ab8ba106dce4dfa49a03601c132ce66f0
2020-06-03 22:29:12 +00:00
Jeff Wendling
c6310b34d2 private/tagsql: add finalizer based leak checks during dev
what would win? thousands of man-hours spent trying to make
the best, most bug-free code possible, or one leaky boi?

this way we hopefully reduce the number of times we deadlock
everything by forgetting a single rows.Close.

Change-Id: I191727bbb75f74f5f4d0664e9e7b6ccf46c931f5
2020-06-03 15:06:58 -06:00
Jeff Wendling
254b42ff65 satellite/satellitedb: fix leaked rows from repairQueue.Insert
Change-Id: If5e62c49770f591ebe3f4d2dd4dd2658c229a022
2020-06-03 14:31:21 -06:00
Matt Robinson
eced8dcfec
build: Go 1.14.4 (#3904) 2020-06-03 17:35:35 +02:00
Michal Niewrzal
b20ced9519 satellite/satellitedb: drop project_id column from coupons table
This is last part of https://storjlabs.atlassian.net/browse/USR-818

Change-Id: I053d11b37df962c12e46645bae2fc2dad49c9755
2020-06-03 14:56:41 +00:00
Kaloyan Raev
fbfc3e5d18 satellite/payments: adjust label for bonus
This change adjusts the label for STORJ deposit bonuses in billing
history to be more consistent with other labels.

Change-Id: I5e7179ae3ac52dafb0dcef084e9a7c4742491f9e
2020-06-03 13:15:14 +00:00
Cameron Ayer
6a60e1e96b satellite/satellitedb: inclusive interval_start in GetAllocatedBandwidthTotal
The DB query in GetAllocatedBandwidthTotal uses an exclusive range:

'WHERE interval_start > ?'

The value that is used for this condition is the first day of current the month,
00:00:00 UTC.

By using the exclusive '>', we exclude the entire first hour of the month from the
result set.

Change-Id: I3ed300f5230c7514dc9495a85e8166213cd0842e
2020-06-02 13:06:45 -04:00
Bill Thorp
3aa3732b0d jenkins: Enable Cobertura failing for code coverage
Measure code coverage "by line" (Go actually measures by block, but whatever)
The '100, 0, 0' means anything above 100 auto-passes, anything below 0 auto-fails,
and anything below 0 is reported as unstable. autoupdateHealth, if I understand
correctly, will raise the functional auto-fail limit each time, so that a drop
in coverage will result in a failed test.  Finally failUnhealty is the trigger
which actually causes the build to fail, rather than simply reporting an error.

I have tested these changes using replay, but I was unable to verify 100% that
they work as designed.  The risks of having to revert this code seems less than
the cost of spending more time playing around with Jenkins locally..

Change-Id: Ica855b36aef7e1c3023fa80611229ca691017021
2020-06-02 16:00:08 +00:00
Egon Elbre
4d0cb1af7e storagenode/piecestore: verify before checking free disk
Change-Id: I3fe0383f9b13b99ef9d63ff235616ff204cf6d76
2020-06-02 17:49:14 +03:00
Egon Elbre
fca4f43a04 storage/filestore: benchmark diskInfoFromPath
Change-Id: I996057b1c650aec7cec84b49877d1e184a12514e
2020-06-02 17:49:14 +03:00
NickolaiYurchenko
18914e4d1a web/storagenode: months online calculations changed
Change-Id: I01d0ab5f9929a8cf2fca81adef43d9c979a18118
2020-06-02 17:09:14 +03:00
VitaliiShpital
c272872d5d satellite/payments: available coupon value feature
Change-Id: Ieae9385fbd917230298afff91a6be2838ad9b313
2020-06-02 10:00:45 +00:00