Commit Graph

4281 Commits

Author SHA1 Message Date
Egon Elbre
44f9193404 satellite/orders: make optimal threshold multiplier into an argument
It feels weird having a repairer configuration part of order services.
Let's have a single source of truth for it.

Change-Id: I24f7c897aec80f3293f8af24876cbb6733d85a0b
2020-07-24 16:35:59 +03:00
Egon Elbre
ba4c3d9986 satellite/orders: remove unused node status logging flag
Change-Id: I24da78a11cc5d3d88cdf6aca85c4238e4086e59c
2020-07-24 16:35:59 +03:00
Yaroslav Vorobiov
4d2a505788 storagenode/db: explicitly open and create dbs
To prevent storagenode from implicitly recreating missing dbs and storage,
as such behaviour leads to audit failures. Do not allow storagenode to
start if any of dbs or storage is missing, corrupted, or dedicated storage disk is
unmounted, to get downtime instead.

Change-Id: Ic64e1f0ff4d8ef5b2fddbe7a7e53df4f4bd8652e
2020-07-24 14:08:47 +03:00
NickolaiYurchenko
e982173d61 web/storagenode: payout page bug fix
Change-Id: I6d1eb16ecc18f069a5c3b0e4e5cfec9db3dfb9ea
2020-07-24 13:24:18 +03:00
Qweder93
92efffb48a storagenode/version: notification flow now based on cursor, chore_test added, versioncontrol added to reconfigure.
Change-Id: I70713def8d585228270ec5a8c586ecc5b4d510c4
2020-07-23 14:13:24 +00:00
Cameron Ayer
46bdb1dbfe Revert "satellite/metainfo: log if replacing pieces in pointer goes wrong"
This reverts commit 9edb363558.

Change-Id: Ia6219f4cca4e766c7919a6d830dfa6900fd4f6e0
2020-07-23 13:42:50 +00:00
Cameron Ayer
e14f7a3fb4 satellite/repair: update healthyPieces and unhealthyPieces after CreateGetRepairOrderLimits
Inside CreateGetRepairOrderLimits we pass in a list of healthy pieces,
but when we query node info from this list we apply the "reliable" filter
again. We sometimes end up with nodes which at first were healthy, but then
became unhealthy, and thus can be repaired, but we do not update the 'unhealthyPieces'
list with these nodes.

This causes an error, 'piece to add already exists', as we fail to remove these
pieces from the pointer before replacing them with repaired pieces.

Change-Id: I6e2445f342ac117ded30351fa7e5e523c9ec26bd
2020-07-23 13:24:46 +00:00
Egon Elbre
1f9ee4a536 mod: bump common
Update testrand.Metadata to prevent flaky test.

Change-Id: I8085f694403e738e3c6cbd8370945cf2c7dc17d3
2020-07-23 12:51:16 +00:00
Kaloyan Raev
4bcf308a04 satellite/payments: fetch old deposit bonuses from Stripe metadata
Jira: https://storjlabs.atlassian.net/browse/USR-822

The balance history in Satellite GUI display the deposit bonuses as
separate rows. These bonuses used to be stored in the satellite DB. We
recently started depositing the bonus directly to the Stripe balance and
migrated old bonuses to Stripe metadata.

This change displays all billing history entirely from Stripe, so we can
remove the `credits` and `credits_spendings` DB tables in a next step.

Change-Id: I14c304c66ec47c6a51f5b8508f11470cf36c4e24
2020-07-23 12:11:17 +00:00
Egon Elbre
20437b43a1 satellite/payments/stripecoinpayments: prevent data race
There's still a possibility of tests clashing due to the shared mock,
however it's slightly better, because it avoids the race.

Change-Id: I80eedf1ca50b6114ebe69ea3c4d61176452f4df0
2020-07-23 13:41:48 +03:00
Ethan
cfca021839 satellite/accounting: Add chore to cleanup old project bandwidth rollups data
Removes old project_bandwidth_rollups records that are no longer used.

Uses a retain months configuration to determine how many months to save.  Current month cannot be removed.
Tests retainMonths=-1, 0, 2

Change-Id: Ia4be2546cdb28802427acf41ecd85ad66df3e62c
2020-07-22 18:56:49 +00:00
Cameron Ayer
1f5d5235a6 storagenode/{monitor,piecestore}: if free disk < expected available space, return free disk
We only sync free disk and available space, if necessary, on startup. If the
SNs disk fills up with non-storj data, we will not know about it when reporting
available space to the satellite.

Solution: whenever we check the node's capacity, double check free disk.
If free disk < than expected available space, return free disk.

Change-Id: I66265c16e03be45b6e1f5817c70df7eac0a76455
2020-07-22 15:08:37 +00:00
Michal Niewrzal
18b3495248 satellite/metainfo: restore tests disabled to remove StreamID from
BeginDeleteObject method on uplink side

Change-Id: Id10cdeb367759f9949dc557b291726fd331d38ba
2020-07-22 13:54:39 +00:00
Qweder93
aa6afc3879 error handling in heldamount cash and collector delete fixed
Change-Id: I8fe58c50f844a6b819eacc14a40bc5c67268ed5c
2020-07-22 12:26:13 +00:00
Kaloyan Raev
96c83eb0b8 satellite/payments: log returned error
This makes debugging of issues easier as the error in the web browser
does not show all the info.

Change-Id: Ia79902b7275103a78a21d43ba14243bc32bb8be5
2020-07-22 12:00:27 +00:00
Kaloyan Raev
a20e85824a satellite/payments: add STORJ amount and rate to Stripe TX metadata
Jira: https://storjlabs.atlassian.net/browse/USR-968

We want to keep track of the STORJ amount and exchange rate in the
metadata of Stripe Customer Balance Transaction to be able to generate
reports without the need of requesting CoinPayments for this info.

Change-Id: Ia93af95706cd2312cf688f044874495279fe8fa2
2020-07-22 11:57:21 +03:00
Qweder93
0949731caa storagenode/console: estimation payout held split from total payout, calculations fixed
Change-Id: I064f473ffeb3a3051c9228d1dd84fe0fc86dd3ef
2020-07-21 15:31:51 +03:00
Egon Elbre
ce7adc8d85 satellite/console: remove need for nolint
Change-Id: I1b7ab00720f630c4ad3f984adc5698b41368613b
2020-07-21 09:13:57 +00:00
Bill Thorp
65408db6e0 satellite/satellitedb: Coinpayments repeat insert bug fix
I introduced a bug with https://review.dev.storj.io/c/storj/storj/+/2216
Because the log change allowed insert to be called multiple times.
This changes the insert logic to do nothing if the PK already exists.

Change-Id: I90d192a0f6619bfbb360ea104066f00a3348f6dd
2020-07-20 20:21:35 +00:00
Rafael Gomes
375d76638d satellite/metainfo/objectdeletion: Object deletion implementation
Improve our delete logic to require fewer database requests.
This PR creates a new objectdeletion package

Change-Id: I0500173bb9b8c771accb350f076329ede6dbb42c
2020-07-20 16:09:48 +00:00
Egon Elbre
85aff88e7f ci: update to Go 1.14.6
Change-Id: I0926c543612080ecd671d7e3cb67f940258b8ecc
2020-07-20 13:17:49 +00:00
Michal Niewrzal
e69cec0b75 satellite/metainfo: stop generating StreamID for BeginDeleteObject
request

We are no longer using `BeginDeleteSegment` or `ListSegments` so we can
avoid generating StreamID as a result of `BeginDeleteObject`.

StreamID from `BeginDeleteObject` is also not used on Uplink side.

Change-Id: I3b068deab17068459849b5cf05811cad4b8a9034
2020-07-17 17:51:06 +00:00
Isaac Hess
67a292d135 satellite/satellitedb: Monitor node tallies
We are adding a monkit evaluation for the total sum of data stored on
the nodes before it is inserted into the database. This will give us a
time-series history of total data stored so we can see it change over
time.

Change-Id: I41145a2d7a09c8e63b42ae578bd081035b60e529
2020-07-17 10:21:42 -06:00
VitaliiShpital
4684eb5b75 web/satellite: completed transaction state condition changed
WHAT:
completed transaction state condition depends only on stripe balance now

WHY:
avoid waiting for transaction status to update

Change-Id: I09be620f94934209b60ef692ffcc0f5097880920
2020-07-17 17:29:14 +03:00
NickolaiYurchenko
f1c73c127f web/storagenode: suspension tooltip content change
Change-Id: I0d66f9d8c95b20d113c9880602a04ae0176c3b90
2020-07-17 11:21:59 +00:00
paul cannon
fd7bfc94fe private/dbutil: don't sort column names in an index
The order in which column names appear in an index should be
deterministic (for both our sqlite and postgresql code). Also, the order
is very relevant as to whether a given schema is correct.

Change-Id: I227ea057fcd9c3e967dd241a7e1c787d1bc4baa1
2020-07-17 10:07:01 +00:00
Egon Elbre
b67d7ecbc5 cmd/storagenode,storage/cockroachkv: better error handling
Change-Id: I6646aa046dc365c0dee38f23041be4fc2defb759
2020-07-16 20:03:50 +03:00
Egon Elbre
b719eecb28 satellite/admin: close http response body
Change-Id: I702e3a7e53d537d8be30b1aaabad4c4e70efe706
2020-07-16 16:38:10 +00:00
Egon Elbre
d8dcae3075 all: fix error checking
Change-Id: Ia0da1bbd6ce695139922f94096c2419281905e32
2020-07-16 19:13:14 +03:00
Egon Elbre
b84923558b satellite: fix scoping, formatting
Change-Id: I21ef9edc2d449d75ad74891df7f966fb150d80fd
2020-07-16 19:13:14 +03:00
Egon Elbre
e70da5cd4e all: fix comments
Change-Id: I2d2307e3fab87de47a72b3595d051e2c95ff4f8a
2020-07-16 19:13:14 +03:00
stefanbenten
1a6e25579c satellite/admin: check for existing user with given email
To prevent creating multiple users with the same email via API, we should check for an existing user with given email.

Change-Id: Ie35b85c4f94a7ca72d42951dab8ff475d7f0dd7c
2020-07-16 15:17:00 +00:00
stefanbenten
0534b91085 satellite/admin: setup payment account for user
Currently a customer created via the IP does not get an payment account until he signs in.
That causes issues if the account should be deleted again.

Change-Id: I393c8f301e426301bb713c423d6ce011138d4ae4
2020-07-16 15:16:50 +00:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
VitaliiShpital
0a800336cb web/satellite: token deposit dropdown position fixed
WHAT:
dropdown position fixed before first ever transaction

WHY:
bug fix

Change-Id: I049acc1e1bc8311e165efc92c9769c989dbb8e41
2020-07-16 14:36:03 +00:00
NickolaiYurchenko
b33ef461e8 web/storagenode: available payout periods
Change-Id: I644e62ceb7b65cf3bc6f01ae92cf91bbd7ecc797
2020-07-16 17:09:37 +03:00
Qweder93
dfdf73282d storagenode/heldamount: db tests updated with payout.Receipt
Change-Id: I17699b923c5a4d7decbd446c382f0c886c36d5e1
2020-07-16 12:24:22 +00:00
stefanbenten
9ace375ee0 satellite/{console,satellitedb}: change project limiting based on new users field
This change switches the backend logic to use the new DB column on the users table to restrict project creation.
Furthermore it back fills the existing limits from registration tokens to the new column to ensure no users are reset to the new default.

UI is updated to reflect ability to create several projects

Change-Id: Ie29157430ae6b065411ca4c4557c9f1be69cdc4f
2020-07-16 10:57:47 +00:00
Jeff Wendling
1944d734ef satellite/orders: check and enforce node api version
Change-Id: Ibdeb1a85dfed8b534bfed32a7cdaae5c3dc8b420
2020-07-16 10:38:12 +00:00
Moby von Briesen
1b807761bd storagenode/orders: Update orders filestore to be compatible with new satellite endpoint
* Instead of archiving a list of orders and deleting an "unsent" file in
separate steps, archival simply moves the old unsent file to a new
archived file
* Add maxInFlightTime to be used along with grace period for sending
buffer
* Create unsent/archival directories in constructor
* Code cleanup

Change-Id: Ia3bc2aaf60cced6c6d413465423d78c7d5151188
2020-07-15 14:21:56 -04:00
stefanbenten
0209a2095f satellite/{console,satellitedb}: add project_limit column to users table
Change-Id: I603f085f17ca5b413dd1c6837c2081f9e7e791a1
2020-07-15 17:27:31 +00:00
stefanbenten
2c2d284f3d satellite/admin: add bucket limit handling endpoint
Change-Id: I4b199277cff30f11f4a9fff3b0ac4017b694f2e8
2020-07-15 17:27:23 +00:00
Jennifer Johnson
784a156eea satellite: prevents uplink from creating a bucket once it exceeds the max bucket allocation.
Change-Id: I4b3822ed723c03dbbc0df136b2201027e19ba0cd
2020-07-15 17:27:05 +00:00
Qweder93
62fec25104 storagenode/heldamount: returns usage_at_rest in tbm instead of tbh
Change-Id: I183a56460ea76a53680ca6861d02cecebe3576ec
2020-07-15 15:46:13 +03:00
Jeff Wendling
3a8766936b satellite/orders: remove race condition in new endpoint tests
the flush batch size was set to 1 which means that a flush was
async scheduled after the first write. the explicit trigger wait
was then always flushing nothing, and the test would only
pass if the async flush was scheduled before the read.

remove that async flush and pause the flush loop so that we are
in full control of when the flushes happen so there are no races.

the tests are still disabled but that's because the endpoint is
still disabled.

Change-Id: I2b7b07fd5525388c30be8efbf4af7105087228da
2020-07-14 15:12:33 -04:00
Caleb Case
581579e80a
satellite/rewards: Add Duplicati to partners database.
Change-Id: I1d702504adc21cc7a243fe67f1856bd13928c369
2020-07-14 12:49:48 -04:00
stefanbenten
257855b5de all: replace == comparison with errors.Is
Change-Id: I05d9a369c7c6f144b94a4c524e8aea18eb9cb714
2020-07-14 15:50:25 +00:00
Qweder93
7b4a8c4d6d storagenode/heldamount: payoutHistory added
Change-Id: I93dd3d024085d19ecff76075e52bf66796207fd6
2020-07-14 17:35:03 +03:00
Jessica Grebenschikov
1f1e3f0604 satellite/orders: disable settlementwithwindow, skip flaky tests
Change-Id: Ia60d7e0f2d383919650cdc736ba4569bb26ff2d7
2020-07-14 07:17:18 -07:00
Qweder93
7d6973b5a2 satellite: heldamount and nodestats not returning error node not found by rpc
Change-Id: Ifb00b16a4a04603251de60da6a6612fd5e98d597
2020-07-14 16:31:02 +03:00