Commit Graph

5138 Commits

Author SHA1 Message Date
Qweder93
f2a0c64425 storage/filestore: log potential disk corruption
In walkNamespaceWithPrefix log in case of "lstat" error, because this may indicate an underlying disk corruption.

SG-50

Change-Id: I867c3ffc47cfac325ae90658ec4780d213ff3e63
2020-05-27 12:12:55 +00:00
Qweder93
8db848791f storagenode/console: added estimated payout for current month and estimated pay stub for previous month (until there's real data in satellite's table) + heldback percentage rate for previous month.
Change-Id: I9346f6d22ed6fbb7e5346b102fc898467678f384
2020-05-27 14:51:23 +03:00
Kaloyan Raev
ee7de0424b satellite/payments: use quantities in invoices
Jira issue: https://storjlabs.atlassian.net/browse/USR-719

Invoices now show the amount of used resources and their unit price.

This change also makes proper rounding to the nearest cent in few places
to resolve the "off-by-one-cent" issue observed in few invoices.

Change-Id: I2d70d6916b5cf4a9a9138c99c422f5f4d2deb35b
2020-05-27 09:33:12 +00:00
Michal Niewrzal
75b3db5426 satellite/payments/stripecoinpayments: test invoice user with more than
1 project

https://storjlabs.atlassian.net/browse/USR-291

Change-Id: I98286e40254e8868de9eb675a9c9a8cd0bf5f3b1
2020-05-27 09:12:23 +00:00
Moby von Briesen
290c006a10 satellite/repair/{checker,queue}: add metric for new segments added to repair queue
* add monkit stat new_remote_segments_needing_repair, which reports the
number of new unhealthy segments in the repair queue since the previous
checker iteration

Change-Id: I2f10266006fdd6406ece50f4759b91382059dcc3
2020-05-27 06:23:47 +00:00
Stefan Benten
ed857f0fb4
satellite/admin: Add the proper documentation for bandwidth limits (#3900) 2020-05-27 09:22:45 +03:00
Egon Elbre
c5d4a13158 satellite/nodeselection: use NodeURL
Change-Id: I2ebd4dbf993ff5c7864f3a3a665b5c8fc48aa7d1
2020-05-27 05:46:11 +00:00
Cameron Ayer
9edb363558 satellite/metainfo: log if replacing pieces in pointer goes wrong
Every now and then we see the repair error, "piece to add already exists".
With these new logs we should be able to verify if it is due to a change in
the pointer. These logs are only temporary

Change-Id: I029390cc4816668707546df14ed2cfe7ca192b0b
2020-05-26 21:48:40 +00:00
NickolaiYurchenko
427bfc13fb web/storagenode: disk stat chrt added
Change-Id: I6f0c78992598ebbb3f1b3ee73c74e1559fc886b1
2020-05-27 00:14:33 +03:00
NickolaiYurchenko
b6771d0c52 web/storagenode: period selection separation
Change-Id: I9f50f4bb9282410f9f3cc4124e71a7a631767460
2020-05-26 23:23:19 +03:00
NickolaiYurchenko
d13b693b0c web/storagenode: total payout info area reworked
Change-Id: I2c5bf82105467db7c5919681f45d217b96e88792
2020-05-26 20:04:05 +00:00
Isaac Hess
5364a9f74d docs/blueprints: Add access-revocation.md
Change-Id: I74d9cc5564c727430b49d6b6f9040fd7df6f54e6
2020-05-26 18:59:27 +00:00
Natalie Villasana
8bd4d7b43e storage/cockroachkv: add check if retry is needed during iteration
This changeset replaces https://review.dev.storj.io/c/storj/storj/+/1839
which did the same thing but Nat couldn't figure out how to fix conflicting
files the correct gerrity way.

Change-Id: If05a8902aca986ea9f6c9168a90b31beebab839a
2020-05-26 14:32:06 -04:00
Michal Niewrzal
88a2561317 satellite/payments/stripecoinpayments: don't request anything from
Coinpayments if credentials are missing

Workaround for hitting Coinpayment API while tests.

Change-Id: I27114c820d4d7e17d05b066f55a13a709dac1711
2020-05-26 19:38:12 +02:00
Ethan
7014cf2083 satellite/gracefulexit: Change handleFailed to return nil if we can't get the pending transfer
https://storjlabs.atlassian.net/browse/SM-975

Change-Id: Ia1746b5e274e5011b6cd9f5a52f9a5faf703be51
2020-05-26 13:07:38 +00:00
Michal Niewrzal
a4c19b3359 cmd/uplink/cmd: remove port detection when selecting satellite while
setup

We don't have default ports as a part of configuration anymore because
satellite-addr flag was removed.

Change-Id: Ibf9fc4b399beaf51ebb9461de2d8994a322f9686
2020-05-26 13:25:51 +02:00
littleskunk
2fbb34c3ea
nodeselection: Increase minimum free space to 500MB (#3898) 2020-05-25 12:13:28 +02:00
Stefan Benten
3da100c242
cmd/satellite: Install curl per default to docker image (#3899) 2020-05-25 02:39:21 +02:00
Jeff Wendling
074649835b satellite/satellitedb: add some docs and improve some snapshots
This attempts to add a README.md to help create consistent migrations
that maximize our test coverage and do not include unnecessary
statements.

It also adds a feature to have an `-- OLD DATA --` section as well
as a `-- NEW DATA --` section so that we can fix mistakes made in
previous snapshots (like a row that was forgotten to be added when a
table was created) without editing them going forward.

Change-Id: I28a786f8ef163cae1de1bb08f61af1e1104b0a88
2020-05-22 21:27:36 +00:00
Michal Niewrzal
5c10964040 satellite/payments/stripecoinpayments: add test for listing issues while
invoice generation

https://review.dev.storj.io/c/storj/storj/+/1853
https://review.dev.storj.io/c/storj/storj/+/1882

Change-Id: Ie71363b819866dd60dbe7117b42cfa8348479310
2020-05-22 17:24:16 +00:00
Michal Niewrzal
340700eda9 cmd/uplink/cmd: remove old libuplink from Uplink CLI code
Change-Id: I38444a67e3bdec5f63c56895b5fc362464e7dce1
2020-05-22 09:08:58 +00:00
Bill Thorp
bcb867f1de cmd/uplink: add UserAgent flag for partner value attribution
This changeset allows a user agent string to be set during uplink setup, which
is thereafter used for partner value attribution.  EG

uplink setup --client.user-agent ”MyCompany”

Change-Id: Iefa8755fccc06acb8a303a342b943cece44a81f7
2020-05-22 07:39:14 +00:00
Michal Niewrzal
3d332de228 private/testplanet: remove StorageNodeCount from testplanet uplink
definion

Small cleanup.

Change-Id: Icabdf1433c36cd0e9f8e10a67975e98391024e14
2020-05-21 14:51:58 +00:00
crawter
f5ac678b0a storagenode/satellitesdb: added FK constraint to satelliteID
Change-Id: If5adf2b92627fcf80850670ba672b346320ddd87
2020-05-21 13:01:20 +00:00
VitaliiShpital
45ccf59067 web/satellite: onboarding tour adapted to multiple project state
Change-Id: I853772bf2a05c3fe9ef8dea0d1e290e6074fbfec
2020-05-21 12:19:33 +00:00
VitaliiShpital
f0619c6db7 web/satellite: error handling for login/signup rate limit exceeding
related jira issue:
https://storjlabs.atlassian.net/browse/USR-767

Change-Id: I4b4f27435323234ce6803c06a8c5b004f91873bf
2020-05-21 12:03:27 +00:00
VitaliiShpital
7d00b3c792 web/satellite: billing history item's expiration countdown fixed
related jira issue:
https://storjlabs.atlassian.net/browse/USR-774

Change-Id: I4629b3ea24ba4148f2faf4030d46fb5adad192fe
2020-05-21 11:47:38 +00:00
Michal Niewrzal
83cc80ef06 satellite/payments/stripecoinpayments: fix listing issue for coupon
usages and credit spendings

Test will be added as a separate change.

Change-Id: I9341d2ecb736fdbd45beb96f778f04448df68bd7
2020-05-21 07:48:04 +00:00
Egon Elbre
bef84a5f9d storagenode: remove dependency to overlay.NodeDossier
This is the last dependency from storage node to satellite.

Change-Id: I12f7abb91e84f823ba5af126c6e2979519838612
2020-05-21 08:37:13 +03:00
Egon Elbre
b42778c42e private/testplanet: remove some additional Local-s
Change-Id: I49701c41efb92efca27cc18d0a3f6d6b44d3cf8b
2020-05-21 08:37:13 +03:00
Egon Elbre
7f323754a4 metainfo/piecedeletion: use NodeURL-s
Change-Id: I247dbfe03e7864e940e4cd1d0f343f38e84099e0
2020-05-21 08:37:13 +03:00
Jennifer Johnson
03e5f922c3 satellite/overlay: updates node with a vetted_at timestamp if they meet the vetting criteria
What: As soon as a node passes the vetting criteria (total_audit_count and total_uptime_count
are greater than the configured thresholds), we set vetted_at to the current timestamp.

Why: We may want to use this timestamp in future development to select new vs vetted nodes.
It also allows flexibility in node vetting experiments and allows for better metrics around
vetting times.

Please describe the tests: satellitedb_test: TestUpdateStats and TestBatchUpdateStats make sure vetted_at is set appropriately
Please describe the performance impact: This change does add extra logic to BatchUpdateStats and UpdateStats and
commits another variable to the db (vetted_at), but this should be negligible.

Change-Id: I3de804549b5f1bc359da4935bc859758ceac261d
2020-05-20 16:30:26 -04:00
Natalie Villasana
2514d6328d dbutil/cockroachutil: add monkit to QueryContext
This will help us keep track of crdb errors in influx.

Change-Id: I997596aa4eb9a2b9b81305d123c3452ecdf5dde5
2020-05-20 14:56:25 -04:00
Bill Thorp
f43cb1688d private/tagsql: verify SQL connection with ping
Use ping to make sure the database connection is valid

Change-Id: I5217e28e186f487266c8f4a1d39cce0070dc1465
2020-05-20 13:12:16 +00:00
Michal Niewrzal
fe6a6f063f private/testplanet: cleanup predefined data generation
Use Console service to create user and project instead direct DB
modifications.

Change-Id: Ib0074b38313b3dc43b7d8d63ab2775d29028fb7b
2020-05-20 12:38:43 +00:00
crawter
2c9afe7f17 storagenode/console/api/helamount: periods with heldamount data endpoint added
Change-Id: Ie893f56f02c7a76bcfc21c32c10bd1f1d05660e7
2020-05-20 11:45:06 +00:00
Qweder93
49ad90dcd8 storagenode/reputation: unknown_score (unknown_alpha / unknown_alpha+unknown_beta) added to reputation stats, https://storjlabs.atlassian.net/browse/SG-326
Change-Id: I0b29ad736f7a11c7e57a846b6891f4b40755aa48
2020-05-20 11:25:14 +00:00
Egon Elbre
5d016425f1 satellite/{contact,downtime,overlay}: use NodeURL
Change-Id: I555a479a89e0ddbf0499898bdbc8574282cd6846
2020-05-20 11:09:05 +00:00
Egon Elbre
941d10cbc3 private/testplanet: remove Peer.Local()
Currently storagenode depends on overlay.NodeDossier, this is the first
step in removing it.

Change-Id: I034a3f1601835f8349bd41752455022e19bcc707
2020-05-20 11:05:34 +00:00
Egon Elbre
94b2b315f7 storagenode/trust: refactor GetAddress to GetNodeURL
Most places now need the NodeURL rather than the ID and Address
separately. This simplifies code in multiple places.

Change-Id: I52621d8ca52296a8b5bf7afbc1001cf8bfb44239
2020-05-20 11:05:15 +00:00
Egon Elbre
ed627144ed all: use DialNodeURL throughout the codebase
Change-Id: Iaf9ae3aeef7305c937f2660c929744db2d88776c
2020-05-20 10:36:30 +00:00
Ivan Fraixedes
963db8c738 cmd/segment-reaper: Remove unneeded test cases
Remove unneeded observer's tests cases after the changes that enhanced
the segment-reaper not to be limited to objects up to 65 segments.

Also remove some tests cases that were already covered by the
TestObserver_processSegment_from_to test function.

Change-Id: I2385b406b50f7810ed22d3297b903a993f83fcfe
2020-05-20 10:10:51 +00:00
Stefan Benten
1a4ab3df48
satellite/admin: add user update endpoint (#3896) 2020-05-20 12:20:04 +03:00
littleskunk
8ec64f3daf
satellite/overlay: enable node selection cache on all satellites (#3895) 2020-05-19 19:25:53 +02:00
Natalie Villasana
c7cbdcbccf cmd/internal: request lowercase access name
Viper has a feature/bug where all YAML config keys are cast
to lowercase (see https://github.com/spf13/viper/issues/260
and https://github.com/spf13/viper/issues/411).
Since we use Viper to load our config values, it doesn't seem
like there's an easy way to preserve case sensitivity for the
access names chosen by users right now. Adding this prompt
should help user experience by clarifying that all access
names must be lowercase.

Change-Id: I47e8344bb0ca7e78458405496f20e78e3c9f9a88
2020-05-19 16:04:35 +00:00
Jess G
6fac1ed130
satellite/contact: rm unhelpful verbose logging (#3894)
* unhelpful rm verbose logging

Change-Id: I74dc29f18969250fa6838fed9bec8ba7ea160b77

* change to debug

Change-Id: I4ba880e68b40bc3e0f29d9b1aa9154e63124c904
2020-05-19 08:09:00 -07:00
Michal Niewrzal
705e82ea99 private/testplanet: add AddUser and AddProject to satellite
functionality

We want to start adding more complex test cases for billing/invoices and
we need more handy tooling to be able do this easily.

Change-Id: Ib22ac6b4ba9ee77cc91c88b0cfd2d2efc15657df
2020-05-19 13:02:04 +00:00
Stefan Benten
0a26c4af9a
satellite/admin: add coupon deletion (#3893) 2020-05-19 15:49:44 +03:00
Stefan Benten
671aca56b0
satellite/admin: add coupon creation and listing (#3891) 2020-05-19 12:36:13 +02:00
Ivan Fraixedes
cbdd3bfd7b cmd/segment-reaper: Rename test name to what it tests
Rename a test function to indicate the function that it actually tests
rather than indicating the function that calls the function under test.

Change-Id: I2c1fe165e3acf952993a9a65dfbfbdcda0d1cccc
2020-05-19 10:06:56 +00:00