Commit Graph

4066 Commits

Author SHA1 Message Date
Michal Niewrzal
79a562c3ed cmd/uplink/cmd: improve uplink remove bucket command with force flag
* don't stop on single delete failure
* display number of successful and unsuccessful deletes

Change-Id: Id7f29055f250aad515e46622c2bfd3596055b93a
2020-06-01 07:56:14 +00:00
Jeff Wendling
839a6f15d1 stripecoinpayments: use fixed date for test
the test as written fails if it's the last day of the month because
the two tallies that are added span two different periods.

Change-Id: I294397278508035ce640500fc3b1b0ffd82b9b71
2020-06-01 09:02:52 +02:00
Michal Niewrzal
f6452e9c09 satellite/payments/stripecoinpayments: process customers instead of
projects

Currently, to generate invoices we are listing all projects. This is
problematic because we decided that elements like e.g. coupons should be
calculated per user. We want to have
more user oriented processing. With this change to prepare invoice
elements we are:
* listing Stripe customers
* listing projects for each customer
* processing projects to create project records
* calculating coupons per customer
* calculating credit spending per customer

Change-Id: I48b0063e44e4d301f97c87ecffb342649359968a
2020-05-30 13:16:07 +00:00
littleskunk
801a3ab90d
satellite/coinpayments: Reduce update interval to 2 minutes (#3897)
* satellite/coinpayments: Reduce update interval to 2 minutes

* satellite/coinpayments: Reduce balance update

Co-authored-by: paul cannon <thepaul@users.noreply.github.com>
2020-05-29 22:21:27 +02:00
VitaliiShpital
c9b9c686fc web/satellite: logic for new signup/login flow
WHAT:
1. updated verification page URL in config
2. added list of partnered satellites to config
3. added logic for satellites dropdown on new signup/login pages

WHY:
1. signup/login flow was reworked in tardigrade.io repo (iframe removed, new pages etc.)
2. new config flag was added to check if satellite name matches at least one member of partnered satellites list to redirect user to verification page
3. new pages will have dropdown with partnered satellites list. Appropriate logic was added.

Change-Id: I33399ab66ca31f07b297a433f6b1f41da4cb6e66
2020-05-29 17:11:44 +00:00
VitaliiShpital
0148f421c2 web/satellite: onboarding flow's vertical adaptation fixed
Change-Id: I8c3205b49a44907725a2dcd30951a36c8f3f746a
2020-05-29 16:55:58 +00:00
Ethan
b1bb665c78 satellite/metainfo: Handle "server is not accepting clients" error during CRDB node rejoins
https: //storjlabs.atlassian.net/browse/SM-1035
Change-Id: I27243b0d8fc3250916c86ceb915f973cbf80f656
2020-05-29 16:21:56 +00:00
Cameron
d5c86aa987
docs/blueprints: tracking downtime with audits (#3834)
* docs/blueprints: add blueprint for downtime DQ
2020-05-29 11:23:32 -04:00
Moby von Briesen
e7e69f383a satellite/repair/repairer/ec.go: Add monkit tracing for ec repairer
Adds monkit tracing for ecrepairer.downloadAndVerifyPiece and
ecrepairer.putPiece so we can get more accurate estimates of node
performance during repair.

Change-Id: Ic05025bf3c493bb3d6f5d325d090c5b7c9e5465d
2020-05-29 14:00:45 +00:00
Egon Elbre
d6c90b7ab5 pkg/macaroon: remove aliased package
Change-Id: I8b06e9a794d00752948f190623006e1afffdb031
2020-05-29 13:34:20 +00:00
Caleb Case
2cf4f4d1a1 bump uplink to v1.0.6
Change-Id: Idcabd742c2bb6394b27466a5244915a6675dbf15
2020-05-29 13:11:36 +00:00
Egon Elbre
ff7c726e99 ci: fix coverage flags
Change-Id: If9a87ac383eed3b8c7acf03938f44696af6c6f9c
2020-05-29 15:46:25 +03:00
Egon Elbre
d9fec04f50 lib: remove the old implementation
The stable uplink library is located at storj.io/uplink.
The stable uplink C library is located at storj.io/uplink-c.

Change-Id: Ia69115d6384b668490efe7e8ab674d7e8104b4f4
2020-05-29 11:36:43 +00:00
Egon Elbre
59ca5a1c52 scripts: remove lib/uplink from update-access
Change-Id: I70df3fa7ad14287e873e2f955def4231cc002aa6
2020-05-29 13:35:10 +03:00
Egon Elbre
07050eea26 all: use common/storj
Change-Id: Id1e36d52f9807b5ffbb72ce73f4b60cb21b68a78
2020-05-29 11:57:32 +03:00
Michal Niewrzal
48e67a29a9 cmd/storj-sim: remove old libuplink from storj-sim
Change-Id: I146a2d7947d1f358e7bdd8acba68eb3fee68539d
2020-05-29 08:39:37 +00:00
Moby von Briesen
acf8b72cd0 satellite/repair/repairer: cut off long tail when minimum number of required uploads is met
This will speed up the Put step of repair by not waiting to time out for
a handful of slow nodes, at the expense of a slightly less durable
pointer. It will still repair to the optimal threshold, but not every
node that is selected will end up in the pointer.

Change-Id: I02a0658e3fe6fc0383f26af0f50a065b8b11a651
2020-05-28 16:25:28 -04:00
Jeff Wendling
1e065fb450 satellite: migration to fix bad imported payment history
the initial calculations for the historical values of comp_at_rest
were wrong. because our historical data only included total amounts
as well as compensation for bandwidth, the at rest value was
calculated as

    at_rest = total - bandwidth

unfortunately, that calculation did not take surge pricing into
account correctly. the at rest and bandwidth values do not
include surge pricing, but the total that was used did. so what
we actually calculated was

    no_surge_at_rest = surge_total - no_surge_bandwidth

which will create a value that is too large. this migration
fixes the calculation for imports that are old enough and
of a non-negligable difference.

Change-Id: I61eb0b670510f6d7fb8fc3de39ba79150fac10eb
2020-05-28 12:59:08 -06:00
Moby von Briesen
dc57640d9c storagenode/piecestore: switch usedserials db for in-memory usedserials store
Part 2 of moving usedserials in memory
* Drop usedserials table in storagenodedb
* Use in-memory usedserials store in place of db for order limit
verification
* Update order limit grace period to be only one hour - this means
uplinks must send their order limits to storagenodes within an hour of
receiving them

Change-Id: I37a0e1d2ca6cb80854a3ef495af2d1d1f92e9f03
2020-05-28 12:52:52 -04:00
Moby von Briesen
909d6d9668 storagenode/piecestore/usedserials: add in-memory store for used serials
Implement an in-memory store for keeping track of order limit serial
numbers. It automatically deletes items if its size exceeds a configured
limit.
This change is part 1 - it creates the store
In part 2, the in-memory store will replace the usedserials database

Change-Id: I36f540ed809f034a27c1d7cede8a0a8b080af818
2020-05-28 12:52:52 -04:00
paul cannon
7395dd1e6e storagenode/gracefulexit: revalidate existing pieces
..before they are transferred to another node and submitted to the
satellite as successful piece transfers, because if we submit an invalid
signature, the node will be marked as a cheater and disqualified
immediately.

These signatures should have been validated when the piece was
originally stored, but bitrot does happen and needn't be cause for an
immediate DQ.

Change-Id: I8b0ebd5812ea8a2e60766005b7251fbb74ef7857
2020-05-28 09:50:14 -05:00
paul cannon
0c9a4a5e56 satellite/gracefulexit: better validation error messages
Change-Id: I14168dbeaf17302e5e853854956f8fbcb82a0900
2020-05-28 09:41:15 -05:00
Michal Niewrzal
84892631c8 private/testplanet: remove old libuplink from testplanet
Change-Id: Ib1553f84d0b3ae12a5b00382f0f53357b6a273e2
2020-05-28 13:50:23 +00:00
Qweder93
73214c6d1c storagenode/heldamount: heldhistory reworked to all satellites
Change-Id: I8d7707fddfbdc52d29951a8a002978c7fbb07049
2020-05-28 11:44:26 +00:00
Natalie Villasana
4ad163de2f storage/cockroachkv: rename opi to oci
opi was a carry-over from "ordered postgres iterator".

Change-Id: Id28cb8b8b3dccd119a1d1bbbb7f20206e932e4c5
2020-05-27 17:05:02 -04:00
Ethan
d5dfeca84d satellite/payments: Return empty iterator instead of nil from StripeMock List methods
Some satellites do not have payment configured (ex. Salt Lake, Europe North). In this case the StripeMock is used, which returns nil on invoice and charges List methods. This causes a panic.

https://storjlabs.atlassian.net/browse/SM-978

Change-Id: Iec1b0bfd9b383e6f793d03dd63a3dec60e0fd9f3
2020-05-27 16:20:18 +00:00
Yaroslav Vorobiov
e0d71dcaeb storagenode/windows-installer: ignore set firewall exception error
Change-Id: I176bfa4024a7395c0f68c67c5d804346a31aaeaf
2020-05-27 17:56:49 +03:00
Michal Niewrzal
2be93df566 scripts: reduce file size for backward compatibility tests
Change-Id: Ibcf460f908b6e551fe0ca68248affb7ddf49592f
2020-05-27 13:21:55 +00:00
Qweder93
f2a0c64425 storage/filestore: log potential disk corruption
In walkNamespaceWithPrefix log in case of "lstat" error, because this may indicate an underlying disk corruption.

SG-50

Change-Id: I867c3ffc47cfac325ae90658ec4780d213ff3e63
2020-05-27 12:12:55 +00:00
Qweder93
8db848791f storagenode/console: added estimated payout for current month and estimated pay stub for previous month (until there's real data in satellite's table) + heldback percentage rate for previous month.
Change-Id: I9346f6d22ed6fbb7e5346b102fc898467678f384
2020-05-27 14:51:23 +03:00
Kaloyan Raev
ee7de0424b satellite/payments: use quantities in invoices
Jira issue: https://storjlabs.atlassian.net/browse/USR-719

Invoices now show the amount of used resources and their unit price.

This change also makes proper rounding to the nearest cent in few places
to resolve the "off-by-one-cent" issue observed in few invoices.

Change-Id: I2d70d6916b5cf4a9a9138c99c422f5f4d2deb35b
2020-05-27 09:33:12 +00:00
Michal Niewrzal
75b3db5426 satellite/payments/stripecoinpayments: test invoice user with more than
1 project

https://storjlabs.atlassian.net/browse/USR-291

Change-Id: I98286e40254e8868de9eb675a9c9a8cd0bf5f3b1
2020-05-27 09:12:23 +00:00
Moby von Briesen
290c006a10 satellite/repair/{checker,queue}: add metric for new segments added to repair queue
* add monkit stat new_remote_segments_needing_repair, which reports the
number of new unhealthy segments in the repair queue since the previous
checker iteration

Change-Id: I2f10266006fdd6406ece50f4759b91382059dcc3
2020-05-27 06:23:47 +00:00
Stefan Benten
ed857f0fb4
satellite/admin: Add the proper documentation for bandwidth limits (#3900) 2020-05-27 09:22:45 +03:00
Egon Elbre
c5d4a13158 satellite/nodeselection: use NodeURL
Change-Id: I2ebd4dbf993ff5c7864f3a3a665b5c8fc48aa7d1
2020-05-27 05:46:11 +00:00
Cameron Ayer
9edb363558 satellite/metainfo: log if replacing pieces in pointer goes wrong
Every now and then we see the repair error, "piece to add already exists".
With these new logs we should be able to verify if it is due to a change in
the pointer. These logs are only temporary

Change-Id: I029390cc4816668707546df14ed2cfe7ca192b0b
2020-05-26 21:48:40 +00:00
NickolaiYurchenko
427bfc13fb web/storagenode: disk stat chrt added
Change-Id: I6f0c78992598ebbb3f1b3ee73c74e1559fc886b1
2020-05-27 00:14:33 +03:00
NickolaiYurchenko
b6771d0c52 web/storagenode: period selection separation
Change-Id: I9f50f4bb9282410f9f3cc4124e71a7a631767460
2020-05-26 23:23:19 +03:00
NickolaiYurchenko
d13b693b0c web/storagenode: total payout info area reworked
Change-Id: I2c5bf82105467db7c5919681f45d217b96e88792
2020-05-26 20:04:05 +00:00
Isaac Hess
5364a9f74d docs/blueprints: Add access-revocation.md
Change-Id: I74d9cc5564c727430b49d6b6f9040fd7df6f54e6
2020-05-26 18:59:27 +00:00
Natalie Villasana
8bd4d7b43e storage/cockroachkv: add check if retry is needed during iteration
This changeset replaces https://review.dev.storj.io/c/storj/storj/+/1839
which did the same thing but Nat couldn't figure out how to fix conflicting
files the correct gerrity way.

Change-Id: If05a8902aca986ea9f6c9168a90b31beebab839a
2020-05-26 14:32:06 -04:00
Michal Niewrzal
88a2561317 satellite/payments/stripecoinpayments: don't request anything from
Coinpayments if credentials are missing

Workaround for hitting Coinpayment API while tests.

Change-Id: I27114c820d4d7e17d05b066f55a13a709dac1711
2020-05-26 19:38:12 +02:00
Ethan
7014cf2083 satellite/gracefulexit: Change handleFailed to return nil if we can't get the pending transfer
https://storjlabs.atlassian.net/browse/SM-975

Change-Id: Ia1746b5e274e5011b6cd9f5a52f9a5faf703be51
2020-05-26 13:07:38 +00:00
Michal Niewrzal
a4c19b3359 cmd/uplink/cmd: remove port detection when selecting satellite while
setup

We don't have default ports as a part of configuration anymore because
satellite-addr flag was removed.

Change-Id: Ibf9fc4b399beaf51ebb9461de2d8994a322f9686
2020-05-26 13:25:51 +02:00
littleskunk
2fbb34c3ea
nodeselection: Increase minimum free space to 500MB (#3898) 2020-05-25 12:13:28 +02:00
Stefan Benten
3da100c242
cmd/satellite: Install curl per default to docker image (#3899) 2020-05-25 02:39:21 +02:00
Jeff Wendling
074649835b satellite/satellitedb: add some docs and improve some snapshots
This attempts to add a README.md to help create consistent migrations
that maximize our test coverage and do not include unnecessary
statements.

It also adds a feature to have an `-- OLD DATA --` section as well
as a `-- NEW DATA --` section so that we can fix mistakes made in
previous snapshots (like a row that was forgotten to be added when a
table was created) without editing them going forward.

Change-Id: I28a786f8ef163cae1de1bb08f61af1e1104b0a88
2020-05-22 21:27:36 +00:00
Michal Niewrzal
5c10964040 satellite/payments/stripecoinpayments: add test for listing issues while
invoice generation

https://review.dev.storj.io/c/storj/storj/+/1853
https://review.dev.storj.io/c/storj/storj/+/1882

Change-Id: Ie71363b819866dd60dbe7117b42cfa8348479310
2020-05-22 17:24:16 +00:00
Michal Niewrzal
340700eda9 cmd/uplink/cmd: remove old libuplink from Uplink CLI code
Change-Id: I38444a67e3bdec5f63c56895b5fc362464e7dce1
2020-05-22 09:08:58 +00:00
Bill Thorp
bcb867f1de cmd/uplink: add UserAgent flag for partner value attribution
This changeset allows a user agent string to be set during uplink setup, which
is thereafter used for partner value attribution.  EG

uplink setup --client.user-agent ”MyCompany”

Change-Id: Iefa8755fccc06acb8a303a342b943cece44a81f7
2020-05-22 07:39:14 +00:00