Commit Graph

4607 Commits

Author SHA1 Message Date
Egon Elbre
d6c90b7ab5 pkg/macaroon: remove aliased package
Change-Id: I8b06e9a794d00752948f190623006e1afffdb031
2020-05-29 13:34:20 +00:00
Caleb Case
2cf4f4d1a1 bump uplink to v1.0.6
Change-Id: Idcabd742c2bb6394b27466a5244915a6675dbf15
2020-05-29 13:11:36 +00:00
Egon Elbre
ff7c726e99 ci: fix coverage flags
Change-Id: If9a87ac383eed3b8c7acf03938f44696af6c6f9c
2020-05-29 15:46:25 +03:00
Egon Elbre
d9fec04f50 lib: remove the old implementation
The stable uplink library is located at storj.io/uplink.
The stable uplink C library is located at storj.io/uplink-c.

Change-Id: Ia69115d6384b668490efe7e8ab674d7e8104b4f4
2020-05-29 11:36:43 +00:00
Egon Elbre
59ca5a1c52 scripts: remove lib/uplink from update-access
Change-Id: I70df3fa7ad14287e873e2f955def4231cc002aa6
2020-05-29 13:35:10 +03:00
Egon Elbre
07050eea26 all: use common/storj
Change-Id: Id1e36d52f9807b5ffbb72ce73f4b60cb21b68a78
2020-05-29 11:57:32 +03:00
Michal Niewrzal
48e67a29a9 cmd/storj-sim: remove old libuplink from storj-sim
Change-Id: I146a2d7947d1f358e7bdd8acba68eb3fee68539d
2020-05-29 08:39:37 +00:00
Moby von Briesen
acf8b72cd0 satellite/repair/repairer: cut off long tail when minimum number of required uploads is met
This will speed up the Put step of repair by not waiting to time out for
a handful of slow nodes, at the expense of a slightly less durable
pointer. It will still repair to the optimal threshold, but not every
node that is selected will end up in the pointer.

Change-Id: I02a0658e3fe6fc0383f26af0f50a065b8b11a651
2020-05-28 16:25:28 -04:00
Jeff Wendling
1e065fb450 satellite: migration to fix bad imported payment history
the initial calculations for the historical values of comp_at_rest
were wrong. because our historical data only included total amounts
as well as compensation for bandwidth, the at rest value was
calculated as

    at_rest = total - bandwidth

unfortunately, that calculation did not take surge pricing into
account correctly. the at rest and bandwidth values do not
include surge pricing, but the total that was used did. so what
we actually calculated was

    no_surge_at_rest = surge_total - no_surge_bandwidth

which will create a value that is too large. this migration
fixes the calculation for imports that are old enough and
of a non-negligable difference.

Change-Id: I61eb0b670510f6d7fb8fc3de39ba79150fac10eb
2020-05-28 12:59:08 -06:00
Moby von Briesen
dc57640d9c storagenode/piecestore: switch usedserials db for in-memory usedserials store
Part 2 of moving usedserials in memory
* Drop usedserials table in storagenodedb
* Use in-memory usedserials store in place of db for order limit
verification
* Update order limit grace period to be only one hour - this means
uplinks must send their order limits to storagenodes within an hour of
receiving them

Change-Id: I37a0e1d2ca6cb80854a3ef495af2d1d1f92e9f03
2020-05-28 12:52:52 -04:00
Moby von Briesen
909d6d9668 storagenode/piecestore/usedserials: add in-memory store for used serials
Implement an in-memory store for keeping track of order limit serial
numbers. It automatically deletes items if its size exceeds a configured
limit.
This change is part 1 - it creates the store
In part 2, the in-memory store will replace the usedserials database

Change-Id: I36f540ed809f034a27c1d7cede8a0a8b080af818
2020-05-28 12:52:52 -04:00
paul cannon
7395dd1e6e storagenode/gracefulexit: revalidate existing pieces
..before they are transferred to another node and submitted to the
satellite as successful piece transfers, because if we submit an invalid
signature, the node will be marked as a cheater and disqualified
immediately.

These signatures should have been validated when the piece was
originally stored, but bitrot does happen and needn't be cause for an
immediate DQ.

Change-Id: I8b0ebd5812ea8a2e60766005b7251fbb74ef7857
2020-05-28 09:50:14 -05:00
paul cannon
0c9a4a5e56 satellite/gracefulexit: better validation error messages
Change-Id: I14168dbeaf17302e5e853854956f8fbcb82a0900
2020-05-28 09:41:15 -05:00
Michal Niewrzal
84892631c8 private/testplanet: remove old libuplink from testplanet
Change-Id: Ib1553f84d0b3ae12a5b00382f0f53357b6a273e2
2020-05-28 13:50:23 +00:00
Qweder93
73214c6d1c storagenode/heldamount: heldhistory reworked to all satellites
Change-Id: I8d7707fddfbdc52d29951a8a002978c7fbb07049
2020-05-28 11:44:26 +00:00
Natalie Villasana
4ad163de2f storage/cockroachkv: rename opi to oci
opi was a carry-over from "ordered postgres iterator".

Change-Id: Id28cb8b8b3dccd119a1d1bbbb7f20206e932e4c5
2020-05-27 17:05:02 -04:00
Ethan
d5dfeca84d satellite/payments: Return empty iterator instead of nil from StripeMock List methods
Some satellites do not have payment configured (ex. Salt Lake, Europe North). In this case the StripeMock is used, which returns nil on invoice and charges List methods. This causes a panic.

https://storjlabs.atlassian.net/browse/SM-978

Change-Id: Iec1b0bfd9b383e6f793d03dd63a3dec60e0fd9f3
2020-05-27 16:20:18 +00:00
Yaroslav Vorobiov
e0d71dcaeb storagenode/windows-installer: ignore set firewall exception error
Change-Id: I176bfa4024a7395c0f68c67c5d804346a31aaeaf
2020-05-27 17:56:49 +03:00
Michal Niewrzal
2be93df566 scripts: reduce file size for backward compatibility tests
Change-Id: Ibcf460f908b6e551fe0ca68248affb7ddf49592f
2020-05-27 13:21:55 +00:00
Qweder93
f2a0c64425 storage/filestore: log potential disk corruption
In walkNamespaceWithPrefix log in case of "lstat" error, because this may indicate an underlying disk corruption.

SG-50

Change-Id: I867c3ffc47cfac325ae90658ec4780d213ff3e63
2020-05-27 12:12:55 +00:00
Qweder93
8db848791f storagenode/console: added estimated payout for current month and estimated pay stub for previous month (until there's real data in satellite's table) + heldback percentage rate for previous month.
Change-Id: I9346f6d22ed6fbb7e5346b102fc898467678f384
2020-05-27 14:51:23 +03:00
Kaloyan Raev
ee7de0424b satellite/payments: use quantities in invoices
Jira issue: https://storjlabs.atlassian.net/browse/USR-719

Invoices now show the amount of used resources and their unit price.

This change also makes proper rounding to the nearest cent in few places
to resolve the "off-by-one-cent" issue observed in few invoices.

Change-Id: I2d70d6916b5cf4a9a9138c99c422f5f4d2deb35b
2020-05-27 09:33:12 +00:00
Michal Niewrzal
75b3db5426 satellite/payments/stripecoinpayments: test invoice user with more than
1 project

https://storjlabs.atlassian.net/browse/USR-291

Change-Id: I98286e40254e8868de9eb675a9c9a8cd0bf5f3b1
2020-05-27 09:12:23 +00:00
Moby von Briesen
290c006a10 satellite/repair/{checker,queue}: add metric for new segments added to repair queue
* add monkit stat new_remote_segments_needing_repair, which reports the
number of new unhealthy segments in the repair queue since the previous
checker iteration

Change-Id: I2f10266006fdd6406ece50f4759b91382059dcc3
2020-05-27 06:23:47 +00:00
Stefan Benten
ed857f0fb4
satellite/admin: Add the proper documentation for bandwidth limits (#3900) 2020-05-27 09:22:45 +03:00
Egon Elbre
c5d4a13158 satellite/nodeselection: use NodeURL
Change-Id: I2ebd4dbf993ff5c7864f3a3a665b5c8fc48aa7d1
2020-05-27 05:46:11 +00:00
Cameron Ayer
9edb363558 satellite/metainfo: log if replacing pieces in pointer goes wrong
Every now and then we see the repair error, "piece to add already exists".
With these new logs we should be able to verify if it is due to a change in
the pointer. These logs are only temporary

Change-Id: I029390cc4816668707546df14ed2cfe7ca192b0b
2020-05-26 21:48:40 +00:00
NickolaiYurchenko
427bfc13fb web/storagenode: disk stat chrt added
Change-Id: I6f0c78992598ebbb3f1b3ee73c74e1559fc886b1
2020-05-27 00:14:33 +03:00
NickolaiYurchenko
b6771d0c52 web/storagenode: period selection separation
Change-Id: I9f50f4bb9282410f9f3cc4124e71a7a631767460
2020-05-26 23:23:19 +03:00
NickolaiYurchenko
d13b693b0c web/storagenode: total payout info area reworked
Change-Id: I2c5bf82105467db7c5919681f45d217b96e88792
2020-05-26 20:04:05 +00:00
Isaac Hess
5364a9f74d docs/blueprints: Add access-revocation.md
Change-Id: I74d9cc5564c727430b49d6b6f9040fd7df6f54e6
2020-05-26 18:59:27 +00:00
Natalie Villasana
8bd4d7b43e storage/cockroachkv: add check if retry is needed during iteration
This changeset replaces https://review.dev.storj.io/c/storj/storj/+/1839
which did the same thing but Nat couldn't figure out how to fix conflicting
files the correct gerrity way.

Change-Id: If05a8902aca986ea9f6c9168a90b31beebab839a
2020-05-26 14:32:06 -04:00
Michal Niewrzal
88a2561317 satellite/payments/stripecoinpayments: don't request anything from
Coinpayments if credentials are missing

Workaround for hitting Coinpayment API while tests.

Change-Id: I27114c820d4d7e17d05b066f55a13a709dac1711
2020-05-26 19:38:12 +02:00
Ethan
7014cf2083 satellite/gracefulexit: Change handleFailed to return nil if we can't get the pending transfer
https://storjlabs.atlassian.net/browse/SM-975

Change-Id: Ia1746b5e274e5011b6cd9f5a52f9a5faf703be51
2020-05-26 13:07:38 +00:00
Michal Niewrzal
a4c19b3359 cmd/uplink/cmd: remove port detection when selecting satellite while
setup

We don't have default ports as a part of configuration anymore because
satellite-addr flag was removed.

Change-Id: Ibf9fc4b399beaf51ebb9461de2d8994a322f9686
2020-05-26 13:25:51 +02:00
littleskunk
2fbb34c3ea
nodeselection: Increase minimum free space to 500MB (#3898) 2020-05-25 12:13:28 +02:00
Stefan Benten
3da100c242
cmd/satellite: Install curl per default to docker image (#3899) 2020-05-25 02:39:21 +02:00
Jeff Wendling
074649835b satellite/satellitedb: add some docs and improve some snapshots
This attempts to add a README.md to help create consistent migrations
that maximize our test coverage and do not include unnecessary
statements.

It also adds a feature to have an `-- OLD DATA --` section as well
as a `-- NEW DATA --` section so that we can fix mistakes made in
previous snapshots (like a row that was forgotten to be added when a
table was created) without editing them going forward.

Change-Id: I28a786f8ef163cae1de1bb08f61af1e1104b0a88
2020-05-22 21:27:36 +00:00
Michal Niewrzal
5c10964040 satellite/payments/stripecoinpayments: add test for listing issues while
invoice generation

https://review.dev.storj.io/c/storj/storj/+/1853
https://review.dev.storj.io/c/storj/storj/+/1882

Change-Id: Ie71363b819866dd60dbe7117b42cfa8348479310
2020-05-22 17:24:16 +00:00
Michal Niewrzal
340700eda9 cmd/uplink/cmd: remove old libuplink from Uplink CLI code
Change-Id: I38444a67e3bdec5f63c56895b5fc362464e7dce1
2020-05-22 09:08:58 +00:00
Bill Thorp
bcb867f1de cmd/uplink: add UserAgent flag for partner value attribution
This changeset allows a user agent string to be set during uplink setup, which
is thereafter used for partner value attribution.  EG

uplink setup --client.user-agent ”MyCompany”

Change-Id: Iefa8755fccc06acb8a303a342b943cece44a81f7
2020-05-22 07:39:14 +00:00
Michal Niewrzal
3d332de228 private/testplanet: remove StorageNodeCount from testplanet uplink
definion

Small cleanup.

Change-Id: Icabdf1433c36cd0e9f8e10a67975e98391024e14
2020-05-21 14:51:58 +00:00
crawter
f5ac678b0a storagenode/satellitesdb: added FK constraint to satelliteID
Change-Id: If5adf2b92627fcf80850670ba672b346320ddd87
2020-05-21 13:01:20 +00:00
VitaliiShpital
45ccf59067 web/satellite: onboarding tour adapted to multiple project state
Change-Id: I853772bf2a05c3fe9ef8dea0d1e290e6074fbfec
2020-05-21 12:19:33 +00:00
VitaliiShpital
f0619c6db7 web/satellite: error handling for login/signup rate limit exceeding
related jira issue:
https://storjlabs.atlassian.net/browse/USR-767

Change-Id: I4b4f27435323234ce6803c06a8c5b004f91873bf
2020-05-21 12:03:27 +00:00
VitaliiShpital
7d00b3c792 web/satellite: billing history item's expiration countdown fixed
related jira issue:
https://storjlabs.atlassian.net/browse/USR-774

Change-Id: I4629b3ea24ba4148f2faf4030d46fb5adad192fe
2020-05-21 11:47:38 +00:00
Michal Niewrzal
83cc80ef06 satellite/payments/stripecoinpayments: fix listing issue for coupon
usages and credit spendings

Test will be added as a separate change.

Change-Id: I9341d2ecb736fdbd45beb96f778f04448df68bd7
2020-05-21 07:48:04 +00:00
Egon Elbre
bef84a5f9d storagenode: remove dependency to overlay.NodeDossier
This is the last dependency from storage node to satellite.

Change-Id: I12f7abb91e84f823ba5af126c6e2979519838612
2020-05-21 08:37:13 +03:00
Egon Elbre
b42778c42e private/testplanet: remove some additional Local-s
Change-Id: I49701c41efb92efca27cc18d0a3f6d6b44d3cf8b
2020-05-21 08:37:13 +03:00
Egon Elbre
7f323754a4 metainfo/piecedeletion: use NodeURL-s
Change-Id: I247dbfe03e7864e940e4cd1d0f343f38e84099e0
2020-05-21 08:37:13 +03:00