Commit Graph

3815 Commits

Author SHA1 Message Date
Jeff Wendling
d20db90cff private/dbutil/txutil: create new transactions for retries
it was noticed that if you had a long lived transaction A that
was blocking some other transaction B and A was being aborted
due to retriable errors, then transaction B was never given
priority. this was due to using savepoints to do lightweight
retries.

this behavior was problematic becaue we had some queries blocked
for over 16 hours, so this commit addresses the issue with two
prongs:

    1. bound the amount of time we will retry a transaction
    2. create new transactions when a retry is needed

the first ensures that we never wait for 16 hours, and the value
chosen is 10 minutes. that should be long enough for an ample
amount of retries for small queries, and huge queries probably
shouldn't be retried, even if possible: it's more preferrable to
find a way to make them smaller.

the second ensures that even in the case of retries, queries that
are blocked on the aborted transaction gain priority to run.

between those two changes, the maximum stall time due to retries
should be bounded to around 10 minutes.

Change-Id: Icf898501ef505a89738820a3fae2580988f9f5f4
2020-02-01 18:34:28 +00:00
Jeff Wendling
71ff044edb storagenode/bandwidth: fix tests to not fail for 10 hours near the end of the month
Change-Id: I390569a8702164c42edddd3be020e93782227c2e
2020-01-31 16:25:52 -07:00
Jeff Wendling
03166d6be3 storagenode/piecestore: log available bandwidth and space on uploads
Change-Id: Ia92228cb2a178da45f4f123b48c476e5ec821fe8
2020-01-31 19:47:14 +00:00
Egon Elbre
97d360afd1 satellite/satellitedb: use correct type
Array was using a smaller type integer.

Change-Id: I025d61b6cea9869efa0b4ac1d24265356491f6dc
2020-01-31 13:00:14 -05:00
VitaliiShpital
ae3eae31dc web/storagenode: chart point's hit radius extended
Change-Id: Ib231d7970b4fe9e196638d3a09d2ae96e6c4efc5
2020-01-31 17:37:37 +00:00
VitaliiShpital
d3fe122d9b web/satellite: project/billing dropdown bug fix
Change-Id: I1276ad20bd4efc011a1e705c5401260779b7b610
2020-01-31 17:09:54 +00:00
VitaliiShpital
2cd233e1f3 web/storagenode: dashboard content header reworked
Change-Id: I725efffc8aecf0b8b53c9ef86c59fbcb065ecc04
2020-01-31 16:54:49 +00:00
Isaac Hess
78d0868bc9 storagenode/pieces: Log error if cannot calculate piece size
Change-Id: I33b49315a0f6044a801a8b118e6b61dbcd751bfe
2020-01-31 09:57:44 -05:00
Egon Elbre
d0b4272467 storagenode: fix global logger in tests
https://github.com/storj/storj/wiki/Testing#logging

Change-Id: Ic6a31360bcfedae3f37f6b2536a345f00e33cd78
2020-01-31 14:09:28 +00:00
Isaac Hess
2968857e21 storagenode/pieces: Prevent recalculate from having negative numbers
Change-Id: Iafd2bcb9963e85508cb5e2bd69f229d89c589a6c
2020-01-30 17:47:54 -05:00
Michal Niewrzal
a181e0b627 libuplink: adjust tests to changes in encryption store
We move PathCipher to encryption.Store and we need to adjust
storj/uplink for those changes. Uplink repo is also using libuplink to
run tests so we need first adjust storj/storj libuplink and later
storj/uplink.

Change-Id: I84f23e6bad18ac139f72c19939dc526f9f46d88b
2020-01-30 22:00:24 +00:00
Egon Elbre
81d44f19ee storage/filestore: ensure we bail on deleted folder without error
Change-Id: Iecf5f9ac5bc278489b433923c526d60611d356a4
2020-01-30 16:32:10 -05:00
paul cannon
157b8c4d71 storagenode/pieces: accumulate errors in traversal
instead of aborting on the first error, so that we can hit all
satellites and get the best numbers we can

Change-Id: I21d5163884940612d7d39eaf73a6fac07235cd9e
2020-01-30 19:31:29 +00:00
Isaac Hess
5a053483b7 storagenode/pieces: read trash from blobstore
Change-Id: Ib134e63a13b8a5dda5d6a9ead42013ce18411227
2020-01-30 13:30:48 -05:00
Isaac Hess
4dafd03f11 storagenode: Prevent negative values in piece_space_used, migrate negatives to 0
Change-Id: Ibd663db087058c928190aa52c520f22e9338dd04
2020-01-30 13:03:18 -05:00
Isaac Hess
00fc192f6b storagenode/pieces: Explicitly walk satellite pieces in SpaceUsedTotalAndBySatellite
Change-Id: I7ff9a1120d4ced0b5cba7d7765ef8aed7a1edae0
2020-01-30 12:01:50 -06:00
Jeff Wendling
21b65ca3b0 storagenode/storagenodedb: migrate to set total to content_size
Change-Id: I4906c2fe9cdb3a32c045c98039d4bde6b8b809e3
2020-01-30 08:53:12 -07:00
Moby von Briesen
006a2824ba satellite/repair: lock monkit stats in checker and repairer
Change-Id: Ia10fc8da0177389a500359ce51d21a5806f3f7b1
2020-01-30 14:09:56 +00:00
Egon Elbre
8dea4f52db satellite: add control panel
Change-Id: Id48246e9bcd4c6ec643277fe740937b2e42ad85b
2020-01-30 08:06:43 -05:00
Egon Elbre
4e2bf81719 pkg/debug: add better title
Change-Id: Icc6114f4e7523cfe6c7984ef1f6eec664ae4ee65
2020-01-30 07:49:40 -05:00
littleskunk
81eddaa2c1
storagenode/monitor: reduce space requirement to 0
We have added a bug with v0.31.7 and deploying it would kick out all the
storage nodes that are full. Easy fix is setting the requirment to 0.
That will allow them to still start up even if they are full.

Change-Id: Ie66f369952d929fcfd47f44f6e5e57eea8f51ff6
2020-01-30 01:44:45 +01:00
Moby von Briesen
8c19855871 scripts/tests/rollingupgrade: explicitly set debug port for old
satellite api during rolling upgrade test

The old api is using the same config file as the new satellite in the
rolling upgrade test, so we need to set it to something different so
that there is no conflict when we spin up a new storj-sim instance while
the old api is running concurrently.

Change-Id: Ia4ec2db4953f36f43275495710992831ad3916a2
2020-01-29 18:32:03 -05:00
Egon Elbre
d10d6fd153 storagenode,satellite: ignore error on listening debug port
Change-Id: Id3a6d153535776ce41f8edf2bd6f6dad5e2a60bf
2020-01-29 18:06:02 -05:00
crawter
0b898c48a4 satellite/payments: coupons expiration logic fix
Change-Id: Ic8cc4e117957a75a3eb057075204a5b592e62ff4
2020-01-30 00:22:38 +02:00
Egon Elbre
10be538602 storagenode: add pkg/debug support
Change-Id: If941095b886c28a0d53fff4c9bf9fa0ce7471dea
2020-01-29 16:30:31 -05:00
Egon Elbre
a2b2bc676b pkg/debug: implement control panel
Control Panel allows to control different chores and services.
Currently this adds controlling of cycles.

Change-Id: I734f1676b2a0d883b8f5ba937e93c45ac1a9ce21
2020-01-29 16:30:31 -05:00
Egon Elbre
f237d70098 storagenode,satellite: use pkg/debug
Use debug.Server in storage node and satellite for customizing debug server.

Change-Id: I7979412376d028cadf29656d838ab94f18e2aa99
2020-01-29 16:30:31 -05:00
Egon Elbre
f833289e1b pkg/debug: separate debug endpoints to a server
By separating Server it allows Peers to directly embed the server
and provide customizations and hooks into rest of the services.

Change-Id: Ic1d68740fd494d2f82c1739bd990849c561b912b
2020-01-29 16:30:31 -05:00
littleskunk
e0cb8037c1 satellite/projectusage: reduce usage limit from 5GB to 0GB
Change-Id: Ie3d2509613e7a4336e2a8d2b136b32f5f308aafc
2020-01-29 20:38:39 +00:00
crawter
9bb7ceb651 satellite\payments: amount for coupons increased
Change-Id: I1f357b76361e6e3e50bbe4ee66a8edb6ff033f36
2020-01-29 22:08:54 +02:00
Egon Elbre
da5e408afe storage: add DeleteMultiple method
DeleteMultiple will allow metainfo to delete multiple segments
and get the old pointers in a single request.

Change-Id: Ic144f30c5453274fa2b80df2895f123f5a9cc48b
2020-01-29 13:13:54 -05:00
crawter
f4667426b5 satellite\payments: project limits for coupons increased
Change-Id: I51eb47eb635fd096348befd39b7efbe3ce8982d6
2020-01-29 19:34:50 +02:00
NikolaiYurchenko
1ad4110d61 web/storagenode: notification red dot hide
Change-Id: I9a7b312a66d2d30216c324a2a45fd45cad969271
2020-01-29 17:09:52 +00:00
NikolaiYurchenko
e641ff45a5 web/satellite: logout fix
Change-Id: I1b2b14c098e0959e9c5bd36adc889a425d00963c
2020-01-29 16:53:21 +00:00
Matt Robinson
7cbd83a399 jenkins: Remove os_arch from file inside zip file artifact
This changes the contents of the zip files, removing the _darwin_amd64
and other bits. Windows builds should still have filenames ending with
.exe.

Change-Id: I25a582034b59200c1d6034dd2925cdeaf06844e2
2020-01-29 16:29:14 +00:00
Ethan
149273c63f satellite/metainfo: add cache expiration for project level rate limiting
Allow rate limit project cache to expire so we can make project level rate limit changes without restarting the satellite process.

Change-Id: I159ea22edff5de7cbfcd13bfe70898dcef770e42
2020-01-29 16:14:10 +00:00
Stefan Benten
d30d2d920d satellite/metainfo: Adding Monkit Meters to the Request Logs
Change-Id: I33d56510cf72d5f8512c1069ce65856cba7f8957
2020-01-29 15:51:36 +00:00
ccase
e87886696e satellite/metainfo: Too many requests should have RPC status ResourceExhaused
This is necessary to for the client to know that it can retry with a
delay.

Change-Id: Ie0ed95f6ae1c072896285d0714f879611ab0cdb3
2020-01-29 15:06:22 +00:00
crawter
e549e32976 satellite/payments: fix promotional coupons
Change-Id: Ib8b7e38f2cb07085655448264f281fd7fc7867dd
2020-01-29 16:40:43 +02:00
Matt Robinson
f6641aa434 jenkins: use go 1.13.7
Change-Id: I40dc231678aa7909efeb8586f2fd09713ef6f395
2020-01-29 13:42:14 +00:00
Yaroslav Vorobiov
6b72bf92ce satellite/payments: convert egress price to per byte basis
Change-Id: Ia3a07d0afa5d9d55871996a1d2117b4ec290ce8f
2020-01-29 00:06:01 -05:00
Yaroslav Vorobiov
083b396c16 satellite/payments: allow floating point numbers for pricing
Change-Id: I78b60134cf043746efef5371b761939a10f75aaf
2020-01-28 22:52:13 -05:00
littleskunk
a0c9f7f3b0
satellite/projectusage: reduce usage limit from 25GB to 5GB
Change-Id: I2819012b520fd687ab8058000aa38d76b8208158
2020-01-29 04:01:09 +01:00
Egon Elbre
e66b3c9be1 satellite: remove repair worker from core
Core shouldn't be handling any repair load and we have already disabled it in production.
Let's make it official and remove it.

Change-Id: I46e236692a9164421648cfc974dd3246416b2e00
2020-01-28 20:02:30 -05:00
Egon Elbre
e319660f7a private/lifecycle: implement Group
lifecycle.Group implements controlling multiple items such
that their startup and close works.

Change-Id: Idb4f4a6c3a1f07cdcf44d3147a6c959686df0007
2020-01-29 00:37:33 +00:00
Jessica Grebenschikov
a1948ed338 satellite/orders: add old method for CreateGetOrderLimitsOld to maintain compatibility with old versions of the uplink
Change-Id: I7ce1f4fbc6217f1d340cf778c4b010d40961b3f0
2020-01-28 18:54:24 -05:00
Fadila Khadar
3a0f6de66d cmd/uplink: list available accesses
Change-Id: I7748cc68cf21b514285579d15c1cab8329ac3d2b
2020-01-28 22:29:49 +00:00
Jessica Grebenschikov
54dbaaece2 satellite/orders: create as many orderLimits as needed to download a file
Change-Id: I2a39483d35037d9940913c035a78a93ea692ce9f
2020-01-28 20:04:11 +00:00
paul cannon
8ce9ce7f0f satellite/gracefulexit: wait for errgroup to return
credit to Yingrong

Change-Id: I538371040d4dcdf6e943c61e8454320fd57b7526
2020-01-28 19:26:43 +00:00
Yingrong Zhao
d8e3556a22 storagenode/preflight: wait for server to shutdown when tests are
finished

Change-Id: Ie3ede9f285cb61bb6bc6b0158e41d8ea10b2497e
2020-01-28 17:54:19 +00:00