Commit Graph

3596 Commits

Author SHA1 Message Date
Egon Elbre
4e2bf81719 pkg/debug: add better title
Change-Id: Icc6114f4e7523cfe6c7984ef1f6eec664ae4ee65
2020-01-30 07:49:40 -05:00
littleskunk
81eddaa2c1
storagenode/monitor: reduce space requirement to 0
We have added a bug with v0.31.7 and deploying it would kick out all the
storage nodes that are full. Easy fix is setting the requirment to 0.
That will allow them to still start up even if they are full.

Change-Id: Ie66f369952d929fcfd47f44f6e5e57eea8f51ff6
2020-01-30 01:44:45 +01:00
Moby von Briesen
8c19855871 scripts/tests/rollingupgrade: explicitly set debug port for old
satellite api during rolling upgrade test

The old api is using the same config file as the new satellite in the
rolling upgrade test, so we need to set it to something different so
that there is no conflict when we spin up a new storj-sim instance while
the old api is running concurrently.

Change-Id: Ia4ec2db4953f36f43275495710992831ad3916a2
2020-01-29 18:32:03 -05:00
Egon Elbre
d10d6fd153 storagenode,satellite: ignore error on listening debug port
Change-Id: Id3a6d153535776ce41f8edf2bd6f6dad5e2a60bf
2020-01-29 18:06:02 -05:00
crawter
0b898c48a4 satellite/payments: coupons expiration logic fix
Change-Id: Ic8cc4e117957a75a3eb057075204a5b592e62ff4
2020-01-30 00:22:38 +02:00
Egon Elbre
10be538602 storagenode: add pkg/debug support
Change-Id: If941095b886c28a0d53fff4c9bf9fa0ce7471dea
2020-01-29 16:30:31 -05:00
Egon Elbre
a2b2bc676b pkg/debug: implement control panel
Control Panel allows to control different chores and services.
Currently this adds controlling of cycles.

Change-Id: I734f1676b2a0d883b8f5ba937e93c45ac1a9ce21
2020-01-29 16:30:31 -05:00
Egon Elbre
f237d70098 storagenode,satellite: use pkg/debug
Use debug.Server in storage node and satellite for customizing debug server.

Change-Id: I7979412376d028cadf29656d838ab94f18e2aa99
2020-01-29 16:30:31 -05:00
Egon Elbre
f833289e1b pkg/debug: separate debug endpoints to a server
By separating Server it allows Peers to directly embed the server
and provide customizations and hooks into rest of the services.

Change-Id: Ic1d68740fd494d2f82c1739bd990849c561b912b
2020-01-29 16:30:31 -05:00
littleskunk
e0cb8037c1 satellite/projectusage: reduce usage limit from 5GB to 0GB
Change-Id: Ie3d2509613e7a4336e2a8d2b136b32f5f308aafc
2020-01-29 20:38:39 +00:00
crawter
9bb7ceb651 satellite\payments: amount for coupons increased
Change-Id: I1f357b76361e6e3e50bbe4ee66a8edb6ff033f36
2020-01-29 22:08:54 +02:00
Egon Elbre
da5e408afe storage: add DeleteMultiple method
DeleteMultiple will allow metainfo to delete multiple segments
and get the old pointers in a single request.

Change-Id: Ic144f30c5453274fa2b80df2895f123f5a9cc48b
2020-01-29 13:13:54 -05:00
crawter
f4667426b5 satellite\payments: project limits for coupons increased
Change-Id: I51eb47eb635fd096348befd39b7efbe3ce8982d6
2020-01-29 19:34:50 +02:00
NikolaiYurchenko
1ad4110d61 web/storagenode: notification red dot hide
Change-Id: I9a7b312a66d2d30216c324a2a45fd45cad969271
2020-01-29 17:09:52 +00:00
NikolaiYurchenko
e641ff45a5 web/satellite: logout fix
Change-Id: I1b2b14c098e0959e9c5bd36adc889a425d00963c
2020-01-29 16:53:21 +00:00
Matt Robinson
7cbd83a399 jenkins: Remove os_arch from file inside zip file artifact
This changes the contents of the zip files, removing the _darwin_amd64
and other bits. Windows builds should still have filenames ending with
.exe.

Change-Id: I25a582034b59200c1d6034dd2925cdeaf06844e2
2020-01-29 16:29:14 +00:00
Ethan
149273c63f satellite/metainfo: add cache expiration for project level rate limiting
Allow rate limit project cache to expire so we can make project level rate limit changes without restarting the satellite process.

Change-Id: I159ea22edff5de7cbfcd13bfe70898dcef770e42
2020-01-29 16:14:10 +00:00
Stefan Benten
d30d2d920d satellite/metainfo: Adding Monkit Meters to the Request Logs
Change-Id: I33d56510cf72d5f8512c1069ce65856cba7f8957
2020-01-29 15:51:36 +00:00
ccase
e87886696e satellite/metainfo: Too many requests should have RPC status ResourceExhaused
This is necessary to for the client to know that it can retry with a
delay.

Change-Id: Ie0ed95f6ae1c072896285d0714f879611ab0cdb3
2020-01-29 15:06:22 +00:00
crawter
e549e32976 satellite/payments: fix promotional coupons
Change-Id: Ib8b7e38f2cb07085655448264f281fd7fc7867dd
2020-01-29 16:40:43 +02:00
Matt Robinson
f6641aa434 jenkins: use go 1.13.7
Change-Id: I40dc231678aa7909efeb8586f2fd09713ef6f395
2020-01-29 13:42:14 +00:00
Yaroslav Vorobiov
6b72bf92ce satellite/payments: convert egress price to per byte basis
Change-Id: Ia3a07d0afa5d9d55871996a1d2117b4ec290ce8f
2020-01-29 00:06:01 -05:00
Yaroslav Vorobiov
083b396c16 satellite/payments: allow floating point numbers for pricing
Change-Id: I78b60134cf043746efef5371b761939a10f75aaf
2020-01-28 22:52:13 -05:00
littleskunk
a0c9f7f3b0
satellite/projectusage: reduce usage limit from 25GB to 5GB
Change-Id: I2819012b520fd687ab8058000aa38d76b8208158
2020-01-29 04:01:09 +01:00
Egon Elbre
e66b3c9be1 satellite: remove repair worker from core
Core shouldn't be handling any repair load and we have already disabled it in production.
Let's make it official and remove it.

Change-Id: I46e236692a9164421648cfc974dd3246416b2e00
2020-01-28 20:02:30 -05:00
Egon Elbre
e319660f7a private/lifecycle: implement Group
lifecycle.Group implements controlling multiple items such
that their startup and close works.

Change-Id: Idb4f4a6c3a1f07cdcf44d3147a6c959686df0007
2020-01-29 00:37:33 +00:00
Jessica Grebenschikov
a1948ed338 satellite/orders: add old method for CreateGetOrderLimitsOld to maintain compatibility with old versions of the uplink
Change-Id: I7ce1f4fbc6217f1d340cf778c4b010d40961b3f0
2020-01-28 18:54:24 -05:00
Fadila Khadar
3a0f6de66d cmd/uplink: list available accesses
Change-Id: I7748cc68cf21b514285579d15c1cab8329ac3d2b
2020-01-28 22:29:49 +00:00
Jessica Grebenschikov
54dbaaece2 satellite/orders: create as many orderLimits as needed to download a file
Change-Id: I2a39483d35037d9940913c035a78a93ea692ce9f
2020-01-28 20:04:11 +00:00
paul cannon
8ce9ce7f0f satellite/gracefulexit: wait for errgroup to return
credit to Yingrong

Change-Id: I538371040d4dcdf6e943c61e8454320fd57b7526
2020-01-28 19:26:43 +00:00
Yingrong Zhao
d8e3556a22 storagenode/preflight: wait for server to shutdown when tests are
finished

Change-Id: Ie3ede9f285cb61bb6bc6b0158e41d8ea10b2497e
2020-01-28 17:54:19 +00:00
Michal Niewrzal
90fc1922d0 satellite/metainfo: override bucket RS values with satellite config
Satellite now is keeping RS values for uplink but old uplinks were using
default bucket settings. Because of that we need to override buckets
settings with satellite settings to avoid breaking older uplinks.

Change-Id: Ia1068db70e4adbf741c5e81d27d9e39799049c22
2020-01-28 15:51:04 +00:00
Stefan Benten
3abb8c8ed7 Dont require an IP address being set
Per default our server address is listening on all IP addresses on the machine.
This caused our preflight check to fail, as it did not have an hostname to lookup.
With this change, we are fine with this and go ahead.

Change-Id: I9eb5c891c099eb35f679d6d7e79ec38bb43b619f
2020-01-28 15:25:17 +01:00
NikolaiYurchenko
4e819c196b web/satellite: uncaught router exception fixed
Change-Id: Ia20204e710d4d6d162522244fb02e0356ef49483
2020-01-28 16:04:21 +02:00
Jennifer Johnson
2209924d41 satellite/satellitedb: use arrays and batch inserts for SaveTallies query
Cockroachdb is more performant with multi-row inserts

Change-Id: Ie1ce2a9da0be1df4e66e72fc9cae49cbd95023f3
2020-01-27 16:54:20 -05:00
Egon Elbre
227e03dea1 satellite/satellitedb: insert using arrays
Using dynamic query strings is error prone, prefer arrays.

Change-Id: I303fbf21c6a54795bd9f399371943b5c51e6f863
2020-01-27 21:27:28 +00:00
Jeff Wendling
d09bd4a749 satellite/satellitedb/dbx: regenerate with paged composite key fixes
before dbx would generate a compilcated blob of conditions that
encoded a row comparison, which only optimized to an index seek
on cockroachdb. this means that sqlite and postgres both had
quadratic behavior on paged queries of this form. instead, use
the implicit row construction feature supported in all of the
databases to do paged support so that they all optimize well.

Change-Id: Iac8703929ba2a59ee3ffa619b916d12663422887
2020-01-27 12:43:16 -07:00
Bryan White
ae3f471471 cmd/satellite: close pointerDB properly
Change-Id: Ie26d23c2bfe46def2bb93896fb8b60ef17fa2ef6
2020-01-27 19:11:54 +00:00
Yingrong Zhao
f3fcbe256c satellite/metainfo: revert combine CommitSegment and CommitObject in batch
This reverts commit 8772867855.

for uplink versions v0.25.0 through v0.30.7, there's a bug with multiplesegment upload
where the last segment is inline caused by this commit.

Change-Id: If375e186b23265586caf08991c25980e99f3cc1a
2020-01-27 13:26:33 -05:00
NikolaiYurchenko
9bcb81108f web/satellite: verification email change
Change-Id: I0293ef4411b55e42bb372b230d797d6798eda515
2020-01-27 15:55:52 +02:00
Egon Elbre
f4317d257a mod: bump uplink and common
Change-Id: I83874539c705e4b22940bc15ea990fe879dde721
2020-01-27 08:20:10 -05:00
Michal Niewrzal
ca32ffbfc5 satellite/metainfo: move deletion before upload to satellite
Change is adding object deletion to BeginObject request (before upload).
Now when satellite controls deletion we can move deletion before upload
to satellite. This change improves two things:
* no need for additional request to delete object before upload (need
one more change to storj/uplink)
* fix an issue with lack of permissions to upload if caveat allows only
for writing (e.g. disallow deletes but allows to write)

https://storjlabs.atlassian.net/browse/V3-3362

Change-Id: Ic453146298cdd302df290c532123731a3f99e38e
2020-01-27 12:48:10 +00:00
nerdatwork
913a80c20a web/satellite: Fixing typos in website pages
typo for text `website` shown as  `websities`

Attribution to @nerdatwork
2020-01-27 13:29:32 +01:00
nerdatwork
9ea32016c2 storagenode/orders: fix typos in log messages (#3760) 2020-01-26 13:45:57 -05:00
paul cannon
a0a94a9ac7 satellite/satellitedb: insert into reported_serials w/ arrays
Change-Id: Icb682de09ded3e3159e3590594dcf13f2e7f40f0
2020-01-24 18:36:21 -06:00
littleskunk
90cf78e6f2 satellite/coinpayments: fix migration
The old migration was not working. It was updateding pending (status 0)
and failed (status -1) to completed (status 100).

Change-Id: I808ff3cc692fe6c698ce26a8b411b134e67b752b
2020-01-25 00:12:35 +00:00
littleskunk
5c68f4fc7c storagenode/gracefulexit: higher concurrency and shorter timeouts
1 transfer with a minimum speed of 128 Bytes was a nice try but it is
way too low. Even a pi3 was able to handle 7 grpc transfers. We have 4
satellites and with 5 concurrent transfers that should be a total of 20
concurrent transfers. Each transfer will have a minimum speed of 5KB/s.
That should give us a better througput and still be Ok on a pi3.

Change-Id: I650a7baf890080901ef70ea3b5636d93009b4e60
2020-01-24 23:51:39 +00:00
littleskunk
a6c6440ab7 satellite/order: decrease expire time from 7 days to 2 days
For the last few month we had no issues with order submission. I would
call it stable and now it is time to risk a lower expire time. This will
increase the database performance on the satellite and it will reduce
the delay for billing.

The long term goal is 6h but for that step we need to change graceful
exit first. At the moment storage nodes would get disuqlaified for not
transfering alle pieces in less than 6 hours.

Change-Id: I421a2c2421c5374c4e706e2338f1c2161fedc14c
2020-01-24 23:37:39 +00:00
littleskunk
226bc4de36 storagenode/preflightcheck: enable database check by default
With the v0.30.5 release we asked the storage node operators to manually
enable the preflight check while they are in front of their machine. We
didn't want to risk taking too many storage nodes offline at the same
time because of some unknow bug. The preflight check worked. We have no
negative feedback. We can now enable it by default.

Change-Id: Ic670ee52becd0b35eca84af7a0841ea983d7b19d
2020-01-24 23:23:35 +00:00
Moby von Briesen
e4cff1c938 storagenode/preflight: update allowed time difference for preflight
clock sync

Change 24h and 1h to 30m and 10m respectively for clock sync. If a
storagenode's clock is off by more than 30m for every trusted satellite,
it will not start. If it is off by more than 10m for any trusted
satellite, a warning is displayed.

Change-Id: I05ef611a30a49c1783e3b68b513745922c2f7e28
2020-01-24 22:57:13 +00:00