Commit Graph

15 Commits

Author SHA1 Message Date
paul cannon
6e1554652a cmd/tools/segment-verify: handle dq'd nodes
Previously, if any pieces are still on disqualified nodes, this tool
would treat those pieces as fine (if the disqualified node is still
online) or temporarily unavailable (if the disqualified node is
offline). Instead, we should treat such pieces as lost.

This also fixes a slight problem with the code that handles a broken
alias. This is not likely to happen, but if we do see an alias that is
not in the alias map, we return an error instead of nil.

Change-Id: Ib4e2e729ef0535dd7bd9ce2f621680d9f959891c
2023-01-04 17:54:03 +00:00
paul cannon
2feb49afc3 cmd/tools/segment-verify: don't cache offline status forever
Because it was originally intended to work on only a few pieces from
each segment at a time, and would frequently have reset its list of
online nodes, segment-verify has been taking nodes out of its
onlineNodes set and never putting them back. This means that over a long
run in Check=0 mode, we end up treating more and more nodes as offline,
permanently. This trend obfuscates the number of missing pieces that
each segment really has, because we don't check pieces on offline nodes.

This commit changes the onlineNodes set to an "offlineNodes" set, with
an expiration time on the offline-ness quality. So nodes are added to
the offlineNodes set when we see they are offline, and then we only
treat them as offline for the next 30 minutes (configurable). After that
point, we will try connecting to them again.

Change-Id: I14f0332de25cdc6ef655f923739bcb4df71e079e
2023-01-03 23:11:42 +00:00
Clement Sam
cda1d67465 cmd/tools/segment-verify: adjust to SN Exists endpoint
Change-Id: I409aeae29aa87996f2a6047f976d215a69e9d7f5
2022-12-21 19:24:31 +00:00
paul cannon
727136141a satellite/cmd/tools/segment-verify: check all pieces
This adds the capability to the segment-verify tool of checking all
pieces of every indicated segment.

Pieces which could not be accessed (i.e. we couldn't get a single
byte from them) are recorded in a csv file.

I haven't been able to test this in any very meaningful way, yet, but I
am comforted by the fact that the worst things it could possibly do are
(a) download pieces too many times, and (b) miss downloading some
pieces.

Change-Id: I3aba30921572c974993363eb36d0fd5b3ae97907
2022-12-14 19:06:08 +00:00
Egon Elbre
22c0b0ac5c cmd/tools/segment-verify: don't mark node immediately offline
Rather than marking node immediately offline, wait for more failures
until removing from the set.

Change-Id: I4363294a75d7d2844afc1f9c0025f664f933c2d7
2022-10-14 08:10:26 +00:00
Egon Elbre
ea4b3023d9 cmd/tools/segment-verify: fix piece id derivation
Change-Id: Ib27fd8630e1e5a90060dff2a09c51f488960177f
2022-10-06 13:43:08 +00:00
Egon Elbre
c8506cdda3 satellite/metabase,cmd/tools/segment-verify: simplify interface
Change-Id: Icdd445b1713bc26cee3b3a125b68b0cde0739837
2022-10-06 13:42:00 +00:00
Egon Elbre
f98d551c9b cmd/tools/segment-verify: test service
Change-Id: Ibd83960c18123e8f29e22089007dc32c8d532240
2022-09-22 17:23:02 +00:00
Egon Elbre
8b527f2d12 cmd/tools/segment-verify: add throttling
Change-Id: Ia0b4ec255adc90d874f4366b80799414a1a94700
2022-09-21 14:52:51 +00:00
Egon Elbre
cf50696745 cmd/tools/segment-verify: wire up overlay logic
Change-Id: I0a4c737a8b0995a1c3e3adeac728fe833d0ce684
2022-09-19 11:32:18 +03:00
Egon Elbre
0809ae73cf cmd/tools/segment-verify: add main
Change-Id: Ib7161a0f44d447f9ddb9be83f6673587a0bd7712
2022-09-19 10:36:57 +03:00
Jennifer Johnson
8529a169ee cmd/tools/segment-verify: add verifier
Change-Id: I4cc1fbcf964c4a9a37cf80322f6f99dd956f3d7b
2022-09-19 10:36:57 +03:00
Egon Elbre
507b099d44 cmd/tools/segment-verify: add monitoring / error
Change-Id: I6fd0369719ddf176a98208348560004a4134f810
2022-09-14 18:20:48 +00:00
Egon Elbre
6127f465dc cmd/tools/segment-verify: add logic for iterating over segments
This adds parts for:
1. iterating over the segments
2. using an interface for writing the segments
3. stubs for handling deleted segments

Change-Id: I76a17cac6deb0b6c042a8ab7c4155a890db9da84
2022-09-14 18:20:31 +00:00
Egon Elbre
8459a3faa2 cmd/tools/segment-verify: add processing of a collection of segments
This adds logic to put segments into batches and retry them.

Change-Id: I86450e2f5ab13edb3ae9a5d5b9417e0cced2892b
2022-09-12 18:29:04 +03:00