Commit Graph

12 Commits

Author SHA1 Message Date
paul cannon
727136141a satellite/cmd/tools/segment-verify: check all pieces
This adds the capability to the segment-verify tool of checking all
pieces of every indicated segment.

Pieces which could not be accessed (i.e. we couldn't get a single
byte from them) are recorded in a csv file.

I haven't been able to test this in any very meaningful way, yet, but I
am comforted by the fact that the worst things it could possibly do are
(a) download pieces too many times, and (b) miss downloading some
pieces.

Change-Id: I3aba30921572c974993363eb36d0fd5b3ae97907
2022-12-14 19:06:08 +00:00
JT Olio
58a9c55f36 mod: bump dependencies
- storj.io/common

Change-Id: Ib78154acc253a13683495abfdd96d702625fdce8
2022-10-19 17:01:53 +00:00
Egon Elbre
22c0b0ac5c cmd/tools/segment-verify: don't mark node immediately offline
Rather than marking node immediately offline, wait for more failures
until removing from the set.

Change-Id: I4363294a75d7d2844afc1f9c0025f664f933c2d7
2022-10-14 08:10:26 +00:00
Egon Elbre
a80a0ebeae cmd/tools/segment-verify: redial once rather than giving up
It's quite likely to hit some timelimit, rather than giving up
immediately, let's retry after the throttle amount.

Change-Id: I20944b058d771f5d4bfa0eea7a2c26cefcd74739
2022-10-14 01:18:31 +00:00
Egon Elbre
142a04f208 cmd/tools/segment-verify: add connection pool
Change-Id: If0f85edbf99438ac41c23fc7107fdab926288cc2
2022-10-11 09:06:44 +03:00
Egon Elbre
ea4b3023d9 cmd/tools/segment-verify: fix piece id derivation
Change-Id: Ib27fd8630e1e5a90060dff2a09c51f488960177f
2022-10-06 13:43:08 +00:00
Egon Elbre
0bfaadcc6c cmd/tools/segment-verify: fixes and more tests
* Disallow too large listing limit, which would cause a lot of memory to
  be consumed.
* Fix throttling logic and add a test.
* Fix read error handling; depending on the concurrency it can return
  the NotFound status either in the Read or Close.

Change-Id: I778f04a5961988b2480df5c7faaa22393fc5d760
2022-09-22 10:32:30 +00:00
Egon Elbre
8b527f2d12 cmd/tools/segment-verify: add throttling
Change-Id: Ia0b4ec255adc90d874f4366b80799414a1a94700
2022-09-21 14:52:51 +00:00
Egon Elbre
cf50696745 cmd/tools/segment-verify: wire up overlay logic
Change-Id: I0a4c737a8b0995a1c3e3adeac728fe833d0ce684
2022-09-19 11:32:18 +03:00
Egon Elbre
0809ae73cf cmd/tools/segment-verify: add main
Change-Id: Ib7161a0f44d447f9ddb9be83f6673587a0bd7712
2022-09-19 10:36:57 +03:00
Jennifer Johnson
8529a169ee cmd/tools/segment-verify: add verifier
Change-Id: I4cc1fbcf964c4a9a37cf80322f6f99dd956f3d7b
2022-09-19 10:36:57 +03:00
Egon Elbre
8459a3faa2 cmd/tools/segment-verify: add processing of a collection of segments
This adds logic to put segments into batches and retry them.

Change-Id: I86450e2f5ab13edb3ae9a5d5b9417e0cced2892b
2022-09-12 18:29:04 +03:00