storj/cmd/tools/segment-verify
paul cannon 8be7414a52 cmd/tools/segment-verify: fix global race in TestCommandLineTool.
This should stop the failures when TestCommandLineTool is run with
parallel test execution enabled.

Change-Id: Id80d5eacb78fcec886be786ae8b182517b17fbc6
2023-10-27 19:51:24 +00:00
..
batch.go cmd/tools/segment-verify: don't cache offline status forever 2023-01-03 23:11:42 +00:00
csv_test.go cmd/tools/segment-verify: add csv writer 2022-09-15 13:28:21 +00:00
csv.go cmd/tools/segment-verify: fix read-csv subcommand 2023-01-24 09:08:03 +00:00
main_test.go cmd/tools/segment-verify: fix global race in TestCommandLineTool. 2023-10-27 19:51:24 +00:00
main.go cmd/tools/segment-verify: fix global race in TestCommandLineTool. 2023-10-27 19:51:24 +00:00
node_check_test.go cmd/tools/segment-verify: check for unvetted nodes 2022-12-19 09:59:13 +00:00
node_check.go satellite: move satellite/nodeselection/uploadselection => satellite/nodeselection 2023-07-07 10:32:03 +02:00
nodealias_test.go cmd/tools/segment-verify: don't cache offline status forever 2023-01-03 23:11:42 +00:00
nodealias.go cmd/tools/segment-verify: don't cache offline status forever 2023-01-03 23:11:42 +00:00
process_test.go satellite/console/dbcleanup: create console DB cleanup chore 2023-05-05 19:11:53 +00:00
process.go segment-verify: synchronize refreshing of the nodeAliasMap 2023-10-19 21:01:56 +00:00
README.md cmd/tools/segment-verify: adjust to SN Exists endpoint 2022-12-21 19:24:31 +00:00
service_test.go satellite: move satellite/nodeselection/uploadselection => satellite/nodeselection 2023-07-07 10:32:03 +02:00
service.go cmd/tools/segment-verify: add CreatedBefore and CreatedAfter flags 2023-10-25 18:06:46 +00:00
summarize.go cmd/tools/segment-verify: add tool for summarizing log 2022-10-10 20:02:50 +03:00
verify_test.go cmd/tools/segment-verify: add test for whole command line tool 2023-10-25 10:21:53 +00:00
verify.go cmd/tools/segment-verify: satisfy wastedassign linter 2023-03-31 08:52:46 +00:00

segment-verify is a tool for verifying the segments.

High Level Overview

segment-verify verifies segment status on storage nodes in a few stages:

  1. First it loads the metabase for a batch of --service.batch-size=10000 segments.
  2. They are then distributed into queues using every storage nodes. It will preferentially choose nodes specified in --service.priority-nodes-path file, one storagenode id per line.
  3. Then it will query each storage node a single byte for each segment. --service.concurrency=1000 concurrent connections at a time are made.
  4. Every segment will be checked --service.check=3 times. However, any failed attempt (e.g. node is offline) is only retried once.
  5. When there are failures in verification process itself, then those segments are written into --service.retry-path=segments-retry.csv path.
  6. When the segment isn't found at least on one of the nodes, then it's written into --service.not-found-path=segments-not-found.csv file.

There are few parameters for controlling the verification itself:

# This allows to throttle requests, to avoid overloading the storage nodes.
--verify.request-throttle minimum interval for sending out each request (default 150ms)
# When there's a failure to make a request, the process will retry after this duration.
--verify.order-retry-throttle duration     how much to wait before retrying order creation (default 50ms)
# This is the time each storage-node has to respond to the request.
--verify.per-piece-timeout duration        duration to wait per piece download (default 800ms)
# Just the regular dialing timeout.
--verify.dial-timeout duration             how long to wait for a successful dial (default 2s)
# This allows to specify the minimum node version that has the Exists endpoint.
--verify.version-with-exists string        minimum storage node version with implemented Exists method (default "v1.69.2")

Running the tool

  • by specifying range boundaries:
segment-verify run range --low 00 --high ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff --config-dir ./satellite-config-dir
  • by specifying buckets to be checked:
segment-verify run buckets --buckets-csv bucket.csv