3a9ad48345
This csv file was being closed as soon as the service was created. All subsequent writes to the closed file handle produced errors, which were logged but otherwise ignored. Instead, we would like the file to remain open and writable, until the service is destroyed. Change-Id: Ib29944d25b2f5b2d0f90fdbdcde44fea8d769321 |
||
|---|---|---|
| .. | ||
| batch.go | ||
| csv_test.go | ||
| csv.go | ||
| main.go | ||
| node_check_test.go | ||
| node_check.go | ||
| nodealias_test.go | ||
| nodealias.go | ||
| process_test.go | ||
| process.go | ||
| README.md | ||
| service_test.go | ||
| service.go | ||
| summarize.go | ||
| verify_test.go | ||
| verify.go | ||
segment-verify is a tool for verifying the segments.
High Level Overview
segment-verify verifies segment status on storage nodes in a few stages:
- First it loads the metabase for a batch of
--service.batch-size=10000segments. - They are then distributed into queues using every storage nodes. It will preferentially choose nodes specified in
--service.priority-nodes-pathfile, one storagenode id per line. - Then it will query each storage node a single byte for each segment.
--service.concurrency=1000concurrent connections at a time are made. - Every segment will be checked
--service.check=3times. However, any failed attempt (e.g. node is offline) is only retried once. - When there are failures in verification process itself, then those segments are written into
--service.retry-path=segments-retry.csvpath. - When the segment isn't found at least on one of the nodes, then it's written into
--service.not-found-path=segments-not-found.csvfile.
There are few parameters for controlling the verification itself:
# This allows to throttle requests, to avoid overloading the storage nodes.
--verify.request-throttle minimum interval for sending out each request (default 150ms)
# When there's a failure to make a request, the process will retry after this duration.
--verify.order-retry-throttle duration how much to wait before retrying order creation (default 50ms)
# This is the time each storage-node has to respond to the request.
--verify.per-piece-timeout duration duration to wait per piece download (default 800ms)
# Just the regular dialing timeout.
--verify.dial-timeout duration how long to wait for a successful dial (default 2s)
# This allows to specify the minimum node version that has the Exists endpoint.
--verify.version-with-exists string minimum storage node version with implemented Exists method (default "v1.69.2")
Running the tool
- by specifying range boundaries:
segment-verify run range --low 00 --high ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff --config-dir ./satellite-config-dir
- by specifying buckets to be checked:
segment-verify run buckets --buckets-csv bucket.csv