d21bbab2b2
New flag 'MultipleVersions' was not correctly passed from metainfo configuration to metabase configuration. Because configuration was set correctly for unit tests we didn't catch it and issue was found while testing on QA satellite. This change reduce number of places where new metabase flags needs to be propagated from metainfo configuration to avoid problems with setting new flags in the future. Fixes https://github.com/storj/storj/issues/5274 Change-Id: I74bc122649febefd87f665be2fba628f6bfd9044 |
||
|---|---|---|
| .. | ||
| batch.go | ||
| csv_test.go | ||
| csv.go | ||
| main.go | ||
| nodealias.go | ||
| process_test.go | ||
| process.go | ||
| README.md | ||
| service_test.go | ||
| service.go | ||
| summarize.go | ||
| verify_test.go | ||
| verify.go | ||
segment-verify is a tool for verifying the segments.
High Level Overview
segment-verify verifies segment status on storage nodes in a few stages:
- First it loads the metabase for a batch of
--service.batch-size=10000segments. - They are then distributed into queues using every storage nodes. It will preferentially choose nodes specified in
--service.priority-nodes-pathfile, one storagenode id per line. - Then it will query each storage node a single byte for each segment.
--service.concurrency=1000concurrent connections at a time are made. - Every segment will be checked
--service.check=3times. However, any failed attempt (e.g. node is offline) is only retried once. - When there are failures in verification process itself, then those segments are written into
--service.retry-path=segments-retry.csvpath. - When the segment isn't found at least on one of the nodes, then it's written into
--service.not-found-path=segments-not-found.csvfile.
There are few parameters for controlling the verification itself:
# This allows to throttle requests, to avoid overloading the storage nodes.
--verify.request-throttle minimum interval for sending out each request (default 150ms)
# When there's a failure to make a request, the process will retry after this duration.
--verify.order-retry-throttle duration how much to wait before retrying order creation (default 50ms)
# This is the time each storage-node has to respond to the request.
--verify.per-piece-timeout duration duration to wait per piece download (default 800ms)
# Just the regular dialing timeout.
--verify.dial-timeout duration how long to wait for a successful dial (default 2s)
Running the tool
segment-verify run range --low 00 --high ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff --config-dir ./satellite-config-dir