c1817ab743
The flags weren't properly loading from config. The code assumed that every node that's online for downloading also have data uploaded to them -- which is not true. Change-Id: Ifd65a47b9eca5b4841231928244fab17acbde6fb
1.8 KiB
1.8 KiB
segment-verify is a tool for verifying the segments.
High Level Overview
segment-verify verifies segment status on storage nodes in a few stages:
- First it loads the metabase for a batch of
--service.batch-size=10000segments. - They are then distributed into queues using every storage nodes. It will preferentially choose nodes specified in
--service.priority-nodes-pathfile, one storagenode id per line. - Then it will query each storage node a single byte for each segment.
--service.concurrency=1000concurrent connections at a time are made. - Every segment will be checked
--service.check=3times. However, any failed attempt (e.g. node is offline) is only retried once. - When there are failures in verification process itself, then those segments are written into
--service.retry-path=segments-retry.csvpath. - When the segment isn't found at least on one of the nodes, then it's written into
--service.not-found-path=segments-not-found.csvfile.
There are few parameters for controlling the verification itself:
# This allows to throttle requests, to avoid overloading the storage nodes.
--verify.request-throttle minimum interval for sending out each request (default 150ms)
# When there's a failure to make a request, the process will retry after this duration.
--verify.order-retry-throttle duration how much to wait before retrying order creation (default 50ms)
# This is the time each storage-node has to respond to the request.
--verify.per-piece-timeout duration duration to wait per piece download (default 800ms)
# Just the regular dialing timeout.
--verify.dial-timeout duration how long to wait for a successful dial (default 2s)
Running the tool
segment-verify run range --low 00 --high ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff --config-dir ./satellite-config-dir