d3604a5e90
The chief segment health models we've come up with are the "immediate danger" model and the "survivability" model. The former calculates the chance of losing a segment becoming lost in the next time period (using the CDF of the binomial distribution to estimate the chance of x nodes failing in that period), while the latter estimates the number of iterations for which a segment can be expected to survive (using the mean of the negative binomial distribution). The immediate danger model was a promising one for comparing segment health across segments with different RS parameters, as it is more precisely what we want to prevent, but it turns out that practically all segments in production have infinite health, as the chance of losing segments with any reasonable estimate of node failure rate is smaller than DBL_EPSILON, the smallest possible difference from 1.0 representable in a float64 (about 1e-16). Leaving aside the wisdom of worrying about the repair of segments that have less than a 1e-16 chance of being lost, we want to be extremely conservative and proactive in our repair efforts, and the health of the segments we have been repairing thus far also evaluates to infinity under the immediate danger model. Thus, we find ourselves reaching for an alternative. Dr. Ben saves the day: the survivability model is a reasonably close approximation of the immediate danger model, and even better, it is far simpler to calculate and yields manageable values for real-world segments. The downside to it is that it requires as input an estimate of the total number of active nodes. This change replaces the segment health calculation to use the survivability model, and reinstates the call to SegmentHealth() where it was reverted. It gets estimates for the total number of active nodes by leveraging the reliability cache. Change-Id: Ia5d9b9031b9f6cf0fa7b9005a7011609415527dc
55 lines
1.5 KiB
Go
55 lines
1.5 KiB
Go
// Copyright (C) 2020 Storj Labs, Inc.
|
|
// See LICENSE for copying information.
|
|
|
|
package repair
|
|
|
|
import (
|
|
"math"
|
|
"testing"
|
|
|
|
"github.com/stretchr/testify/assert"
|
|
)
|
|
|
|
func TestSegmentHealth(t *testing.T) {
|
|
const failureRate = 0.01
|
|
assert.Less(t,
|
|
SegmentHealth(11, 10, 10000, failureRate),
|
|
SegmentHealth(10, 5, 10000, failureRate))
|
|
assert.Less(t,
|
|
SegmentHealth(11, 10, 10000, failureRate),
|
|
SegmentHealth(10, 9, 10000, failureRate))
|
|
assert.Less(t,
|
|
SegmentHealth(10, 10, 10000, failureRate),
|
|
SegmentHealth(9, 9, 10000, failureRate))
|
|
assert.Greater(t,
|
|
SegmentHealth(11, 10, 10000, failureRate),
|
|
SegmentHealth(12, 11, 10000, failureRate))
|
|
assert.Greater(t,
|
|
SegmentHealth(13, 10, 10000, failureRate),
|
|
SegmentHealth(12, 10, 10000, failureRate))
|
|
}
|
|
|
|
func TestSegmentHealthForDecayedSegment(t *testing.T) {
|
|
const failureRate = 0.01
|
|
got := SegmentHealth(9, 10, 10000, failureRate)
|
|
assert.Equal(t, float64(0), got)
|
|
}
|
|
|
|
func TestHighHealthAndLowFailureRate(t *testing.T) {
|
|
const failureRate = 0.00005435
|
|
assert.Less(t,
|
|
SegmentHealth(36, 35, 10000, failureRate), math.Inf(1))
|
|
assert.Greater(t,
|
|
SegmentHealth(36, 35, 10000, failureRate),
|
|
SegmentHealth(35, 35, 10000, failureRate))
|
|
assert.Less(t,
|
|
SegmentHealth(60, 29, 10000, failureRate), math.Inf(1))
|
|
assert.Greater(t,
|
|
SegmentHealth(61, 29, 10000, failureRate),
|
|
SegmentHealth(60, 29, 10000, failureRate))
|
|
|
|
assert.Greater(t,
|
|
SegmentHealth(11, 10, 10000, failureRate),
|
|
SegmentHealth(39, 34, 10000, failureRate))
|
|
}
|