2522ff09b6
Up to now, we have been implementing the DistinctIP preference with code
in two places:
1. On check-in, the last_net is determined by taking the /24 or /64
(in ResolveIPAndNetwork()) and we store it with the node record.
2. On node selection, a preference parameter defines whether to return
results that are distinct on last_net.
It can be observed that we have never yet had the need to switch from
DistinctIP to !DistinctIP, or from !DistinctIP to DistinctIP, on the
same satellite, and we will probably never need to do so in an automated
way. It can also be observed that this arrangement makes tests more
complicated, because we often have to arrange for test nodes to have IP
addresses in different /24 networks (a particular pain on macOS).
Those two considerations, plus some pending work on the repair framework
that will make repair take last_net into consideration, motivate this
change.
With this change, in the #2 place, we will _always_ return results that
are distinct on last_net. We implement the DistinctIP preference, then,
by making the #1 place (ResolveIPAndNetwork()) more flexible. When
DistinctIP is enabled, last_net will be calculated as it was before. But
when DistinctIP is _off_, last_net can be the same as address (IP and
port). That will effectively implement !DistinctIP because every
record will have a distinct last_net already.
As a side effect, this flexibility will allow us to change the rules
about last_net construction arbitrarily. We can do tests where last_net
is set to the source IP, or to a /30 prefix, or a /16 prefix, etc., and
be able to exercise the production logic without requiring a virtual
network bridge.
This change should be safe to make without any migration code, because
all known production satellite deployments use DistinctIP, and the
associated last_net values will not change for them. They will only
change for satellites with !DistinctIP, which are mostly test
deployments that can be recreated trivially. For those satellites which
are both permanent and !DistinctIP, node selection will suddenly start
acting as though DistinctIP is enabled, until the operator runs a single
SQL update "UPDATE nodes SET last_net = last_ip_port". That can be done
either before or after deploying software with this change.
I also assert that this will not hurt performance for production
deployments. It's true that adding the distinct requirement to node
selection makes things a little slower, but the distinct requirement is
already present for all production deployments, and they will see no
change.
Refs: https://github.com/storj/storj/issues/5391
Change-Id: I0e7e92498c3da768df5b4d5fb213dcd2d4862924
82 lines
3.9 KiB
Go
82 lines
3.9 KiB
Go
// Copyright (C) 2019 Storj Labs, Inc.
|
|
// See LICENSE for copying information.
|
|
|
|
package overlay
|
|
|
|
import (
|
|
"time"
|
|
|
|
"github.com/spacemonkeygo/monkit/v3"
|
|
"github.com/zeebo/errs"
|
|
|
|
"storj.io/common/memory"
|
|
)
|
|
|
|
var (
|
|
mon = monkit.Package()
|
|
// Error represents an overlay error.
|
|
Error = errs.Class("overlay")
|
|
)
|
|
|
|
// Config is a configuration for overlay service.
|
|
type Config struct {
|
|
Node NodeSelectionConfig
|
|
NodeSelectionCache UploadSelectionCacheConfig
|
|
GeoIP GeoIPConfig
|
|
UpdateStatsBatchSize int `help:"number of update requests to process per transaction" default:"100"`
|
|
NodeCheckInWaitPeriod time.Duration `help:"the amount of time to wait before accepting a redundant check-in from a node (unmodified info since last check-in)" default:"2h" testDefault:"30s"`
|
|
NodeSoftwareUpdateEmailCooldown time.Duration `help:"the amount of time to wait between sending Node Software Update emails" default:"168h"`
|
|
RepairExcludedCountryCodes []string `help:"list of country codes to exclude nodes from target repair selection" default:"" testDefault:"FR,BE"`
|
|
SendNodeEmails bool `help:"whether to send emails to nodes" default:"false"`
|
|
MinimumNewNodeIDDifficulty int `help:"the minimum node id difficulty required for new nodes. existing nodes remain allowed" devDefault:"0" releaseDefault:"36"`
|
|
}
|
|
|
|
// AsOfSystemTimeConfig is a configuration struct to enable 'AS OF SYSTEM TIME' for CRDB queries.
|
|
type AsOfSystemTimeConfig struct {
|
|
Enabled bool `help:"enables the use of the AS OF SYSTEM TIME feature in CRDB" default:"true"`
|
|
DefaultInterval time.Duration `help:"default duration for AS OF SYSTEM TIME" devDefault:"-1ms" releaseDefault:"-10s" testDefault:"-1µs"`
|
|
}
|
|
|
|
// NodeSelectionConfig is a configuration struct to determine the minimum
|
|
// values for nodes to select.
|
|
type NodeSelectionConfig struct {
|
|
NewNodeFraction float64 `help:"the fraction of new nodes allowed per request" releaseDefault:"0.05" devDefault:"1"`
|
|
MinimumVersion string `help:"the minimum node software version for node selection queries" default:""`
|
|
OnlineWindow time.Duration `help:"the amount of time without seeing a node before its considered offline" default:"4h" testDefault:"1m"`
|
|
DistinctIP bool `help:"require distinct IPs when choosing nodes for upload" releaseDefault:"true" devDefault:"false"`
|
|
NetworkPrefixIPv4 int `help:"the prefix to use in determining 'network' for IPv4 addresses" default:"24" hidden:"true"`
|
|
NetworkPrefixIPv6 int `help:"the prefix to use in determining 'network' for IPv6 addresses" default:"64" hidden:"true"`
|
|
MinimumDiskSpace memory.Size `help:"how much disk space a node at minimum must have to be selected for upload" default:"500.00MB" testDefault:"100.00MB"`
|
|
|
|
AsOfSystemTime AsOfSystemTimeConfig
|
|
|
|
UploadExcludedCountryCodes []string `help:"list of country codes to exclude from node selection for uploads" default:"" testDefault:"FR,BE"`
|
|
}
|
|
|
|
// GeoIPConfig is a configuration struct that helps configure the GeoIP lookup features on the satellite.
|
|
type GeoIPConfig struct {
|
|
DB string `help:"the location of the maxmind database containing geoip country information"`
|
|
MockCountries []string `help:"a mock list of countries the satellite will attribute to nodes (useful for testing)"`
|
|
}
|
|
|
|
func (aost *AsOfSystemTimeConfig) isValid() error {
|
|
if aost.Enabled {
|
|
if aost.DefaultInterval >= 0 {
|
|
return errs.New("AS OF SYSTEM TIME interval must be a negative number")
|
|
}
|
|
if aost.DefaultInterval > -time.Microsecond {
|
|
return errs.New("AS OF SYSTEM TIME interval cannot be in nanoseconds")
|
|
}
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// Interval returns the configured interval respecting Enabled property.
|
|
func (aost *AsOfSystemTimeConfig) Interval() time.Duration {
|
|
if !aost.Enabled {
|
|
return 0
|
|
}
|
|
return aost.DefaultInterval
|
|
}
|