Go to file
paul cannon 2522ff09b6 satellite/overlay: configurable meaning of last_net
Up to now, we have been implementing the DistinctIP preference with code
in two places:

 1. On check-in, the last_net is determined by taking the /24 or /64
    (in ResolveIPAndNetwork()) and we store it with the node record.
 2. On node selection, a preference parameter defines whether to return
    results that are distinct on last_net.

It can be observed that we have never yet had the need to switch from
DistinctIP to !DistinctIP, or from !DistinctIP to DistinctIP, on the
same satellite, and we will probably never need to do so in an automated
way. It can also be observed that this arrangement makes tests more
complicated, because we often have to arrange for test nodes to have IP
addresses in different /24 networks (a particular pain on macOS).

Those two considerations, plus some pending work on the repair framework
that will make repair take last_net into consideration, motivate this
change.

With this change, in the #2 place, we will _always_ return results that
are distinct on last_net. We implement the DistinctIP preference, then,
by making the #1 place (ResolveIPAndNetwork()) more flexible. When
DistinctIP is enabled, last_net will be calculated as it was before. But
when DistinctIP is _off_, last_net can be the same as address (IP and
port). That will effectively implement !DistinctIP because every
record will have a distinct last_net already.

As a side effect, this flexibility will allow us to change the rules
about last_net construction arbitrarily. We can do tests where last_net
is set to the source IP, or to a /30 prefix, or a /16 prefix, etc., and
be able to exercise the production logic without requiring a virtual
network bridge.

This change should be safe to make without any migration code, because
all known production satellite deployments use DistinctIP, and the
associated last_net values will not change for them. They will only
change for satellites with !DistinctIP, which are mostly test
deployments that can be recreated trivially. For those satellites which
are both permanent and !DistinctIP, node selection will suddenly start
acting as though DistinctIP is enabled, until the operator runs a single
SQL update "UPDATE nodes SET last_net = last_ip_port". That can be done
either before or after deploying software with this change.

I also assert that this will not hurt performance for production
deployments. It's true that adding the distinct requirement to node
selection makes things a little slower, but the distinct requirement is
already present for all production deployments, and they will see no
change.

Refs: https://github.com/storj/storj/issues/5391
Change-Id: I0e7e92498c3da768df5b4d5fb213dcd2d4862924
2023-03-09 02:20:12 +00:00
.github .github: remove invalid codeowners 2022-11-10 15:52:48 +02:00
certificate certificate/authorization,cmd/certificates: remove gob code 2023-02-08 15:05:40 +00:00
cmd cmd/multinode: Removes dependency on deprecated identity-dir flag, code and documentation. (#5646) 2023-03-08 13:56:15 +01:00
crashcollect crashcollect: removed redundant structure 2021-04-28 00:35:39 +03:00
docs blueprint: tcp fastopen 2023-01-17 20:35:47 +00:00
installer/windows storj/storj: more domain changes 2021-04-15 20:51:43 +00:00
multinode all: fix math/rand deprecations 2023-02-17 15:05:54 +02:00
private satellite/overlay: configurable meaning of last_net 2023-03-09 02:20:12 +00:00
resources cmd: add ca-certificates to Docker images (#3986) 2020-12-08 01:38:33 +01:00
satellite satellite/overlay: configurable meaning of last_net 2023-03-09 02:20:12 +00:00
scripts satellite/overlay: configurable meaning of last_net 2023-03-09 02:20:12 +00:00
storage storage/filestore: fix panic on fs error in EmptyTrash 2023-02-14 18:09:19 -06:00
storagenode storagenode/piecestore: be more flexible with bandwidth usage max 2023-03-06 18:00:17 +00:00
testsuite satellite:{console, web}: remove old project dashboard 2023-03-07 13:34:59 +02:00
versioncontrol all: fix math/rand deprecations 2023-02-17 15:05:54 +02:00
web web/satellite: migrated StorageChart to use composition API 2023-03-07 15:20:36 +00:00
.dockerignore Forward-port release-alpha8 build script issues (#1726) 2019-04-09 23:01:10 -06:00
.earthlyignore build: provides earthfile for nightly build 2022-10-27 09:25:17 +00:00
.gitattributes web/: add check for change to eslint import 2021-12-21 15:59:23 +00:00
.gitignore gitignore: add go workspace files 2022-12-13 10:15:53 -07:00
.gitreview add config file for git review usage 2021-10-14 18:01:30 +00:00
CODE_OF_CONDUCT.md Adding CODE_OF_CONDUCT to storj/storj repo (#779) 2018-12-07 15:10:02 -05:00
CODEOWNERS update CODEOWNERS 2023-02-02 10:01:50 +00:00
CONTRIBUTING.md go.mod: update to minimum supported go version (#4239) 2021-10-22 21:12:13 +02:00
DEVELOPING.md Makefile: run lint locally in docker 2022-05-25 12:30:15 -05:00
docker-compose.tests.yaml Makefile: disable postgres fsync in the test container 2022-12-01 22:03:31 +00:00
Earthfile earthfile: use latest storj-up base container for ad-hoc containers 2023-02-08 12:23:27 +00:00
go.mod satellite/audit: fix go1.19 dial timeouts and log more 2023-02-28 17:09:47 +00:00
go.sum go.mod: bump storj.io/{common,drpc,uplink} 2023-02-24 23:17:16 +00:00
Jenkinsfile Jenkinsfile,Makefile: bump to go v1.19.6 2023-02-15 14:00:52 +00:00
Jenkinsfile.premerge mod: bump storj.io/common 2023-02-03 16:49:41 +02:00
Jenkinsfile.public mod: bump storj.io/common 2023-02-03 16:49:41 +02:00
Jenkinsfile.verify mod: bump storj.io/common 2023-02-03 16:49:41 +02:00
LICENSE license code with agplv3 (#126) 2018-07-05 10:24:26 -04:00
MAINTAINERS.md Maintainers: remove link 2022-03-14 14:16:31 +02:00
Makefile Jenkinsfile,Makefile: bump to go v1.19.6 2023-02-15 14:00:52 +00:00
monkit.lock satellite/audit: Begin using piecewise reverifications 2022-12-16 14:21:13 +00:00
proto.lock certificate/certificatepb: add definitions for migration 2023-01-25 10:28:36 +02:00
README.md Update README.md (#4320) 2021-12-22 14:12:58 +01:00

Storj V3 Network

Go Report Card Go Doc Coverage Status

Storj is building a decentralized cloud storage network. Check out our white paper for more info!


Storj is an S3-compatible platform and suite of decentralized applications that allows you to store data in a secure and decentralized manner. Your files are encrypted, broken into little pieces and stored in a global decentralized network of computers. Luckily, we also support allowing you (and only you) to retrieve those files!

Table of Contents

Contributing to Storj

All of our code for Storj v3 is open source. If anything feels off, or if you feel that some functionality is missing, please check out the contributing page. There you will find instructions for sharing your feedback, building the tool locally, and submitting pull requests to the project.

A Note about Versioning

While we are practicing semantic versioning for our client libraries such as uplink, we are not practicing semantic versioning in this repo, as we do not intend for it to be used via Go modules. We may have backwards-incompatible changes between minor and patch releases in this repo.

Start using Storj

Our wiki has documentation and tutorials. Check out these three tutorials:

License

This repository is currently licensed with the AGPLv3 license.

For code released under the AGPLv3, we request that contributors sign our Contributor License Agreement (CLA) so that we can relicense the code under Apache v2, or other licenses in the future.

Support

If you have any questions or suggestions please reach out to us on our community forum or file a ticket at https://support.storj.io/.