1cbad0fcab
for very large machines (>10Gbit) it is still useful
to have parallelism for uploads because we're actually
bound by getting new pieces from the satellite, so doing
that in parallel provides a big win.
this change adds back that flag to exist for uploads, and
removes the backwards compatibility code for the flag with
the maximum-concurrent-pieces as they are now independent.
the upload code parallelism story is now this:
- each object is a transfer
- each transfer happens in N parts (size dynamically
chosen to avoid having >10000 parts)
- each part can happen in parallel up to the limit
specified
- each parallel part can have up to the limit of
max concurrent pieces and segments
this change also changes some defaults to be better.
- the connection pool capacity now takes into acount
transfers, parallelism and max concurrent pieces
- the default smallest part size is 1GiB to allow the
new upload code path to upload multiple segments
Change-Id: Iff6709ae73425fbc2858ed360faa2d3ece297c2d
|
||
|---|---|---|
| .github | ||
| certificate | ||
| cmd | ||
| crashcollect | ||
| docs | ||
| installer/windows | ||
| multinode | ||
| private | ||
| resources | ||
| satellite | ||
| scripts | ||
| storagenode | ||
| testsuite | ||
| versioncontrol | ||
| web | ||
| .dockerignore | ||
| .earthlyignore | ||
| .gitattributes | ||
| .gitignore | ||
| .gitreview | ||
| CODE_OF_CONDUCT.md | ||
| CODEOWNERS | ||
| CONTRIBUTING.md | ||
| DEVELOPING.md | ||
| docker-compose.tests.yaml | ||
| Earthfile | ||
| go.mod | ||
| go.sum | ||
| Jenkinsfile | ||
| Jenkinsfile.premerge | ||
| Jenkinsfile.public | ||
| Jenkinsfile.verify | ||
| LICENSE | ||
| MAINTAINERS.md | ||
| Makefile | ||
| monkit.lock | ||
| proto.lock | ||
| README.md | ||
Storj V3 Network
Storj is building a distributed cloud storage network. Check out our white paper for more info!
Storj is an S3-compatible platform and suite of distributed applications that allows you to store data in a secure and distributed manner. Your files are encrypted, broken into little pieces and stored in a global distributed network of computers. Luckily, we also support allowing you (and only you) to retrieve those files!
Table of Contents
Contributing to Storj
All of our code for Storj v3 is open source. If anything feels off, or if you feel that some functionality is missing, please check out the contributing page. There you will find instructions for sharing your feedback, building the tool locally, and submitting pull requests to the project.
A Note about Versioning
While we are practicing semantic versioning for our client libraries such as uplink, we are not practicing semantic versioning in this repo, as we do not intend for it to be used via Go modules. We may have backwards-incompatible changes between minor and patch releases in this repo.
Start using Storj
Our wiki has documentation and tutorials. Check out these three tutorials:
License
This repository is currently licensed with the AGPLv3 license.
For code released under the AGPLv3, we request that contributors sign our Contributor License Agreement (CLA) so that we can relicense the code under Apache v2, or other licenses in the future.
Support
If you have any questions or suggestions please reach out to us on our community forum or file a ticket at https://support.storj.io/.