Go to file
Michal Niewrzal 98562d06c8 satellite/gc/bloomfilter: add sync observer
Current observer used with ranged loop is using massive amount of
memory because each range is generating separate set of bloom filters.
Each bloom filter can be up to 2MB of memory. That's a lot.

This change is initial change to reduce used memory by sharing bloom
filters between ranges and just synchronize access to them. This
implementation is rather simple and even naive but maybe it will be
enough without doing something more complex.

https://github.com/storj/storj/issues/5803

Change-Id: Ie62d19276aa9023076b1c97f712b788bce963cbe
2023-04-28 07:40:56 +00:00
.github .github: remove invalid codeowners 2022-11-10 15:52:48 +02:00
certificate private/kvstore: move storage package 2023-04-06 17:26:29 +03:00
cmd satellite/main,stripe/{client,service}: stripe balance invoice item cmd 2023-04-25 21:08:23 +00:00
crashcollect crashcollect: removed redundant structure 2021-04-28 00:35:39 +03:00
docs docs: add test plan for ranged loop (#5629) 2023-04-25 14:10:13 +02:00
installer/windows storj/storj: more domain changes 2021-04-15 20:51:43 +00:00
multinode all: fix math/rand deprecations 2023-02-17 15:05:54 +02:00
private satellite/audit: remove segments loop parts 2023-04-24 15:52:11 +00:00
resources cmd: add ca-certificates to Docker images (#3986) 2020-12-08 01:38:33 +01:00
satellite satellite/gc/bloomfilter: add sync observer 2023-04-28 07:40:56 +00:00
scripts satellite/gc/bloomfilter: add sync observer 2023-04-28 07:40:56 +00:00
storagenode satellite/gracefulexit: remove segments loop parts 2023-04-24 15:00:26 +00:00
testsuite go.mod: bump monkit 2023-04-25 14:26:11 +00:00
versioncontrol all: fix math/rand deprecations 2023-02-17 15:05:54 +02:00
web web/satellite: rework delete access grant modal 2023-04-27 21:26:38 +00:00
.dockerignore Forward-port release-alpha8 build script issues (#1726) 2019-04-09 23:01:10 -06:00
.earthlyignore build: provides earthfile for nightly build 2022-10-27 09:25:17 +00:00
.gitattributes web/: add check for change to eslint import 2021-12-21 15:59:23 +00:00
.gitignore gitignore: add go workspace files 2022-12-13 10:15:53 -07:00
.gitreview add config file for git review usage 2021-10-14 18:01:30 +00:00
CODE_OF_CONDUCT.md Adding CODE_OF_CONDUCT to storj/storj repo (#779) 2018-12-07 15:10:02 -05:00
CODEOWNERS update CODEOWNERS 2023-02-02 10:01:50 +00:00
CONTRIBUTING.md go.mod: update to minimum supported go version (#4239) 2021-10-22 21:12:13 +02:00
DEVELOPING.md Makefile: run lint locally in docker 2022-05-25 12:30:15 -05:00
docker-compose.tests.yaml Makefile: disable postgres fsync in the test container 2022-12-01 22:03:31 +00:00
Earthfile earthfile: use latest storj-up base container for ad-hoc containers 2023-02-08 12:23:27 +00:00
go.mod go.mod: bump monkit 2023-04-25 14:26:11 +00:00
go.sum go.mod: bump monkit 2023-04-25 14:26:11 +00:00
Jenkinsfile ci: bump release to Go 1.20.3 2023-04-12 19:06:26 +03:00
Jenkinsfile.premerge ci: move go-licenses check to premerge 2023-03-20 09:20:28 +00:00
Jenkinsfile.public private/kvstore: move storage package 2023-04-06 17:26:29 +03:00
Jenkinsfile.verify testsuite/ui: bump gateway-mt and enable vet 2023-04-17 22:07:57 +00:00
LICENSE license code with agplv3 (#126) 2018-07-05 10:24:26 -04:00
MAINTAINERS.md Maintainers: remove link 2022-03-14 14:16:31 +02:00
Makefile ci: bump release to Go 1.20.3 2023-04-12 19:06:26 +03:00
monkit.lock satellite/repair: repair pieces on the same last_net 2023-04-06 17:34:25 +00:00
proto.lock certificate/certificatepb: add definitions for migration 2023-01-25 10:28:36 +02:00
README.md Update README.md (#4320) 2021-12-22 14:12:58 +01:00

Storj V3 Network

Go Report Card Go Doc Coverage Status

Storj is building a decentralized cloud storage network. Check out our white paper for more info!


Storj is an S3-compatible platform and suite of decentralized applications that allows you to store data in a secure and decentralized manner. Your files are encrypted, broken into little pieces and stored in a global decentralized network of computers. Luckily, we also support allowing you (and only you) to retrieve those files!

Table of Contents

Contributing to Storj

All of our code for Storj v3 is open source. If anything feels off, or if you feel that some functionality is missing, please check out the contributing page. There you will find instructions for sharing your feedback, building the tool locally, and submitting pull requests to the project.

A Note about Versioning

While we are practicing semantic versioning for our client libraries such as uplink, we are not practicing semantic versioning in this repo, as we do not intend for it to be used via Go modules. We may have backwards-incompatible changes between minor and patch releases in this repo.

Start using Storj

Our wiki has documentation and tutorials. Check out these three tutorials:

License

This repository is currently licensed with the AGPLv3 license.

For code released under the AGPLv3, we request that contributors sign our Contributor License Agreement (CLA) so that we can relicense the code under Apache v2, or other licenses in the future.

Support

If you have any questions or suggestions please reach out to us on our community forum or file a ticket at https://support.storj.io/.