JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

satellite/gc/bloomfilter: reduce number of filters in single zip

Browse Source 
Current value is too big and can cause memory issues. With 500 bloom
filters where single filter can go up to 2MB we can end up with 1GB zip
which is more problematic if we don't want to download it before
processing.

Change-Id: I4b983afbcac39c718afbbeb764a30d32c4e80c08
This commit is contained in:
Michal Niewrzal 2023-12-04 12:22:29 +01:00 committed by Storj Robot
parent 5d775e61b1
commit a3f84bcb0c
2 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
satellite/gc/bloomfilter/config.go
View File

@ -17,6 +17,6 @@ type Config struct {
AccessGrant string `help:"Access Grant which will be used to upload bloom filters to the bucket" default:""`
Bucket string `help:"Bucket which will be used to upload bloom filters" default:"" testDefault:"gc-queue"` // TODO do we need full location?
ZipBatchSize int `help:"how many bloom filters will be packed in a single zip" default:"500" testDefault:"2"`
ZipBatchSize int `help:"how many bloom filters will be packed in a single zip" default:"40" testDefault:"2"`
ExpireIn time.Duration `help:"how long bloom filters will remain in the bucket for gc/sender to consume before being automatically deleted" default:"336h"`
}

2
scripts/testdata/satellite-config.yaml.lock vendored
View File

@ -566,7 +566,7 @@ contact.external-address: ""
# garbage-collection-bf.use-sync-observer: true
# how many bloom filters will be packed in a single zip
# garbage-collection-bf.zip-batch-size: 500
# garbage-collection-bf.zip-batch-size: 40
# Access to download the bloom filters. Needs read and write permission.
# garbage-collection.access-grant: ""