Commit Graph

7 Commits

Author SHA1 Message Date
Michal Niewrzal
2592aaef9c satellite/gc/bloomfilter: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I1a22ac4b242998e287b2b7d8167b64e850b61a0f
2023-05-15 11:46:26 +00:00
Erik van Velzen
464ceb1c0e satellite/gc: improve comments
Change-Id: I9e71c9bee3447f78365ba1593e4a4ef55b28356f
2023-03-08 13:15:13 +00:00
Michal Niewrzal
75b77d53ff satellite/gc/sender: avoid sending BF to disqualified and exited nodes
We don't want to waste our time on disqualified and exited nodes.

Change-Id: I11709350ad291c24f3b46670dd6a418c0ddbb44f
2022-11-29 09:56:32 +00:00
Ethan
9a09d8920e satellite/gc: Upload bloomfilters with prefix and update LATEST when complete
Change he bloomfilter generation process to prefix the objects with a date and update the LATEST object with the prefix.  The sender will read the LATEST file to get the prefix to process.

Change-Id: Iae0d3c49015d57f391d87789fb799a7d774066bf
2022-11-01 21:24:46 +00:00
Erik van Velzen
d2a67fb8f7 satellite/gc/sender: concurrency
Restore functionality where retain filters can be sent out to multiple
storage nodes simultaneously.

Fixes https://github.com/storj/team-metainfo/issues/121

Change-Id: I2bf86a166b09c6a277c1cb455cdca0165ce6b8af
2022-09-27 08:10:10 +00:00
Michal Niewrzal
c3ca98f552 satellite/gc/bloomfilter: minor cleanups
* service was running wrong RunOnce method
* after doing integration with GC sender we concluded that we don't
need special flag "gc-sender" to be uploaded as its safe to consume
partial results by GC senger. This part was removed.
* prefix format for moving data after error was unified with GC
sender

https: //github.com/storj/team-metainfo/issues/120
Change-Id: I204b696b9c2def4874ad1d17d0e84231cc98d583
2022-09-20 18:29:00 +02:00
Erik van Velzen
e6b5501f9b satellite/gc/sender: new service to send retain filters
Implement a new service to read retain filter from a bucket and
send them out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Paralellism (setting ConcurrentSends) and end-to-end garbage collection
tests will be restored in a subsequent commit.

Solves https://github.com/storj/team-metainfo/issues/121

Change-Id: Iaf8a33fbf6987676cc3cf74a18a8078916fe673d
2022-09-20 11:49:40 +00:00