Commit Graph

6 Commits

Author SHA1 Message Date
Michal Niewrzal
94d341bcf3 satellite: use ranged loop with GC-GF peer
Peer for generating bloom filters will be able to use ranged loop.

As an addition some cleanup were made:
* remove unused parts of GC BF peer (identity, version control)
* added missing Close method for ranged loop service
* some additional tests added

https://github.com/storj/storj/issues/5545

Change-Id: I9a3d85f5fffd2ebc7f2bf7ed024220117ab2be29
2023-02-13 18:32:21 +00:00
Erik van Velzen
ed910b6087
satellite/metabase/rangedloop: continue after error (#5430)
When an observer errors we still want to finish the other observers.

This changes store the error and continues the loop, skipping
the observer which errored out and setting the duration metric to -1.

When the error occurs in the process stage, it does continue the other
ranges of the same observer. It removes the observer entirely after the process
stage. To improve this would make it more complex due to race
conditions.

Closes https://github.com/storj/storj/issues/5389

Change-Id: I528432c491d4340817d6950f1200ee2b9e703309
2023-01-11 22:23:17 +01:00
Erik van Velzen
1d4411f166
satellite/metabase/rangedloop: cancellation (#5364)
Support interruption of the ranged segment loop through context.

Part of https://github.com/storj/storj/issues/5223

Change-Id: Iae0260e250f8ea33affed95c6592a1f42df384eb
2023-01-05 16:32:30 +01:00
Erik van Velzen
37b4981cc0
satellite/metabase/rangedloop: measure observer duration (#5350)
Track duration of all segment loop observers. Factor out functions to
reduce size.

Still need to send the measurements out via monkit.

Part of https://github.com/storj/storj/issues/5223

Change-Id: Iae0260e250f8ea33affed95c6592a1f42df384eb
2022-12-21 21:58:08 +01:00
Andrew Harding
633ab8dcf6 satellite/metadabase/rangedloop: stream affinity for test provider
Some observers assume that they will observe all the segments for a
given stream, and that they will observe those segments in a sequential
stream over one or more iterations.

This change updates the range provider from rangedlooptest to provide
these guarantees.

The change also removes the Mock suffix from the provider/splitter types
since the package name (rangedlooptest) implies that the type is a test
double.

Change-Id: I927c409807e305787abcde57427baac22f663eaa
2022-12-09 16:49:02 +00:00
Erik van Velzen
ff6d640fca
satellite/metabase/rangedloop: minimal loop (#5334)
Minimal implementation of the ranged (=threaded) segment loop
service, to improve performance over the existing loop.

Has tests with a an inmemory segment database
and example observer.

Does not have yet: database link, observer duration tracking,
suspicious processed ratio guard, rate limiting, minimum execution
interval per observer, etc.

Part of https://github.com/storj/storj/issues/5223

Change-Id: I08ffb392c3539e380f4e7b4f1afd56c4c394668d
2022-12-08 15:27:21 +01:00