Commit Graph

9 Commits

Author SHA1 Message Date
Andrew Harding
b562cbf98f satellite/metrics: provide a rangedloop observer
https://github.com/storj/storj/issues/5236

Change-Id: Ic1ed7a5533dccacd58285b64579dbdd6210de4f9
2022-12-09 12:04:39 -07:00
Andrew Harding
633ab8dcf6 satellite/metadabase/rangedloop: stream affinity for test provider
Some observers assume that they will observe all the segments for a
given stream, and that they will observe those segments in a sequential
stream over one or more iterations.

This change updates the range provider from rangedlooptest to provide
these guarantees.

The change also removes the Mock suffix from the provider/splitter types
since the package name (rangedlooptest) implies that the type is a test
double.

Change-Id: I927c409807e305787abcde57427baac22f663eaa
2022-12-09 16:49:02 +00:00
Erik van Velzen
3cf7ebfad0
satellite/metabase/rangedloop: database abstraction (#5337)
Add an abstraction rangedloop.SegmentProvider to fetch chunks of
segments from the metainfo database in parallel.

Part of https://github.com/storj/storj/issues/5223

Change-Id: Ife26467ea0c3be550bde0b05464ef1db62dd4d2a
2022-12-09 03:01:07 +01:00
Erik van Velzen
ff6d640fca
satellite/metabase/rangedloop: minimal loop (#5334)
Minimal implementation of the ranged (=threaded) segment loop
service, to improve performance over the existing loop.

Has tests with a an inmemory segment database
and example observer.

Does not have yet: database link, observer duration tracking,
suspicious processed ratio guard, rate limiting, minimum execution
interval per observer, etc.

Part of https://github.com/storj/storj/issues/5223

Change-Id: I08ffb392c3539e380f4e7b4f1afd56c4c394668d
2022-12-08 15:27:21 +01:00
Andrew Harding
c6e48fb71d satellite/metabase/rangedloop: clarify observer docs
Change-Id: I171d39fd069186c2c275aab3a5e672427b34e38f
2022-12-07 11:27:35 +00:00
Erik van Velzen
b574ee5e6d satellite/metabase/rangedloop: service skeleton
Create skeleton for multi-threaded segment loop, peer, cmd command for rangedloop.

Change-Id: I52c78a313f15070d43207c52ea94e53169821654
2022-11-22 15:21:41 +02:00
Erik van Velzen
9fb18a43d8 satellite/metabase/rangedloop: observer interface
New interface for parallel segment loop.

Closes https://github.com/storj/storj/issues/5239

Change-Id: I2bcce6f836f6625da8ceb4fc0fc030c0ea4380e7
2022-11-17 20:12:23 +00:00
Erik van Velzen
337b72f310 satellite/metabase/rangedloop: uuid range pairs
Pair uuid's to create ranges. Will be used to parallelize the segment
loop.

Part of https://github.com/storj/storj/issues/5223

Change-Id: I73e2fb8a2cd379b840864449b6251b48feeb7b66
2022-11-03 11:15:56 +00:00
Erik van Velzen
c25391e976 satellite/metabase/rangedloop: uuid generation
Create helper function to generate ranges of UUIDs, for parallelizing
the segment loop.

Change-Id: I17dbc1d5effe27fc1a3491aa9ca56c692bd95df0
2022-10-31 16:05:41 +01:00