storj/satellite
Jeff Wendling 7b06575f6f satellite/meta{base,info}: reduce db round trips for download
This combines the ListStreamPositions and GetSegmentByPosition
calls with a ListSegments call that now knows how to return
only the segments within a Range, just like ListStreamPositions.

It would theoretically be possible to also include the
GetObjectLastCommitted call by having it do one of three
queries based on the incoming request range, but that would
mean duplicating the data for the object in every single
row that is returned for each segment in the range.

One gross thing that ListSegments has to do now is update the
first segment returned with the information from any ancestor
segment because GetSegmentByPosition used to do that. It only
updates the first segment so that it doesn't do O(N) database
queries. It seems difficult to have it do a single query to
update all of the segments at once. I'm not certain this change
should be merged on this basis alone.

This change has made me think a couple of things should happen:

1. Server side copy with ancestor segments strikes again
   making the code less clear and potentially more buggy
   or inefficient for a rare case (empirically <0.1%)

2. The download code requests individual segments from
   the satellite lazily as part of its download which
   requires the satellite telling it the locations of
   all of the segments which requires the satellite
   querying the locations of all of the segments. Instead
   the download RPC could return the orders for all of
   the segments for a range and the download code could
   issue N download calls rather than 1 download call and
   N get segment calls. I believe both sides of the code
   paths would be simpler and more efficient this way.

3. In looking at the timing information for downloads when
   testing this, we really need to focus on getting the
   auth key and bandwidth limit verification times down.
   Here's the timing I saw:

       - 42ms: validate auth
       - 52ms: bandwidth usage checking
       - 14ms: get object info
       - 26ms: get segment position info
       - 26ms: getting the first segment full info
       - 20ms: unaccounted for by spans
       - 6ms:  creating the orders

   This change will remove 26ms, but there's a good 90ms
   in just validation. With improved semantics hitting the
   database only once and improved validation, a download
   rpc taking ~30ms seems doable compared to our current
   ~200ms.

Change-Id: I4109dba082eaedb79e634c61dbf86efa93ab1222
2023-04-03 16:49:00 +00:00
..
abtesting console/abTesting: add support for AB testing 2022-10-27 10:57:12 +00:00
accounting satellite/accounting: update GetProjectSegmentLimit to fix bug 2023-03-31 11:47:21 +00:00
admin satellite/admin: Fix response code for Get bucket 2023-04-03 10:22:22 +02:00
analytics satellite/{console,db,analytics}: better warning handling 2023-03-28 14:30:38 +00:00
attribution satellite: remove rewards package 2023-01-31 11:46:50 +00:00
audit satellite/audit: fix go1.19 dial timeouts and log more 2023-02-28 17:09:47 +00:00
buckets satellite/accounting/tally: fix looping over all buckets 2022-12-07 10:32:23 +00:00
compensation all: reformat comments as required by gofmt 1.19 2022-08-10 18:24:55 +00:00
console satellite/console: purchase package grants credit 2023-03-31 22:29:54 +00:00
contact satellite/{contact,satellitedb}: preserve node message debounce support 2023-03-20 16:13:06 +00:00
gc satellite/gc: improve comments 2023-03-08 13:15:13 +00:00
geoip satellite/overlay: update country code on every node check-in 2023-03-22 08:38:51 +00:00
gracefulexit satellite/metabase: add piece size calculation to segment 2023-02-22 11:04:02 +00:00
inspector {satellite/metabase, satellite/metainfo, satellite/inspector} : Use metabase.GetObjectLastCommitted instead metabase.GetObjectExactVersion 2022-09-08 07:27:22 +00:00
internalpb all: fix deprecated ioutil commands 2022-10-11 15:27:29 +00:00
mailservice satellite/consoleweb: fix flaky TestAuth tests 2022-08-04 19:06:07 +00:00
metabase satellite/meta{base,info}: reduce db round trips for download 2023-04-03 16:49:00 +00:00
metainfo satellite/meta{base,info}: reduce db round trips for download 2023-04-03 16:49:00 +00:00
metrics satellite/metabase/rangedloop: measure observer duration (#5350) 2022-12-21 21:58:08 +01:00
nodeapiversion satellite/nodeapiversion: new table for tracking node api usage 2020-07-09 15:02:25 +00:00
nodeevents satellite/satellitedb/dbx: document node tables 2023-02-24 14:05:55 +00:00
nodeselection/uploadselection satellite/overlay: configurable meaning of last_net 2023-03-09 02:20:12 +00:00
nodestats satellite: return interval_end_time in DailyStorageUsage endpoint 2022-07-27 18:24:27 +00:00
oidc all: fix deprecated ioutil commands 2022-10-11 15:27:29 +00:00
orders satellite/orders: cleanup after altering primary key 2023-03-06 16:03:11 +00:00
overlay satellite/overlay: update country code on every node check-in 2023-03-22 08:38:51 +00:00
payments satellite/console: purchase package grants credit 2023-03-31 22:29:54 +00:00
repair satellite/overlay: remove unused mail service from overlay 2023-03-09 12:17:35 +00:00
reputation satellite/reputation: if node is DQd or exited skip applying audit 2023-02-21 18:31:21 +00:00
revocation satellite/satellitedb: move tests to their domains 2021-02-19 17:29:15 +02:00
satellitedb satellite/satellitedb: add package info to customers db 2023-03-31 15:51:30 +00:00
snopayouts all: fix error naming 2021-04-29 15:38:21 +03:00
admin.go satellite/{console,analytics,accountfreeze}: send unfreeze event 2023-03-10 23:21:20 +00:00
api.go satellite/metainfo: use project limit cache with limiter 2023-03-14 08:11:11 +00:00
auditor.go satellite: cleanup orders dependencies 2023-03-09 13:34:21 +00:00
configlock_test.go all: fix deprecated ioutil commands 2022-10-11 15:27:29 +00:00
core.go satellite/accounting: Use metabase.AliasPiece with tally observer 2023-03-29 12:24:05 +00:00
gc-bf_test.go satellite: use ranged loop with GC-GF peer 2023-02-13 18:32:21 +00:00
gc-bf.go satellite: use ranged loop with GC-GF peer 2023-02-13 18:32:21 +00:00
gc.go gc: remove segment loop dependency 2023-02-03 10:31:16 +00:00
peer.go satellite/peer: better error handling for SMTP parameters 2023-03-09 11:29:07 +00:00
rangedloop.go satellite/accounting: Use metabase.AliasPiece with tally observer 2023-03-29 12:24:05 +00:00
repairer.go satellite: cleanup orders dependencies 2023-03-09 13:34:21 +00:00