storj/docs/blueprints/uplink-telemetry.md

Uplink Telemetry

Abstract

Our telemetry uses monkit to monitor various functions and other tasks. Currently we do not collect any data from uplinks. This design doc proposes how to collect telemetry data from uplinks.

Background

Uplinks currently do not send any telemetry data. As we move into production we want to collect this data from uplinks so we can monitor the aggregate health of uplinks, include their data in distributed tracing applications, and generally gain insight into the entire system end-to-end.

Uplinks present a challenge for collecting this data for a number of reasons:

1. Many uplink operations are short in duration, so a chore that periodically sends to the collector may not execute, or may miss many metrics.
2. We cannot control the configuration of uplinks that are simply using the library code, so if we rely on configuration to control where and when metrics are sent we will miss many metrics.

Design

Problem 1: When to send uplink telemetry data

We will tie the flushing of metrics to the uplink library's OpenProject and Project.Close calls.

• OpenProject will start a periodic flush so that long-running and long-used projects will periodically flush metrics. It is anticipated that most uplink use-cases will not hold open projects long enough for this flush to occur. But if an uplink opens a project for a long duration to perform many operations, this periodic flush will ensure a steady flow of metrics. This will likely mean we store a loop on the project which will be stopped during project close.
• Project.Close will flush metrics to the collector and stop the periodic flushing mentioned above. The final flush will need to use the background context, and will need to be a blocking call to ensure the metrics are actually sent.

Problem 2: How to configure uplinks to send telemetry data

We will hard-code the URL for our default collector, only to be used for release builds. If users do not want to send telemetry information they can override this setting with an empty string.

Rationale

The advantages of this approach is that each uplink call will flush telemetry data, regardless of whether we control the binary using the uplink code.

Disadvantages include that we need to hard-code the URL of where to send the data. An alternate approach include:

• Have a service discovery feature on each satellite which returns the URL of where to send telemetry information. This allows other satellite operators to use uplink libarary without accidentally sending the data to us, but is more complicated.
• Send key metrics directly to a satellite endpoint after specific operations. This prevents the need for an additional URL for uplink telemetry, and makes it easy to track data per satellite. The disadvantage is this requires additional round-trips for operations.

Implementation

Implementation should be simple, simply add the logic described above to the uplink project.