e19e3c1101
Now that we are trying to identify the root cause of the satellite load limitations (i.e. currently the satellite has a max ability of 400 rps for uploads and we need this to be higher), we are using the golang diagnostic tools to collect insight into what the bottlenecks are. We currently have a debug endpoint to gather some cpu and mem data, but it could be useful to have continuous profiling. GCP stackdriver has support for continuous profiling so lets set that up and see if it is helpful to gather more data. This PR adds support for [GCP continuous profiler](https://cloud.google.com/profiler) which allows enabling continuous cpu/mem profiling and the stats are sent to stackdriver in google cloud console. To enable the continuous profiling for a storj component, do the following: - prereq: the workload must be running in GKE and have Stackdriver Profiling IAM role permissions - provide the config flag `debug.profilename` in the config.yaml file for the workload (i.e. satellite api process, etc). The profilename should be the workload name, for example "satellite-api". - once the above config flag is provided, the profiler will be initialized and profiling stats will automatically be sent to GCP project where the workload is running and viewable in the Stackdriver Profile page in the console The current implementation assumes the workload is running in GKE, however if we find if useful we can add support to enable this from anywhere. But for simplicity, its configured this way assuming the main goal is to enable in production systems. Change-Id: Ibf8ebe2df7bf06fdd4951ee6a1e48854dd36ad47
33 lines
696 B
Go
33 lines
696 B
Go
// Copyright (C) 2019 Storj Labs, Inc.
|
|
// See LICENSE for copying information.
|
|
|
|
package process
|
|
|
|
import (
|
|
"flag"
|
|
|
|
"cloud.google.com/go/profiler"
|
|
"github.com/zeebo/errs"
|
|
"go.uber.org/zap"
|
|
)
|
|
|
|
var (
|
|
initProfilerError = errs.Class("initializing profiler")
|
|
|
|
debugProfilerName = flag.String("debug.profilername", "", "provide the name of the peer to enable continuous cpu/mem profiling for")
|
|
)
|
|
|
|
func initProfiler(log *zap.Logger) error {
|
|
name := *debugProfilerName
|
|
if name != "" {
|
|
if err := profiler.Start(profiler.Config{
|
|
Service: name,
|
|
ServiceVersion: "",
|
|
}); err != nil {
|
|
return initProfilerError.Wrap(err)
|
|
}
|
|
log.Debug("success debug profiler init")
|
|
}
|
|
return nil
|
|
}
|