scx/scripts/process_runqlat.bt

78 lines
1.6 KiB
Plaintext
Raw Normal View History

scripts: Add script to measure runqlat for a process Add a `scripts` folder to hold scripts that are useful for the project, and add a bpftrace program to measure the runqueue latency for a process. bpftrace's built-in runqlat.bt program instruments runqueue latency for all processes and does not provide a way to filter by PID. For sched_ext performance work, we are interested in the runqueue latency of a specific process we are trying to optimize, such as a video game. Therefore, we need to create a custom bpftrace program to achieve this. `process_runqlat.bt` instruments runqueue latency for a PID and its threads. This script measures the runqueue latency for a specified PID and includes all threads spawned by that process. USAGE: sudo ./scripts/process_runqlat.bt <PID> The program output will include: - Stats by thread (count, avg latency, total latency) - A histogram of all latency measurements - Aggregated total stats (count, avg latency, total latency) Example output when targeting Terraria's main process: $ sudo ./scripts/process_runqlat.bt 652644 Attaching 5 probes... Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end. @tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67 @tasks[Finalizer, 652646]: count 24, average 6, total 151 @tasks[Main Thread, 652668]: count 1432, average 8, total 12561 @tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120 @tasks[Main Thread, 652667]: count 1037, average 9, total 10091 @tasks[FACT Thread, 652679]: count 1033, average 10, total 11005 @tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047 @tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210 @tasks[Main Thread, 652666]: count 104, average 10, total 1088 @tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461 @tasks[Main Thread, 652644]: count 5840, average 11, total 69177 @tasks[Main Thread, 694917]: count 44, average 14, total 659 @tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300 @tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873 @tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848 @tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836 @tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518 @usec_hist: [0] 16 | | [1] 360 |@ | [2, 4) 4650 |@@@@@@@@@@@@@@ | [4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 5056 |@@@@@@@@@@@@@@@ | [32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ | [128, 256) 1926 |@@@@@@ | [256, 512) 130 | | [512, 1K) 2 | | @usec_total_stats: count 57152, average 29, total 1699656 Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
#!/usr/bin/env bpftrace
/*
* Copyright (c) Jose Fernandez <josef@netflix.com>
*
* process_runqlat.bt - Instrument runqueue latency for a PID and its threads.
*
* This script measures the runqueue latency for a specified process ID (PID)
* and includes all threads spawned by that process.
*
* USAGE: sudo ./process_runqlat.bt <PID>
*
* The program output will include:
* - Stats by task (count, avg latency, total latency)
* - A histogram of all latency measurements
* - Aggregated total stats (count, avg latency, total latency)
*/
#include <linux/sched.h>
BEGIN
{
if ($1 == 0) {
printf("PID is missing, use `sudo ./process_runqlat.bt <pid>`\n");
exit();
}
printf("Instrumenting runqueue latency for PID %d. Hit Ctrl-C to end.\n", $1);
scripts: Add script to measure runqlat for a process Add a `scripts` folder to hold scripts that are useful for the project, and add a bpftrace program to measure the runqueue latency for a process. bpftrace's built-in runqlat.bt program instruments runqueue latency for all processes and does not provide a way to filter by PID. For sched_ext performance work, we are interested in the runqueue latency of a specific process we are trying to optimize, such as a video game. Therefore, we need to create a custom bpftrace program to achieve this. `process_runqlat.bt` instruments runqueue latency for a PID and its threads. This script measures the runqueue latency for a specified PID and includes all threads spawned by that process. USAGE: sudo ./scripts/process_runqlat.bt <PID> The program output will include: - Stats by thread (count, avg latency, total latency) - A histogram of all latency measurements - Aggregated total stats (count, avg latency, total latency) Example output when targeting Terraria's main process: $ sudo ./scripts/process_runqlat.bt 652644 Attaching 5 probes... Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end. @tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67 @tasks[Finalizer, 652646]: count 24, average 6, total 151 @tasks[Main Thread, 652668]: count 1432, average 8, total 12561 @tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120 @tasks[Main Thread, 652667]: count 1037, average 9, total 10091 @tasks[FACT Thread, 652679]: count 1033, average 10, total 11005 @tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047 @tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210 @tasks[Main Thread, 652666]: count 104, average 10, total 1088 @tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461 @tasks[Main Thread, 652644]: count 5840, average 11, total 69177 @tasks[Main Thread, 694917]: count 44, average 14, total 659 @tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300 @tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873 @tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848 @tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836 @tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518 @usec_hist: [0] 16 | | [1] 360 |@ | [2, 4) 4650 |@@@@@@@@@@@@@@ | [4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 5056 |@@@@@@@@@@@@@@@ | [32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ | [128, 256) 1926 |@@@@@@ | [256, 512) 130 | | [512, 1K) 2 | | @usec_total_stats: count 57152, average 29, total 1699656 Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
}
/*
* args:
* - struct task_struct *p
scripts: Add script to measure runqlat for a process Add a `scripts` folder to hold scripts that are useful for the project, and add a bpftrace program to measure the runqueue latency for a process. bpftrace's built-in runqlat.bt program instruments runqueue latency for all processes and does not provide a way to filter by PID. For sched_ext performance work, we are interested in the runqueue latency of a specific process we are trying to optimize, such as a video game. Therefore, we need to create a custom bpftrace program to achieve this. `process_runqlat.bt` instruments runqueue latency for a PID and its threads. This script measures the runqueue latency for a specified PID and includes all threads spawned by that process. USAGE: sudo ./scripts/process_runqlat.bt <PID> The program output will include: - Stats by thread (count, avg latency, total latency) - A histogram of all latency measurements - Aggregated total stats (count, avg latency, total latency) Example output when targeting Terraria's main process: $ sudo ./scripts/process_runqlat.bt 652644 Attaching 5 probes... Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end. @tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67 @tasks[Finalizer, 652646]: count 24, average 6, total 151 @tasks[Main Thread, 652668]: count 1432, average 8, total 12561 @tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120 @tasks[Main Thread, 652667]: count 1037, average 9, total 10091 @tasks[FACT Thread, 652679]: count 1033, average 10, total 11005 @tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047 @tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210 @tasks[Main Thread, 652666]: count 104, average 10, total 1088 @tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461 @tasks[Main Thread, 652644]: count 5840, average 11, total 69177 @tasks[Main Thread, 694917]: count 44, average 14, total 659 @tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300 @tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873 @tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848 @tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836 @tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518 @usec_hist: [0] 16 | | [1] 360 |@ | [2, 4) 4650 |@@@@@@@@@@@@@@ | [4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 5056 |@@@@@@@@@@@@@@@ | [32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ | [128, 256) 1926 |@@@@@@ | [256, 512) 130 | | [512, 1K) 2 | | @usec_total_stats: count 57152, average 29, total 1699656 Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
*/
rawtracepoint:sched_wakeup,
rawtracepoint:sched_wakeup_new,
{
$task = (struct task_struct *)arg0;
// We filter by tgid to include all threads of the process
if ($task->tgid == $1) {
@qtime[$task->pid] = nsecs;
}
scripts: Add script to measure runqlat for a process Add a `scripts` folder to hold scripts that are useful for the project, and add a bpftrace program to measure the runqueue latency for a process. bpftrace's built-in runqlat.bt program instruments runqueue latency for all processes and does not provide a way to filter by PID. For sched_ext performance work, we are interested in the runqueue latency of a specific process we are trying to optimize, such as a video game. Therefore, we need to create a custom bpftrace program to achieve this. `process_runqlat.bt` instruments runqueue latency for a PID and its threads. This script measures the runqueue latency for a specified PID and includes all threads spawned by that process. USAGE: sudo ./scripts/process_runqlat.bt <PID> The program output will include: - Stats by thread (count, avg latency, total latency) - A histogram of all latency measurements - Aggregated total stats (count, avg latency, total latency) Example output when targeting Terraria's main process: $ sudo ./scripts/process_runqlat.bt 652644 Attaching 5 probes... Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end. @tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67 @tasks[Finalizer, 652646]: count 24, average 6, total 151 @tasks[Main Thread, 652668]: count 1432, average 8, total 12561 @tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120 @tasks[Main Thread, 652667]: count 1037, average 9, total 10091 @tasks[FACT Thread, 652679]: count 1033, average 10, total 11005 @tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047 @tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210 @tasks[Main Thread, 652666]: count 104, average 10, total 1088 @tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461 @tasks[Main Thread, 652644]: count 5840, average 11, total 69177 @tasks[Main Thread, 694917]: count 44, average 14, total 659 @tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300 @tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873 @tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848 @tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836 @tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518 @usec_hist: [0] 16 | | [1] 360 |@ | [2, 4) 4650 |@@@@@@@@@@@@@@ | [4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 5056 |@@@@@@@@@@@@@@@ | [32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ | [128, 256) 1926 |@@@@@@ | [256, 512) 130 | | [512, 1K) 2 | | @usec_total_stats: count 57152, average 29, total 1699656 Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
}
/*
* args:
* - bool preempt
* - struct task_struct *prev
* - struct task_struct *next
* - unsigned int prev_state
scripts: Add script to measure runqlat for a process Add a `scripts` folder to hold scripts that are useful for the project, and add a bpftrace program to measure the runqueue latency for a process. bpftrace's built-in runqlat.bt program instruments runqueue latency for all processes and does not provide a way to filter by PID. For sched_ext performance work, we are interested in the runqueue latency of a specific process we are trying to optimize, such as a video game. Therefore, we need to create a custom bpftrace program to achieve this. `process_runqlat.bt` instruments runqueue latency for a PID and its threads. This script measures the runqueue latency for a specified PID and includes all threads spawned by that process. USAGE: sudo ./scripts/process_runqlat.bt <PID> The program output will include: - Stats by thread (count, avg latency, total latency) - A histogram of all latency measurements - Aggregated total stats (count, avg latency, total latency) Example output when targeting Terraria's main process: $ sudo ./scripts/process_runqlat.bt 652644 Attaching 5 probes... Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end. @tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67 @tasks[Finalizer, 652646]: count 24, average 6, total 151 @tasks[Main Thread, 652668]: count 1432, average 8, total 12561 @tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120 @tasks[Main Thread, 652667]: count 1037, average 9, total 10091 @tasks[FACT Thread, 652679]: count 1033, average 10, total 11005 @tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047 @tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210 @tasks[Main Thread, 652666]: count 104, average 10, total 1088 @tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461 @tasks[Main Thread, 652644]: count 5840, average 11, total 69177 @tasks[Main Thread, 694917]: count 44, average 14, total 659 @tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300 @tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873 @tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848 @tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836 @tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518 @usec_hist: [0] 16 | | [1] 360 |@ | [2, 4) 4650 |@@@@@@@@@@@@@@ | [4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 5056 |@@@@@@@@@@@@@@@ | [32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ | [128, 256) 1926 |@@@@@@ | [256, 512) 130 | | [512, 1K) 2 | | @usec_total_stats: count 57152, average 29, total 1699656 Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
*/
rawtracepoint:sched_switch
{
$prev = (struct task_struct *)arg1;
$next = (struct task_struct *)arg2;
$prev_state = arg3;
scripts: Add script to measure runqlat for a process Add a `scripts` folder to hold scripts that are useful for the project, and add a bpftrace program to measure the runqueue latency for a process. bpftrace's built-in runqlat.bt program instruments runqueue latency for all processes and does not provide a way to filter by PID. For sched_ext performance work, we are interested in the runqueue latency of a specific process we are trying to optimize, such as a video game. Therefore, we need to create a custom bpftrace program to achieve this. `process_runqlat.bt` instruments runqueue latency for a PID and its threads. This script measures the runqueue latency for a specified PID and includes all threads spawned by that process. USAGE: sudo ./scripts/process_runqlat.bt <PID> The program output will include: - Stats by thread (count, avg latency, total latency) - A histogram of all latency measurements - Aggregated total stats (count, avg latency, total latency) Example output when targeting Terraria's main process: $ sudo ./scripts/process_runqlat.bt 652644 Attaching 5 probes... Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end. @tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67 @tasks[Finalizer, 652646]: count 24, average 6, total 151 @tasks[Main Thread, 652668]: count 1432, average 8, total 12561 @tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120 @tasks[Main Thread, 652667]: count 1037, average 9, total 10091 @tasks[FACT Thread, 652679]: count 1033, average 10, total 11005 @tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047 @tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210 @tasks[Main Thread, 652666]: count 104, average 10, total 1088 @tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461 @tasks[Main Thread, 652644]: count 5840, average 11, total 69177 @tasks[Main Thread, 694917]: count 44, average 14, total 659 @tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300 @tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873 @tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848 @tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836 @tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518 @usec_hist: [0] 16 | | [1] 360 |@ | [2, 4) 4650 |@@@@@@@@@@@@@@ | [4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 5056 |@@@@@@@@@@@@@@@ | [32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ | [128, 256) 1926 |@@@@@@ | [256, 512) 130 | | [512, 1K) 2 | | @usec_total_stats: count 57152, average 29, total 1699656 Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
if ($next->tgid != $1) {
return;
}
scripts: Add script to measure runqlat for a process Add a `scripts` folder to hold scripts that are useful for the project, and add a bpftrace program to measure the runqueue latency for a process. bpftrace's built-in runqlat.bt program instruments runqueue latency for all processes and does not provide a way to filter by PID. For sched_ext performance work, we are interested in the runqueue latency of a specific process we are trying to optimize, such as a video game. Therefore, we need to create a custom bpftrace program to achieve this. `process_runqlat.bt` instruments runqueue latency for a PID and its threads. This script measures the runqueue latency for a specified PID and includes all threads spawned by that process. USAGE: sudo ./scripts/process_runqlat.bt <PID> The program output will include: - Stats by thread (count, avg latency, total latency) - A histogram of all latency measurements - Aggregated total stats (count, avg latency, total latency) Example output when targeting Terraria's main process: $ sudo ./scripts/process_runqlat.bt 652644 Attaching 5 probes... Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end. @tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67 @tasks[Finalizer, 652646]: count 24, average 6, total 151 @tasks[Main Thread, 652668]: count 1432, average 8, total 12561 @tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120 @tasks[Main Thread, 652667]: count 1037, average 9, total 10091 @tasks[FACT Thread, 652679]: count 1033, average 10, total 11005 @tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047 @tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210 @tasks[Main Thread, 652666]: count 104, average 10, total 1088 @tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461 @tasks[Main Thread, 652644]: count 5840, average 11, total 69177 @tasks[Main Thread, 694917]: count 44, average 14, total 659 @tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300 @tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873 @tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848 @tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836 @tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518 @usec_hist: [0] 16 | | [1] 360 |@ | [2, 4) 4650 |@@@@@@@@@@@@@@ | [4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 5056 |@@@@@@@@@@@@@@@ | [32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ | [128, 256) 1926 |@@@@@@ | [256, 512) 130 | | [512, 1K) 2 | | @usec_total_stats: count 57152, average 29, total 1699656 Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
if ($prev_state == TASK_RUNNING && $prev->tgid == $1) {
@qtime[$prev->pid] = nsecs;
}
$nsec = @qtime[$next->pid];
if ($nsec) {
$usec = (nsecs - $nsec) / 1000;
@usec_total_stats = stats($usec);
@usec_hist = hist($usec);
@tasks[$next->comm, $next->pid] = stats($usec);
scripts: Add script to measure runqlat for a process Add a `scripts` folder to hold scripts that are useful for the project, and add a bpftrace program to measure the runqueue latency for a process. bpftrace's built-in runqlat.bt program instruments runqueue latency for all processes and does not provide a way to filter by PID. For sched_ext performance work, we are interested in the runqueue latency of a specific process we are trying to optimize, such as a video game. Therefore, we need to create a custom bpftrace program to achieve this. `process_runqlat.bt` instruments runqueue latency for a PID and its threads. This script measures the runqueue latency for a specified PID and includes all threads spawned by that process. USAGE: sudo ./scripts/process_runqlat.bt <PID> The program output will include: - Stats by thread (count, avg latency, total latency) - A histogram of all latency measurements - Aggregated total stats (count, avg latency, total latency) Example output when targeting Terraria's main process: $ sudo ./scripts/process_runqlat.bt 652644 Attaching 5 probes... Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end. @tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67 @tasks[Finalizer, 652646]: count 24, average 6, total 151 @tasks[Main Thread, 652668]: count 1432, average 8, total 12561 @tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120 @tasks[Main Thread, 652667]: count 1037, average 9, total 10091 @tasks[FACT Thread, 652679]: count 1033, average 10, total 11005 @tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047 @tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210 @tasks[Main Thread, 652666]: count 104, average 10, total 1088 @tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461 @tasks[Main Thread, 652644]: count 5840, average 11, total 69177 @tasks[Main Thread, 694917]: count 44, average 14, total 659 @tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300 @tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873 @tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848 @tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836 @tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518 @usec_hist: [0] 16 | | [1] 360 |@ | [2, 4) 4650 |@@@@@@@@@@@@@@ | [4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 5056 |@@@@@@@@@@@@@@@ | [32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ | [128, 256) 1926 |@@@@@@ | [256, 512) 130 | | [512, 1K) 2 | | @usec_total_stats: count 57152, average 29, total 1699656 Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
}
delete(@qtime[$next->pid]);
}
END
{
clear(@qtime);
scripts: Add script to measure runqlat for a process Add a `scripts` folder to hold scripts that are useful for the project, and add a bpftrace program to measure the runqueue latency for a process. bpftrace's built-in runqlat.bt program instruments runqueue latency for all processes and does not provide a way to filter by PID. For sched_ext performance work, we are interested in the runqueue latency of a specific process we are trying to optimize, such as a video game. Therefore, we need to create a custom bpftrace program to achieve this. `process_runqlat.bt` instruments runqueue latency for a PID and its threads. This script measures the runqueue latency for a specified PID and includes all threads spawned by that process. USAGE: sudo ./scripts/process_runqlat.bt <PID> The program output will include: - Stats by thread (count, avg latency, total latency) - A histogram of all latency measurements - Aggregated total stats (count, avg latency, total latency) Example output when targeting Terraria's main process: $ sudo ./scripts/process_runqlat.bt 652644 Attaching 5 probes... Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end. @tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67 @tasks[Finalizer, 652646]: count 24, average 6, total 151 @tasks[Main Thread, 652668]: count 1432, average 8, total 12561 @tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120 @tasks[Main Thread, 652667]: count 1037, average 9, total 10091 @tasks[FACT Thread, 652679]: count 1033, average 10, total 11005 @tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047 @tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210 @tasks[Main Thread, 652666]: count 104, average 10, total 1088 @tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461 @tasks[Main Thread, 652644]: count 5840, average 11, total 69177 @tasks[Main Thread, 694917]: count 44, average 14, total 659 @tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300 @tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873 @tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848 @tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836 @tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518 @usec_hist: [0] 16 | | [1] 360 |@ | [2, 4) 4650 |@@@@@@@@@@@@@@ | [4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 5056 |@@@@@@@@@@@@@@@ | [32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ | [128, 256) 1926 |@@@@@@ | [256, 512) 130 | | [512, 1K) 2 | | @usec_total_stats: count 57152, average 29, total 1699656 Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
}