scripts: Add script to measure runqlat for a process
Add a `scripts` folder to hold scripts that are useful for the project,
and add a bpftrace program to measure the runqueue latency for a process.
bpftrace's built-in runqlat.bt program instruments runqueue latency for
all processes and does not provide a way to filter by PID. For sched_ext
performance work, we are interested in the runqueue latency of a
specific process we are trying to optimize, such as a video game.
Therefore, we need to create a custom bpftrace program to achieve this.
`process_runqlat.bt` instruments runqueue latency for a PID and its
threads. This script measures the runqueue latency for a specified PID
and includes all threads spawned by that process.
USAGE: sudo ./scripts/process_runqlat.bt <PID>
The program output will include:
- Stats by thread (count, avg latency, total latency)
- A histogram of all latency measurements
- Aggregated total stats (count, avg latency, total latency)
Example output when targeting Terraria's main process:
$ sudo ./scripts/process_runqlat.bt 652644
Attaching 5 probes...
Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end.
@tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67
@tasks[Finalizer, 652646]: count 24, average 6, total 151
@tasks[Main Thread, 652668]: count 1432, average 8, total 12561
@tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120
@tasks[Main Thread, 652667]: count 1037, average 9, total 10091
@tasks[FACT Thread, 652679]: count 1033, average 10, total 11005
@tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047
@tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210
@tasks[Main Thread, 652666]: count 104, average 10, total 1088
@tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461
@tasks[Main Thread, 652644]: count 5840, average 11, total 69177
@tasks[Main Thread, 694917]: count 44, average 14, total 659
@tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300
@tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873
@tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848
@tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836
@tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518
@usec_hist:
[0] 16 | |
[1] 360 |@ |
[2, 4) 4650 |@@@@@@@@@@@@@@ |
[4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32) 5056 |@@@@@@@@@@@@@@@ |
[32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ |
[64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ |
[128, 256) 1926 |@@@@@@ |
[256, 512) 130 | |
[512, 1K) 2 | |
@usec_total_stats: count 57152, average 29, total 1699656
Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
|
|
|
#!/usr/bin/env bpftrace
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Copyright (c) Jose Fernandez <josef@netflix.com>
|
|
|
|
*
|
|
|
|
* process_runqlat.bt - Instrument runqueue latency for a PID and its threads.
|
|
|
|
*
|
|
|
|
* This script measures the runqueue latency for a specified process ID (PID)
|
|
|
|
* and includes all threads spawned by that process.
|
|
|
|
*
|
|
|
|
* USAGE: sudo ./process_runqlat.bt <PID>
|
|
|
|
*
|
|
|
|
* The program output will include:
|
|
|
|
* - Stats by task (count, avg latency, total latency)
|
|
|
|
* - A histogram of all latency measurements
|
|
|
|
* - Aggregated total stats (count, avg latency, total latency)
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/sched.h>
|
|
|
|
|
|
|
|
BEGIN
|
|
|
|
{
|
2024-05-15 21:53:03 +01:00
|
|
|
if ($1 == 0) {
|
|
|
|
printf("PID is missing, use `sudo ./process_runqlat.bt <pid>`\n");
|
|
|
|
exit();
|
|
|
|
}
|
|
|
|
printf("Instrumenting runqueue latency for PID %d. Hit Ctrl-C to end.\n", $1);
|
scripts: Add script to measure runqlat for a process
Add a `scripts` folder to hold scripts that are useful for the project,
and add a bpftrace program to measure the runqueue latency for a process.
bpftrace's built-in runqlat.bt program instruments runqueue latency for
all processes and does not provide a way to filter by PID. For sched_ext
performance work, we are interested in the runqueue latency of a
specific process we are trying to optimize, such as a video game.
Therefore, we need to create a custom bpftrace program to achieve this.
`process_runqlat.bt` instruments runqueue latency for a PID and its
threads. This script measures the runqueue latency for a specified PID
and includes all threads spawned by that process.
USAGE: sudo ./scripts/process_runqlat.bt <PID>
The program output will include:
- Stats by thread (count, avg latency, total latency)
- A histogram of all latency measurements
- Aggregated total stats (count, avg latency, total latency)
Example output when targeting Terraria's main process:
$ sudo ./scripts/process_runqlat.bt 652644
Attaching 5 probes...
Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end.
@tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67
@tasks[Finalizer, 652646]: count 24, average 6, total 151
@tasks[Main Thread, 652668]: count 1432, average 8, total 12561
@tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120
@tasks[Main Thread, 652667]: count 1037, average 9, total 10091
@tasks[FACT Thread, 652679]: count 1033, average 10, total 11005
@tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047
@tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210
@tasks[Main Thread, 652666]: count 104, average 10, total 1088
@tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461
@tasks[Main Thread, 652644]: count 5840, average 11, total 69177
@tasks[Main Thread, 694917]: count 44, average 14, total 659
@tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300
@tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873
@tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848
@tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836
@tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518
@usec_hist:
[0] 16 | |
[1] 360 |@ |
[2, 4) 4650 |@@@@@@@@@@@@@@ |
[4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32) 5056 |@@@@@@@@@@@@@@@ |
[32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ |
[64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ |
[128, 256) 1926 |@@@@@@ |
[256, 512) 130 | |
[512, 1K) 2 | |
@usec_total_stats: count 57152, average 29, total 1699656
Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2024-05-15 21:53:03 +01:00
|
|
|
* args:
|
|
|
|
* - struct task_struct *p
|
scripts: Add script to measure runqlat for a process
Add a `scripts` folder to hold scripts that are useful for the project,
and add a bpftrace program to measure the runqueue latency for a process.
bpftrace's built-in runqlat.bt program instruments runqueue latency for
all processes and does not provide a way to filter by PID. For sched_ext
performance work, we are interested in the runqueue latency of a
specific process we are trying to optimize, such as a video game.
Therefore, we need to create a custom bpftrace program to achieve this.
`process_runqlat.bt` instruments runqueue latency for a PID and its
threads. This script measures the runqueue latency for a specified PID
and includes all threads spawned by that process.
USAGE: sudo ./scripts/process_runqlat.bt <PID>
The program output will include:
- Stats by thread (count, avg latency, total latency)
- A histogram of all latency measurements
- Aggregated total stats (count, avg latency, total latency)
Example output when targeting Terraria's main process:
$ sudo ./scripts/process_runqlat.bt 652644
Attaching 5 probes...
Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end.
@tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67
@tasks[Finalizer, 652646]: count 24, average 6, total 151
@tasks[Main Thread, 652668]: count 1432, average 8, total 12561
@tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120
@tasks[Main Thread, 652667]: count 1037, average 9, total 10091
@tasks[FACT Thread, 652679]: count 1033, average 10, total 11005
@tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047
@tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210
@tasks[Main Thread, 652666]: count 104, average 10, total 1088
@tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461
@tasks[Main Thread, 652644]: count 5840, average 11, total 69177
@tasks[Main Thread, 694917]: count 44, average 14, total 659
@tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300
@tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873
@tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848
@tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836
@tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518
@usec_hist:
[0] 16 | |
[1] 360 |@ |
[2, 4) 4650 |@@@@@@@@@@@@@@ |
[4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32) 5056 |@@@@@@@@@@@@@@@ |
[32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ |
[64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ |
[128, 256) 1926 |@@@@@@ |
[256, 512) 130 | |
[512, 1K) 2 | |
@usec_total_stats: count 57152, average 29, total 1699656
Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
|
|
|
*/
|
|
|
|
rawtracepoint:sched_wakeup,
|
|
|
|
rawtracepoint:sched_wakeup_new,
|
|
|
|
{
|
2024-05-15 21:53:03 +01:00
|
|
|
$task = (struct task_struct *)arg0;
|
|
|
|
// We filter by tgid to include all threads of the process
|
|
|
|
if ($task->tgid == $1) {
|
|
|
|
@qtime[$task->pid] = nsecs;
|
|
|
|
}
|
scripts: Add script to measure runqlat for a process
Add a `scripts` folder to hold scripts that are useful for the project,
and add a bpftrace program to measure the runqueue latency for a process.
bpftrace's built-in runqlat.bt program instruments runqueue latency for
all processes and does not provide a way to filter by PID. For sched_ext
performance work, we are interested in the runqueue latency of a
specific process we are trying to optimize, such as a video game.
Therefore, we need to create a custom bpftrace program to achieve this.
`process_runqlat.bt` instruments runqueue latency for a PID and its
threads. This script measures the runqueue latency for a specified PID
and includes all threads spawned by that process.
USAGE: sudo ./scripts/process_runqlat.bt <PID>
The program output will include:
- Stats by thread (count, avg latency, total latency)
- A histogram of all latency measurements
- Aggregated total stats (count, avg latency, total latency)
Example output when targeting Terraria's main process:
$ sudo ./scripts/process_runqlat.bt 652644
Attaching 5 probes...
Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end.
@tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67
@tasks[Finalizer, 652646]: count 24, average 6, total 151
@tasks[Main Thread, 652668]: count 1432, average 8, total 12561
@tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120
@tasks[Main Thread, 652667]: count 1037, average 9, total 10091
@tasks[FACT Thread, 652679]: count 1033, average 10, total 11005
@tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047
@tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210
@tasks[Main Thread, 652666]: count 104, average 10, total 1088
@tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461
@tasks[Main Thread, 652644]: count 5840, average 11, total 69177
@tasks[Main Thread, 694917]: count 44, average 14, total 659
@tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300
@tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873
@tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848
@tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836
@tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518
@usec_hist:
[0] 16 | |
[1] 360 |@ |
[2, 4) 4650 |@@@@@@@@@@@@@@ |
[4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32) 5056 |@@@@@@@@@@@@@@@ |
[32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ |
[64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ |
[128, 256) 1926 |@@@@@@ |
[256, 512) 130 | |
[512, 1K) 2 | |
@usec_total_stats: count 57152, average 29, total 1699656
Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2024-05-15 21:53:03 +01:00
|
|
|
* args:
|
|
|
|
* - bool preempt
|
|
|
|
* - struct task_struct *prev
|
|
|
|
* - struct task_struct *next
|
|
|
|
* - unsigned int prev_state
|
scripts: Add script to measure runqlat for a process
Add a `scripts` folder to hold scripts that are useful for the project,
and add a bpftrace program to measure the runqueue latency for a process.
bpftrace's built-in runqlat.bt program instruments runqueue latency for
all processes and does not provide a way to filter by PID. For sched_ext
performance work, we are interested in the runqueue latency of a
specific process we are trying to optimize, such as a video game.
Therefore, we need to create a custom bpftrace program to achieve this.
`process_runqlat.bt` instruments runqueue latency for a PID and its
threads. This script measures the runqueue latency for a specified PID
and includes all threads spawned by that process.
USAGE: sudo ./scripts/process_runqlat.bt <PID>
The program output will include:
- Stats by thread (count, avg latency, total latency)
- A histogram of all latency measurements
- Aggregated total stats (count, avg latency, total latency)
Example output when targeting Terraria's main process:
$ sudo ./scripts/process_runqlat.bt 652644
Attaching 5 probes...
Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end.
@tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67
@tasks[Finalizer, 652646]: count 24, average 6, total 151
@tasks[Main Thread, 652668]: count 1432, average 8, total 12561
@tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120
@tasks[Main Thread, 652667]: count 1037, average 9, total 10091
@tasks[FACT Thread, 652679]: count 1033, average 10, total 11005
@tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047
@tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210
@tasks[Main Thread, 652666]: count 104, average 10, total 1088
@tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461
@tasks[Main Thread, 652644]: count 5840, average 11, total 69177
@tasks[Main Thread, 694917]: count 44, average 14, total 659
@tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300
@tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873
@tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848
@tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836
@tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518
@usec_hist:
[0] 16 | |
[1] 360 |@ |
[2, 4) 4650 |@@@@@@@@@@@@@@ |
[4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32) 5056 |@@@@@@@@@@@@@@@ |
[32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ |
[64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ |
[128, 256) 1926 |@@@@@@ |
[256, 512) 130 | |
[512, 1K) 2 | |
@usec_total_stats: count 57152, average 29, total 1699656
Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
|
|
|
*/
|
|
|
|
rawtracepoint:sched_switch
|
|
|
|
{
|
2024-05-15 21:53:03 +01:00
|
|
|
$prev = (struct task_struct *)arg1;
|
|
|
|
$next = (struct task_struct *)arg2;
|
|
|
|
$prev_state = arg3;
|
scripts: Add script to measure runqlat for a process
Add a `scripts` folder to hold scripts that are useful for the project,
and add a bpftrace program to measure the runqueue latency for a process.
bpftrace's built-in runqlat.bt program instruments runqueue latency for
all processes and does not provide a way to filter by PID. For sched_ext
performance work, we are interested in the runqueue latency of a
specific process we are trying to optimize, such as a video game.
Therefore, we need to create a custom bpftrace program to achieve this.
`process_runqlat.bt` instruments runqueue latency for a PID and its
threads. This script measures the runqueue latency for a specified PID
and includes all threads spawned by that process.
USAGE: sudo ./scripts/process_runqlat.bt <PID>
The program output will include:
- Stats by thread (count, avg latency, total latency)
- A histogram of all latency measurements
- Aggregated total stats (count, avg latency, total latency)
Example output when targeting Terraria's main process:
$ sudo ./scripts/process_runqlat.bt 652644
Attaching 5 probes...
Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end.
@tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67
@tasks[Finalizer, 652646]: count 24, average 6, total 151
@tasks[Main Thread, 652668]: count 1432, average 8, total 12561
@tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120
@tasks[Main Thread, 652667]: count 1037, average 9, total 10091
@tasks[FACT Thread, 652679]: count 1033, average 10, total 11005
@tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047
@tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210
@tasks[Main Thread, 652666]: count 104, average 10, total 1088
@tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461
@tasks[Main Thread, 652644]: count 5840, average 11, total 69177
@tasks[Main Thread, 694917]: count 44, average 14, total 659
@tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300
@tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873
@tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848
@tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836
@tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518
@usec_hist:
[0] 16 | |
[1] 360 |@ |
[2, 4) 4650 |@@@@@@@@@@@@@@ |
[4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32) 5056 |@@@@@@@@@@@@@@@ |
[32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ |
[64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ |
[128, 256) 1926 |@@@@@@ |
[256, 512) 130 | |
[512, 1K) 2 | |
@usec_total_stats: count 57152, average 29, total 1699656
Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
|
|
|
|
2024-05-15 21:53:03 +01:00
|
|
|
if ($next->tgid != $1) {
|
|
|
|
return;
|
|
|
|
}
|
scripts: Add script to measure runqlat for a process
Add a `scripts` folder to hold scripts that are useful for the project,
and add a bpftrace program to measure the runqueue latency for a process.
bpftrace's built-in runqlat.bt program instruments runqueue latency for
all processes and does not provide a way to filter by PID. For sched_ext
performance work, we are interested in the runqueue latency of a
specific process we are trying to optimize, such as a video game.
Therefore, we need to create a custom bpftrace program to achieve this.
`process_runqlat.bt` instruments runqueue latency for a PID and its
threads. This script measures the runqueue latency for a specified PID
and includes all threads spawned by that process.
USAGE: sudo ./scripts/process_runqlat.bt <PID>
The program output will include:
- Stats by thread (count, avg latency, total latency)
- A histogram of all latency measurements
- Aggregated total stats (count, avg latency, total latency)
Example output when targeting Terraria's main process:
$ sudo ./scripts/process_runqlat.bt 652644
Attaching 5 probes...
Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end.
@tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67
@tasks[Finalizer, 652646]: count 24, average 6, total 151
@tasks[Main Thread, 652668]: count 1432, average 8, total 12561
@tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120
@tasks[Main Thread, 652667]: count 1037, average 9, total 10091
@tasks[FACT Thread, 652679]: count 1033, average 10, total 11005
@tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047
@tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210
@tasks[Main Thread, 652666]: count 104, average 10, total 1088
@tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461
@tasks[Main Thread, 652644]: count 5840, average 11, total 69177
@tasks[Main Thread, 694917]: count 44, average 14, total 659
@tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300
@tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873
@tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848
@tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836
@tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518
@usec_hist:
[0] 16 | |
[1] 360 |@ |
[2, 4) 4650 |@@@@@@@@@@@@@@ |
[4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32) 5056 |@@@@@@@@@@@@@@@ |
[32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ |
[64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ |
[128, 256) 1926 |@@@@@@ |
[256, 512) 130 | |
[512, 1K) 2 | |
@usec_total_stats: count 57152, average 29, total 1699656
Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
|
|
|
|
|
|
|
if ($prev_state == TASK_RUNNING && $prev->tgid == $1) {
|
|
|
|
@qtime[$prev->pid] = nsecs;
|
|
|
|
}
|
|
|
|
|
|
|
|
$nsec = @qtime[$next->pid];
|
|
|
|
if ($nsec) {
|
2024-05-15 21:53:03 +01:00
|
|
|
$usec = (nsecs - $nsec) / 1000;
|
|
|
|
@usec_total_stats = stats($usec);
|
|
|
|
@usec_hist = hist($usec);
|
|
|
|
@tasks[$next->comm, $next->pid] = stats($usec);
|
scripts: Add script to measure runqlat for a process
Add a `scripts` folder to hold scripts that are useful for the project,
and add a bpftrace program to measure the runqueue latency for a process.
bpftrace's built-in runqlat.bt program instruments runqueue latency for
all processes and does not provide a way to filter by PID. For sched_ext
performance work, we are interested in the runqueue latency of a
specific process we are trying to optimize, such as a video game.
Therefore, we need to create a custom bpftrace program to achieve this.
`process_runqlat.bt` instruments runqueue latency for a PID and its
threads. This script measures the runqueue latency for a specified PID
and includes all threads spawned by that process.
USAGE: sudo ./scripts/process_runqlat.bt <PID>
The program output will include:
- Stats by thread (count, avg latency, total latency)
- A histogram of all latency measurements
- Aggregated total stats (count, avg latency, total latency)
Example output when targeting Terraria's main process:
$ sudo ./scripts/process_runqlat.bt 652644
Attaching 5 probes...
Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end.
@tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67
@tasks[Finalizer, 652646]: count 24, average 6, total 151
@tasks[Main Thread, 652668]: count 1432, average 8, total 12561
@tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120
@tasks[Main Thread, 652667]: count 1037, average 9, total 10091
@tasks[FACT Thread, 652679]: count 1033, average 10, total 11005
@tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047
@tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210
@tasks[Main Thread, 652666]: count 104, average 10, total 1088
@tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461
@tasks[Main Thread, 652644]: count 5840, average 11, total 69177
@tasks[Main Thread, 694917]: count 44, average 14, total 659
@tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300
@tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873
@tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848
@tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836
@tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518
@usec_hist:
[0] 16 | |
[1] 360 |@ |
[2, 4) 4650 |@@@@@@@@@@@@@@ |
[4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32) 5056 |@@@@@@@@@@@@@@@ |
[32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ |
[64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ |
[128, 256) 1926 |@@@@@@ |
[256, 512) 130 | |
[512, 1K) 2 | |
@usec_total_stats: count 57152, average 29, total 1699656
Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
|
|
|
}
|
|
|
|
delete(@qtime[$next->pid]);
|
|
|
|
}
|
|
|
|
|
|
|
|
END
|
|
|
|
{
|
2024-05-15 21:53:03 +01:00
|
|
|
clear(@qtime);
|
scripts: Add script to measure runqlat for a process
Add a `scripts` folder to hold scripts that are useful for the project,
and add a bpftrace program to measure the runqueue latency for a process.
bpftrace's built-in runqlat.bt program instruments runqueue latency for
all processes and does not provide a way to filter by PID. For sched_ext
performance work, we are interested in the runqueue latency of a
specific process we are trying to optimize, such as a video game.
Therefore, we need to create a custom bpftrace program to achieve this.
`process_runqlat.bt` instruments runqueue latency for a PID and its
threads. This script measures the runqueue latency for a specified PID
and includes all threads spawned by that process.
USAGE: sudo ./scripts/process_runqlat.bt <PID>
The program output will include:
- Stats by thread (count, avg latency, total latency)
- A histogram of all latency measurements
- Aggregated total stats (count, avg latency, total latency)
Example output when targeting Terraria's main process:
$ sudo ./scripts/process_runqlat.bt 652644
Attaching 5 probes...
Instrumenting runqueue latency for PID 652644. Hit Ctrl-C to end.
@tasks[AsyncActionDisp, 652676]: count 24, average 2, total 67
@tasks[Finalizer, 652646]: count 24, average 6, total 151
@tasks[Main Thread, 652668]: count 1432, average 8, total 12561
@tasks[Terraria.b:gl0, 652672]: count 1421, average 9, total 13120
@tasks[Main Thread, 652667]: count 1037, average 9, total 10091
@tasks[FACT Thread, 652679]: count 1033, average 10, total 11005
@tasks[Terraria:gdrv0, 652671]: count 3511, average 10, total 37047
@tasks[SDLAudioP3, 652678]: count 982, average 10, total 10210
@tasks[Main Thread, 652666]: count 104, average 10, total 1088
@tasks[SDLAudioP2, 652675]: count 982, average 10, total 10461
@tasks[Main Thread, 652644]: count 5840, average 11, total 69177
@tasks[Main Thread, 694917]: count 44, average 14, total 659
@tasks[Terraria.b:cs0, 652650]: count 3288, average 14, total 47300
@tasks[Thread Pool Wor, 695239]: count 1001, average 35, total 35873
@tasks[Thread Pool Wor, 696059]: count 986, average 35, total 34848
@tasks[Thread Pool Wor, 695458]: count 985, average 36, total 35836
@tasks[Thread Pool Wor, 696058]: count 982, average 37, total 36518
@usec_hist:
[0] 16 | |
[1] 360 |@ |
[2, 4) 4650 |@@@@@@@@@@@@@@ |
[4, 8) 14517 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 16593 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32) 5056 |@@@@@@@@@@@@@@@ |
[32, 64) 6795 |@@@@@@@@@@@@@@@@@@@@@ |
[64, 128) 7106 |@@@@@@@@@@@@@@@@@@@@@@ |
[128, 256) 1926 |@@@@@@ |
[256, 512) 130 | |
[512, 1K) 2 | |
@usec_total_stats: count 57152, average 29, total 1699656
Signed-off-by: Jose Fernandez <josef@netflix.com>
2024-05-15 15:59:53 +01:00
|
|
|
}
|