As described in [0], there is an open problem in load balancing called
the "infeasible weights" problem. Essentially, the problem boils down to
the fact that a task with disproportionately high load can be granted
more CPU time than they can actually consume per their duty cycle.
This patch implements a solution to that problem, wherein we apply the
algorithm described in this paper to adjust all infeasible weights in
the system down to a feasible wight that gives them their full duty
cycle, while allowing the remaining feasible tasks on the system to
share the remaining compute capacity on the machine.
[0]: https://drive.google.com/file/d/1fAoWUlmW-HTp6akuATVpMxpUpvWcGSAv/view?usp=drive_link
Signed-off-by: David Vernet <void@manifault.com>
SCX_KICK_IDLE is a new feature which isn't defined in older kernels. Add
compat wrapper and use it for idle CPU wakeups.
Signed-off-by: Tejun Heo <tj@kernel.org>
Search for potential errors only in the kernel logs and the scheduler
stderr.
In this way we can use "error keywords" in the scheduler's output
without triggering false positives in the CI (see for example #127).
NOTE: this works, because virtme-ng, when executed in verbose mode,
sends the kernel messages to stderr (together with the command's stderr)
and it channels the command's stdout to the stdout of the host.
Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
This is meant to be an example scheduler that won't necessarily run well
in production. Let's remove the 3 second timeout and use the system
default of 30.
Signed-off-by: David Vernet <void@manifault.com>
We're basically always runnin two CI jobs: one for a remote push, and
another for when a PR is opened. These are essentially measuring the
same thing, so let's save CI bandwidth and just do a PR run. This will
hopefully make things a bit less noisy as well.
Signed-off-by: David Vernet <void@manifault.com>
Let's make it clear that this scheduler isn't expected to perform well,
and instead point people to scx_rustland.
Signed-off-by: David Vernet <void@manifault.com>
Items in the task BTreeSet are stored by pid and vruntime. Make sure
that we never store multiple items with the same PID, so that
re-enqueued tasks are not dispatched multiple times.
Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Only scx_simple/qmap are in the kernel tree now. Drop the rest from the sync
script. Also update the sync script so that it can handle empty
rust_scheds variable.
Signed-off-by: Tejun Heo <tj@kernel.org>
Pierre Jacquet pointed out that our docs in the scx repo are out of date
for the latest APIs. Let's update it so readers don't get confused.
Signed-off-by: David Vernet <void@manifault.com>
Allow to scale the effective time slice down to 250 us. This can help to
maintain a good quality of the audio even when the system is overloaded
by multiple CPU-intensive tasks.
Moreover, always round up the time slice scaling factor to be a little
more aggressive and prioritize at scaling the time slice, so that we can
prioritize low latency tasks even more.
Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Evaluate the number of voluntary context switches per second (nvcsw/sec)
for each task using an exponentially weighted moving average (EWMA) with
weight 0.5, that allows to classify interactive tasks with more
accuracy.
Using a simple average over a period of time of 10 sec can introduce
small lags every 10 sec, as the statistics for the number of voluntary
context switches are refreshed. This can result in interactive tasks
taking a brief time to catch up in order to be accurately classified as
so, causing for example short audio cracks, small drop of 5-10 fps in
games, etc.
Using a EMWA allows to smooth the average of nvcsw/sec, preventing short
lags in the interactive tasks, while also preventing to incorrectly
classify as interactive tasks that may experience an isolated short
burst of voluntary context switches.
This patch has been tested with the usual test case of playing a
videogame while running a parallel kernel build in the background.
Without this patch the short lag every 10 sec is clearly noticeable,
with this patch applied the game and audio run smoothly.
Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Simplify the idle selection logic by relying only on the built-in idle
selection performed in the BPF layer.
When there are idle CPUs available in the system, tasks are dispatched
directly by the BPF dispatcher without invoking the user-space
scheduler. This allows to avoid the user-space overhead and get the best
system performance when CPU resources are not overcommitted.
Once the number of tasks exceeds the available CPUs, the user-space
scheduler takes over. However, by this time, the system is already
overcommitted, so there's little advantage in attempting to pinpoint the
optimal idle CPU through the user-space scheduler. Instead, tasks can be
executed on the first available CPU, consistently dispatching them to
the shared DSQ.
This allows to achieve the optimal performance both with system
under-utilization and over-utilization.
With this change in place the user-space scheduler won't dispatch tasks
directly to specific CPUs, but we still want to keep this as a generic
feature in the BPF layer, so that it can be potentially used in the
future by this scheduler or even by other user-space schedulers (once
the BPF layer will be moved to a more generic place).
Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
When the user-space scheduler dispatches a task on a specific CPU, that
CPU might not be valid, since the user-space doesn't have visibility of
the task's cpumask.
When this happens the BPF dispatcher (that has direct visibility of the
cpumask) should automatically redirect the task to a valid CPU, but
instead of bouncing the task on the shared DSQ, we should try to use the
CPU assigned by the built-in idle selection logic.
If this CPU is also not valid, then we can simply ignore the task, that
has been de-queued and re-enqueued, since a valid CPU will be naturally
re-selected at a later time.
Moreover, avoid to kick any specific CPU when the task is dispatched to
shared DSQ, since the task can be consumed on any CPU and the additional
kick would simply add more overhead.
Lastly, rename dsq_id_to_cpu() to dsq_to_cpu() and cpu_to_dsq_id() to
cpu_to_dsq() for more clarity.
Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
With commit c6ada25 ("scx_rustland: use custom pcpu DSQ instead of
SCX_DSQ_LOCAL{_ON}") we tried to introduce custom per-CPU DSQs, instead
of using SCX_DSQ_LOCAL and SCX_DSQ_LOCAL_ON to dispatch tasks.
This was required, because dispatching tasks using SCX_DSQ_LOCAL_ON
doesn't provide a guarantee that the cpumask, checked at dispatch time
to determine the validity of a target CPU, remains valid.
This method solved the cpumask validity issue, but unfortunately it
introduced a noticeable performance regression and a potential
starvation issue (that were probably caused by the same problem): if a
task is assigned to a CPU in select_cpu() and the scheduler decides to
dispatch it on a different CPU, the task will be added to the new CPU's
DSQ, but if no dispatch event happens there, the task may remain stuck
in the per-CPU DSQ for a long time, triggering the sched-ext watchdog
timeout that would kick out the scheduler, for example:
12:53:28 [WARN] FAIL: IPC:CSteamEngin[7217] failed to run for 6.482s (err=1026)
12:53:28 [INFO] Unregister RustLand scheduler
Therefore, we reverted this change with 6d89ece ("scx_rustland: dispatch
tasks only on the global DSQ"), dispatching all the tasks to the global
DSQ, completely delegating the kernel to distribute tasks among the
available CPUs.
This is not the ideal solution, because we still want to give the
possibility to the user-space scheduler to assign tasks to specific
CPUs.
Therefore, re-introduce distinct per-CPU DSQs, but also provide a global
shared DSQ. Tasks dispatched in the per-CPU DSQs are consumed from the
dispatch() callback of their corresponding CPU, tasks dispatched in the
global shared DSQ are consumed from any CPU.
In this way the BPF layer is able to provide an interface that gives
the flexibility to the user-space to dispatch a task on a specific CPU
or on the first CPU available, depending on the particular scheduler's
need.
If an invalid CPU (according to the cpumask) is selected the BPF
dispatcher will transparently redirect the task to a valid CPU, selected
using the built-in idle selection logic.
In the future we may want to improve this part, giving to the
user-space the visibility of the cpumask, in order to pick a valid CPU
in advance and in a proper synchronized way.
Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
No functional change, just some refactoring to make the code more clear.
We have is_usersched_needed() and set_usersched_needed() that are doing
different things (the former is checkig if there are pending tasks for
the scheduler, the latter is setting the usersched_needed flag to
activate the dispatch of the user-space scheduler).
Rename is_usersched_needed() to usersched_has_pending_tasks() to make
the code more clear and understandable.
Also move dispatch_user_scheduler() closer to the other dispatch-related
helper functions.
Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
If we are doing local dispatch, we can avoid enqueue() altogether by
dispatching from select_cpu()
Signed-off-by: Dan Schatzberg <schatzberg.dan@gmail.com>
This is a really minor optimization, but we don't need idle_smtmask to
schedule pinned tasks, so defer it so the nr_cpus_allowed == 1 path is
marginally faster.
Signed-off-by: Dan Schatzberg <schatzberg.dan@gmail.com>
idle_cpumask isn't used at all in pick_idle_cpu_from. The only need for
these cpumasks is to check if prev_cpu is a wholly idle CPU (and we only
do this when smt_enabled). idle_smtmask is sufficient for that check.
Signed-off-by: Dan Schatzberg <schatzberg.dan@gmail.com>
