Sched_ext Schedulers and Tools

sched_ext is a Linux kernel feature which enables implementing kernel thread schedulers in BPF and dynamically loading them. This repository contains various scheduler implementations and support utilities.

sched_ext enables safe and rapid iterations of scheduler implementations radically widening the scope of scheduling strategies that can be experimented with and deployed even in massive and complex production environments.

• The scx_layered case study concretely demonstrates the power and benefits of sched_ext.
• For more detailed high-level discussion, please refer to the overview document.

While the kernel feature is not upstream yet, we believe sched_ext has a reasonable chance of landing upstream in the foreseeable future. Both Meta and Google are fully committed to sched_ext and Meta is in the process of mass production deployment. See (#kernel-feature-status) for more details.

In all example shell commands, $SCX refers to the root of this repository.

Getting Started

TO BE WRITTEN.

Repository Structure

scx
|-- scheds               : Sched_ext scheduler implementations
|   |-- include          : Shared BPF and user C include files including vmlinux.h
|   |-- kernel-examples  : Example schedulers from kernel tree (tools/sched_ext)
|   \-- rust-user        : Schedulers that use Rust for userspace component
\-- rust                 : Rust support code
    \-- scx_utils        : Common utility library for rust-user schedulers

Build & Install

meson is the main build system but each Rust sub-project is its own self-contained cargo project and can be built and published separately. The followings are the dependencies and version requirements.

• meson: >=1.2, build scripts under meson-scripts/ use bash and standard utilities including awk.
• clang: >=16 required, >=17 recommended
• libbpf: >=1.2.2 required, >=1.3 recommended (RESIZE_ARRAY support is new in 1.3)
• Rust toolchain: >=1.72
• libelf, libz, libzstd if linking against staic libbpf.a

Setting Up and Building

meson always uses a separate build directory. Running the following commands in the root of the tree builds and installs all schedulers under ~/bin.

$ cd $SCX
$ meson setup build --prefix ~
$ cd build
$ meson compile
$ meson install

Note that meson compile step is not strictly necessary as install implies compile. The above also will build debug binaries with optimizations turned off, which is useful for development but they aren't optimized and big. For actual use you want to build release binaries. meson uses -D argument to specify build options. The configuration options can be specified at setup time but can also be changed afterwards and meson will do the right thing. To switch to release builds, run the following in the build directory and then compile and install again.

$ meson configure -Dbuildtype=release

Running meson configure without any argument shows all current build options. For more information on meson arguments and built-in options, please refer to meson --help and its documentation.

Building Specific Schedulers and Binary Locations

If you just want to build a subset of schedulers, you can specify the scheduler names as arguments to meson compile. For example, if we just want to build the simple example scheduler scheds/kernel-examples/scx_simple and the Rust userspace scheduler scheds/rust-user/scx_rusty:

$ cd $SCX
$ meson setup build -Dbuildtype=release
$ cd build
$ meson compile -v scx_simple scx_rusty

You can also specify -v if you want to see the commands being used:

$ meson compile -v scx_pair

For C userspace schedulers such as the ones under scheds/kernel-examples, the built binaries are located in the same directory under the build root. For example, here, the scx_simple binary can be found at $SCX/build/scheds/kernel-examples/scx_simple.

For Rust userspace schedulers such as the ones under scheds/rust-user, the same directory under the build root is used as the cargo build target directory. Thus, here, the scx_rusty binary can be found at $SCX/build/scheds/rust-user/scx_rusty/release/scx_rusty.

SCX specific build options

While the default options should work in most cases, it may be desirable to override some of the toolchains and dependencies - e.g. to directly use libbpf built from the kernel source tree. The following meson build options can be used in such cases.

• bpf_clang: clang to use when compiling .bpf.c
• bpftool: bpftool to use when generating .bpf.skel.h
• libbpf_a: Static libbpf.a to use
• libbpf_h: libbpf header directories, only meaningful with libbpf_a option
• cargo: cargo to use when building rust sub-projects

For example, let's say you want to use bpftool and libbpf shipped in the kernel tree located at $KERNEL. We need to build bpftool in the kernel tree first, set up SCX build with the related options and then build & install.

$ cd $KERNEL
$ make -C tools/bpf/bpftool
$ cd $SCX
$ BPFTOOL=$KERNEL/tools/bpf/bpftool
$ meson setup build -Dbuildtype=release -Dprefix=~/bin \
    -Dbpftool=$BPFTOOL/bpftool \
    -Dlibbpf_a=$BPFTOOL/libbpf/libbpf.a \
    -Dlibbpf_h=$BPFTOOL/libbpf/include
$ cd build
$ meson install

Note that we use libbpf which was produced as a part of bpftool build process rather than buliding libbpf directly. This is necessary because libbpf header files need to be installed for them to be in the expected relative locations.

Working with Rust Sub-projects

Each Rust sub-project is its own self-contained cargo project. When buildng as a part of this repository, meson invokes cargo with the appropriate options and environment variables to sync the build environment. When building separately by running cargo build directly in a sub-project directory, it will automatically figure out build environment. Please take a look at the scx_utils::BpfBuilder documentation for details.

For example, the following builds and runs the scx_rusty scheduler:

$ cd $SCX/scheds/rust-user/scx_rusty
$ cargo build --release
$ cargo run --release

Here too, the build step is not strictly necessary as it's implied by run.

Note that Rust userspace schedulers are published on crates.io and can be built and installed without cloning this repository as long as the necessary toolchains are available. Simply run:

$ cargo install scx_rusty

and scx_rusty will be built and installed as ~/.cargo/bin/scx_rusty.

Kernel Feature Status

The kernel feature is not yet upstream and can be found in the sched_ext repository. The followings are important branches:

• sched_ext: The main development branch. This branch periodically pulls from the bpf-next tree to stay in sync with the kernel and BPF developments.
• sched_ext-release-*: sched_ext backports on top of released kernels. We plan to maintain backports for a few recent kernel releases until sched_ext is merged upstream. Currently maintained backports:
  • sched_ext-release-v6.6
• sched_ext-vN: Patchsets posted upstream. The v4 LKML thread has high-level discussions.
  • RFC: LMKL thread
  • `sched_ext-v2': LKML thread
  • `sched_ext-v3': LKML thread
  • `sched_ext-v4': LKML thread
  • `sched_ext-v5': LKML thread

Getting in Touch

We aim to build a friendly and approachable community around sched_ext. You can reach us through the following channels:

• github: https://github.com/sched-ext/scx
• Slack: https://schedextworkspace.slack.com
• Reddit: https://reddit.com/r/sched_ext

We also hold weekly office hours every monday. Please see the #office-hours channel on slack for details.