scx_utils: Documentation and other minor updates

rust/scx_utils/README.md

@@ -0,0 +1,14 @@
+# Utility collection for sched_ext schedulers
+
+[sched_ext](https://github.com/sched-ext/scx) is a Linux kernel feature
+which enables implementing kernel thread schedulers in BPF and dynamically
+loading them.
+
+Thie crate is a collection of utilities for sched_ext scheduler
+implementations which use Rust for userspace component. This enables
+implementing hot paths in BPF while offloading colder and more complex
+operations to userspace Rust code which can be significantly more convenient
+and powerful.
+
+Please see [documentation](https://docs.rs/scx_utils/latest/scx_utils/) for
+more details.

rust/scx_utils/build.rs

@@ -1,5 +1,5 @@
 // Copyright (c) Meta Platforms, Inc. and affiliates.
-
+//
 // This software may be used and distributed according to the terms of the
 // GNU General Public License version 2.
 

rust/scx_utils/src/bpf_builder.rs

@@ -1,5 +1,5 @@
 // Copyright (c) Meta Platforms, Inc. and affiliates.
-
+//
 // This software may be used and distributed according to the terms of the
 // GNU General Public License version 2.
 
@@ -40,6 +40,172 @@ lazy_static::lazy_static! {
 }
 
 #[derive(Debug)]
+/// # Build helpers for sched_ext schedulers with Rust userspace component
+///
+/// This is to be used from `build.rs` of a cargo project which implements a
+/// [sched_ext](https://github.com/sched-ext/scx) scheduler with C BPF
+/// component and Rust userspace component. `BpfBuilder` provides everything
+/// necessary to build the BPF component and generate Rust bindings.
+/// BpfBuilder provides the followings.
+///
+/// 1. *`vmlinux.h` and other common BPF header files*
+///
+/// All sched_ext BPF implementations require `vmlinux.h` and many make use
+/// of common constructs such as
+/// [`user_exit_info`](https://github.com/sched-ext/scx/blob/main/scheds/include/common/user_exit_info.h).
+/// `BpfBuilder` makes these headers available when compiling BPF source
+/// code and generating bindings for it. The included headers can be browsed
+/// at <https://github.com/sched-ext/scx/tree/main/scheds/include>.
+///
+/// These headers can be superseded using environment variables which will
+/// be discussed later.
+///
+/// 2. *Header bindings using `bindgen`*
+///
+/// If enabled with `.enable_intf()`, the input `.h` file is processed by
+/// `bindgen` to generate Rust bindings. This is useful in establishing
+/// shared constants and data types between the BPF and user components.
+///
+/// Note that the types generated with `bindgen` are different from the
+/// types used by the BPF skeleton even when they are the same types in BPF.
+/// This is a source of ugliness and we are hoping to address it by
+/// improving `libbpf-cargo` in the future.
+///
+/// 3. *BPF compilation and generation of the skeleton and its bindings*
+///
+/// If enabled with `.enable_skel()`, the input `.bpf.c` file is compiled
+/// and its skeleton and bindings are generated using `libbpf-cargo`.
+///
+/// ## An Example
+///
+/// This section shows how `BpfBuilder` can be used in an example project.
+/// For a concrete example, take a look at
+/// [`scx_rusty`](https://github.com/sched-ext/scx/tree/main/scheds/rust-user/scx_rusty).
+///
+/// A minimal source tree using all features would look like the following:
+///
+/// ```text
+/// scx_hello_world
+/// |-- Cargo.toml
+/// |-- build.rs
+/// \-- src
+///     |-- main.rs
+///     |-- bpf_intf.rs
+///     |-- bpf_skel.rs
+///     \-- bpf
+///         |-- intf.h
+///         \-- main.c
+/// ```
+///
+/// The following three files would contain the actual implementation:
+///
+/// - `src/main.rs`: Rust userspace component which loads the BPF blob and
+/// interacts it using the generated bindings.
+///
+/// - `src/bpf/intf.h`: C header file definining constants and structs
+/// that will be used by both the BPF and userspace components.
+///
+/// - `src/bpf/main.c`: C source code implementing the BPF component -
+/// including `struct sched_ext_ops`.
+///
+/// And then there are boilerplates to generate the bindings and make them
+/// available as modules to `main.rs`.
+///
+/// - `Cargo.toml`: Includes `scx_utils` in the `[build-dependencies]`
+/// section.
+///
+/// - `build.rs`: Uses `scx_utils::BpfBuilder` to build and generate
+/// bindings for the BPF component. For this project, it can look like the
+/// following.
+///
+/// ```should_panic
+/// fn main() {
+///     scx_utils::BpfBuilder::new()
+///         .unwrap()
+///         .enable_intf("src/bpf/intf.h", "bpf_intf.rs")
+///         .enable_skel("src/bpf/main.bpf.c", "bpf")
+///         .build()
+///         .unwrap();
+/// }
+/// ```
+///
+/// - `bpf_intf.rs`: Import the bindings generated by `bindgen` into a
+/// module. Above, we told `.enable_intf()` to generate the bindings into
+/// `bpf_intf.rs`, so the file would look like the following. The `allow`
+/// directives are useful if the header is including `vmlinux.h`.
+///
+/// ```ignore
+/// #![allow(non_upper_case_globals)]
+/// #![allow(non_camel_case_types)]
+/// #![allow(non_snake_case)]
+/// #![allow(dead_code)]
+///
+/// include!(concat!(env!("OUT_DIR"), "/bpf_intf.rs"));
+/// ```
+///
+/// - `bpf_skel.rs`: Import the BPF skeleton bindings generated by
+/// `libbpf-cargo` into a module. Above, we told `.enable_skel()` to use the
+/// skeleton name `bpf`, so the file would look like the following.
+///
+/// ```ignore
+/// include!(concat!(env!("OUT_DIR"), "/bpf_skel.rs"));
+/// ```
+///
+/// ## Compiler Flags and Environment Variables
+///
+/// BPF being its own CPU architecture and independent runtime environment,
+/// build environment and steps are already rather complex. The need to
+/// interface between two different languages - C and Rust - adds further
+/// complexities. `BpfBuilder` automates most of the process. The determined
+/// build environment is recorded in the `build.rs` output and can be
+/// obtained with a command like the following:
+///
+/// ```text
+/// $ grep '^scx_utils:clang=' target/release/build/scx_rusty-*/output
+/// ```
+///
+/// While the automatic settings should work most of the time, there can be
+/// times when overriding them is necessary. The following environment
+/// variables can be used to customize the build environment.
+///
+/// - `BPF_CLANG`: The clang command to use. (Default: `clang`)
+///
+/// - `BPF_CFLAGS`: Compiler flags to use when building BPF source code. If
+///   specified, the flags from this variable are the only flags passed to
+///   the compiler. `BpfBuilder` won't generate any flags including `-I`
+///   flags for the common header files and other `CFLAGS` related variables
+///   are ignored.
+///
+/// - `BPF_BASE_CFLAGS`: Override the non-include part of cflags.
+///
+/// - `BPF_EXTRA_CFLAGS_PRE_INCL`: Add cflags before the automic include
+///   search path options. Header files in the search paths added by this
+///   variable will supercede the automatic ones.
+///
+/// - `BPF_EXTRA_CFLAGS_POST_INCL`: Add cflags after the automic include
+///   search path options. Header paths added by this variable will be
+///   searched only if the target header file can't be found in the
+///   automatic header paths.
+///
+/// - `RUSTFLAGS`: This is a generic `cargo` flag and can be useful for
+///   specifying extra linker flags.
+///
+/// A common case for using the above flags is using the latest `libbpf`
+/// from the kernel tree. Let's say the kernel tree is at `$KERNEL` and
+/// `libbpf`. The following builds `libbpf` shipped with the kernel:
+///
+/// ```test
+/// $ cd $KERNEL
+/// $ make -C tools/lib/bpf
+/// ```
+///
+/// To link the scheduler against the resulting `libbpf`:
+///
+/// ```test
+/// $ env BPF_EXTRA_CFLAGS_POST_INCL=$KERNEL/tools/lib/bpf \
+///   RUSTFLAGS="-C link-args=-lelf -C link-args=-lz -C link-args=-lzstd \
+///   -L$KERNEL/tools/lib/bpf" cargo build --release
+/// ```
 pub struct BpfBuilder {
     clang: (String, String, String), // (clang, ver, arch)
     cflags: Vec<String>,
@@ -183,7 +349,8 @@ impl BpfBuilder {
         Ok(())
     }
 
-    pub fn vmlinux_h_version() -> (String, String) {
+    /// Return `(VER, SHA1)` from which the bulit-in `vmlinux.h` is generated.
+    pub fn vmlinux_h_ver_sha1() -> (String, String) {
         let mut ar = tar::Archive::new(Self::BPF_H_TAR);
 
         for file in ar.entries().unwrap() {
@@ -244,6 +411,9 @@ impl BpfBuilder {
         Ok(cflags)
     }
 
+    /// Create a new `BpfBuilder` struct. Call `enable` and `set` methods to
+    /// configure and `build` method to compile and generate bindings. See
+    /// the struct documentation for details.
     pub fn new() -> Result<Self> {
         let out_dir = PathBuf::from(env::var("OUT_DIR")?);
 
@@ -266,16 +436,27 @@ impl BpfBuilder {
         })
     }
 
+    /// Enable generation of header bindings using `bindgen`. `@input` is
+    /// the `.h` file defining the constants and types to be shared between
+    /// BPF and Rust components. `@output` is the `.rs` file to be
+    /// generated.
     pub fn enable_intf(&mut self, input: &str, output: &str) -> &mut Self {
         self.intf_input_output = Some((input.into(), output.into()));
         self
     }
 
+    /// Enable compilation of BPF code and generation of the skeleton and
+    /// its Rust bindings. `@input` is the `.bpf.c` file containing the BPF
+    /// source code and `@output` is the `.rs` file to be generated.
     pub fn enable_skel(&mut self, input: &str, name: &str) -> &mut Self {
         self.skel_input_name = Some((input.into(), name.into()));
         self
     }
 
+    /// By default, all `.[hc]` files in the same directory as the source
+    /// BPF `.c` file are treated as dependencies and the skeleton is
+    /// regenerated if any has changed. This method replaces the automatic
+    /// dependencies with `@deps`.
     pub fn set_skel_deps<'a, I>(&mut self, deps: I) -> &mut Self
     where
         I: IntoIterator<Item = &'a str>,
@@ -365,6 +546,7 @@ impl BpfBuilder {
         Ok(())
     }
 
+    /// Build and generate the enabled bindings.
     pub fn build(&self) -> Result<()> {
         let mut deps = BTreeSet::new();
 
@@ -385,10 +567,6 @@ impl BpfBuilder {
 
 #[cfg(test)]
 mod tests {
-    use std::fs::File;
-    use std::io::BufRead;
-    use std::io::BufReader;
-
     #[test]
     fn test_bpf_builder_new() {
         let res = super::BpfBuilder::new();
@@ -396,8 +574,8 @@ mod tests {
     }
 
     #[test]
-    fn test_vmlinux_h_version() {
-        let (ver, sha1) = super::BpfBuilder::vmlinux_h_version();
+    fn test_vmlinux_h_ver_sha1() {
+        let (ver, sha1) = super::BpfBuilder::vmlinux_h_ver_sha1();
 
         println!("vmlinux.h: ver={:?} sha1={:?}", &ver, &sha1,);
 

rust/scx_utils/src/lib.rs

@@ -1,8 +1,35 @@
 // Copyright (c) Meta Platforms, Inc. and affiliates.
-
+//
 // This software may be used and distributed according to the terms of the
 // GNU General Public License version 2.
 
+//! # Utility collection for sched_ext schedulers
+//!
+//! [sched_ext](https://github.com/sched-ext/scx) is a Linux kernel feature
+//! which enables implementing kernel thread schedulers in BPF and dynamically
+//! loading them.
+//!
+//! Thie crate is a collection of utilities for sched_ext scheduler
+//! implementations which use Rust for userspace component. This enables
+//! implementing hot paths in BPF while offloading colder and more complex
+//! operations to userspace Rust code which can be significantly more convenient
+//! and powerful.
+//!
+//! The utilities can be put into two broad categories.
+//!
+//! ## Build Utilities
+//!
+//! BPF being its own CPU architecture and independent runtime environment,
+//! build environment and steps are already rather complex. The need to
+//! interface between two different languages - C and Rust - adds further
+//! complexities. This crate contains `struct BpfBuilder` which is to be
+//! used from `build.rs` and automates most of the process.
+//!
+//! ## Utilities for Rust Userspace Component
+//!
+//! Utility modules which can be useful for userspace component of sched_ext
+//! schedulers.
+
 mod bpf_builder;
 pub use bpf_builder::BpfBuilder;
 

rust/scx_utils/src/ravg.rs

@@ -1,11 +1,27 @@
 // Copyright (c) Meta Platforms, Inc. and affiliates.
-
+//
 // This software may be used and distributed according to the terms of the
 // GNU General Public License version 2.
 
-/// ravg_read() implementation for rust userland.
+//! # Running Average Utilities
+//!
+//! Rust userland utilities to access running averages tracked by BPF
+//! ravg_data. See
+//! [ravg.bpf.h](https://github.com/sched-ext/scx/blob/main/scheds/include/common/ravg.bpf.h)
+//! and
+//! [ravg_impl.bpf.h](https://github.com/sched-ext/scx/blob/main/scheds/include/common/ravg_impl.bpf.h)
+//! for details.
+
+/// Read the current running average
 ///
-/// See C ravg_read() in ravg_impl.bpf.h.
+/// Read the running average value at `@now` of ravg_data (`@val`,
+/// `@val_at`, `@old`, `@cur`) given `@half_life` and `@frac_bits`. This is
+/// equivalent to C `ravg_read()`.
+///
+/// There currently is no way to make `bindgen` and `libbpf_cargo` generated
+/// code to use a pre-existing type, so this function takes each field of
+/// struct ravg_data as a separate argument. This works but is ugly and
+/// error-prone. Something to improve in the future.
 pub fn ravg_read(
     val: u64,
     val_at: u64,