build: Use workspace to group rust sub-projects

meson build script was building each rust sub-project under rust/ and
scheds/rust/ separately. This means that each rust project is built
independently which leads to a couple problems - 1. There are a lot of
shared dependencies but they have to be built over and over again for each
proejct. 2. Concurrency management becomes sad - we either have to unleash
multiple cargo builds at the same time possibly thrashing the system or
build one by one.

We've been trying to solve this from meson side in vain. Thankfully, in
issue #546, @vimproved suggested using cargo workspace which makes the
sub-projects share the same target directory and built together by the same
cargo instance while still allowing each project to behave independently for
development and publishing purposes.

Make the following changes:

- Create two cargo workspaces - one under rust/, the other under
  scheds/rust/. Each contains all rust projects underneath it.

- Don't let meson descend into rust/. These are libraries used by the rust
  schedulers. No need to build them from meson. Cargo will build them as
  needed.

- Change the rust_scheds build target to invoke `cargo build` in
  scheds/rust/ and let cargo do its thing.

- Remove per-scheduler meson.build files and instead generate custom_targets
  in scheds/rust/meson.build which invokes `cargo build -p $SCHED`.

- This changes rust binary directory. Update README and
  meson-scripts/install_rust_user_scheds accordingly.

- Remove per-scheduler Cargo.lock as scheds/rust/Cargo.lock is shared by all
  schedulers now.

- Unify .gitignore handling.

The followings are build times on Ryzen 3975W:

Before:
  ________________________________________________________
  Executed in  165.93 secs    fish           external
     usr time   40.55 mins    2.71 millis   40.55 mins
     sys time    3.34 mins   36.40 millis    3.34 mins

After:
  ________________________________________________________
  Executed in   36.04 secs    fish           external
     usr time  336.42 secs    0.00 millis  336.42 secs
     sys time   36.65 secs   43.95 millis   36.61 secs

Wallclock time is reduced 5x and CPU time 7x.
This commit is contained in:
Tejun Heo 2024-08-25 00:03:41 -10:00
parent c93191a213
commit 43950c65bd
36 changed files with 191 additions and 9969 deletions

View File

@ -244,9 +244,7 @@ For example, here, the `scx_simple` binary can be found at
`$SCX/build/scheds/c/scx_simple`.
For Rust userspace schedulers such as the ones under `scheds/rust`, the
same directory under the build root is used as the cargo build target
directory. Thus, here, the `scx_rusty` binary can be found at
`$SCX/build/scheds/rust/scx_rusty/release/scx_rusty`.
`scx_rusty` binary can be found at `$SCX/build/scheds/rust/release`.
### SCX specific build options
@ -264,7 +262,6 @@ options can be used in such cases.
- 'cargo_home': 'CARGO_HOME env to use when invoking cargo'
- `offline`: 'Compilation step should not access the internet'
- `enable_rust`: 'Enable the build of rust sub-projects'
- `serialize`: 'Enable/disable the sequential build of the schedulers. Set this to false if you need to build just one scheduler.'
For example, let's say you want to use `bpftool` and `libbpf` shipped in the
kernel tree located at `$KERNEL`. We need to build `bpftool` in the kernel

View File

@ -4,8 +4,8 @@ set -e
for manifest in "$MESON_SOURCE_ROOT"/scheds/rust/*/Cargo.toml; do
source_dir="${manifest%/Cargo.toml}"
target_dir="${MESON_BUILD_ROOT}${source_dir#${MESON_SOURCE_ROOT}}"
name="${target_dir##*/}"
target_dir="${MESON_BUILD_ROOT}/scheds/rust"
name="${source_dir##*/}"
# Skip scx_mitosis
if [ "$name" = "scx_mitosis" ]; then

View File

@ -12,8 +12,6 @@ cc = meson.get_compiler('c')
enable_rust = get_option('enable_rust')
serialize = get_option('serialize')
bpf_clang = find_program(get_option('bpf_clang'))
run_veristat = find_program(join_paths(meson.current_source_dir(),
@ -338,9 +336,6 @@ run_target('veristat_diff', command: [run_veristat_diff, meson.current_build_dir
get_option('veristat_scheduler'), get_option('kernel'),
get_option('veristat_diff_dir')])
if enable_rust
subdir('rust')
endif
subdir('scheds')
if enable_stress

View File

@ -14,8 +14,6 @@ option('offline', type: 'boolean', value: 'false',
description: 'Compilation step should not access the internet')
option('enable_rust', type: 'boolean', value: 'true',
description: 'Enable rust sub-projects')
option('serialize', type: 'boolean', value: 'true',
description: 'Serialize the build of all schedulers')
option('enable_stress', type: 'boolean', value: 'true',
description: 'Enable stress tests')
option('kernel', type: 'string', value: 'vmlinuz',

3
rust/Cargo.toml Normal file
View File

@ -0,0 +1,3 @@
[workspace]
members = ["scx_utils", "scx_stats", "scx_stats/scx_stats_derive", "scx_rustland_core"]
resolver = "2"

View File

@ -1,3 +0,0 @@
subdir('scx_utils')
subdir('scx_stats')
subdir('scx_rustland_core')

View File

@ -1 +0,0 @@
Cargo.lock

View File

@ -1,8 +0,0 @@
custom_target('scx_rustland_core',
output: '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
command: [cargo, 'build', '--manifest-path=@INPUT@', '--target-dir=@OUTDIR@',
cargo_build_args],
env: cargo_env,
depends: [libbpf, bpftool_target],
build_by_default: true)

View File

@ -1,5 +1,3 @@
workspace = { members = ["scx_stats_derive"] }
[package]
name = "scx_stats"
version = "1.0.3"

View File

@ -1,7 +0,0 @@
custom_target('scx_stats',
output: '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
command: [cargo, 'build', '--manifest-path=@INPUT@', '--target-dir=@OUTDIR@',
'--all', cargo_build_args],
env: cargo_env,
build_by_default: true)

View File

@ -1,2 +0,0 @@
Cargo.lock
target

View File

@ -1,8 +0,0 @@
custom_target('scx_utils',
output: '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
command: [cargo, 'build', '--manifest-path=@INPUT@', '--target-dir=@OUTDIR@',
cargo_build_args],
env: cargo_env,
depends: [libbpf, bpftool_target],
build_by_default: true)

2
scheds/rust/.gitignore vendored Normal file
View File

@ -0,0 +1,2 @@
*/src/bpf/.output
target

View File

@ -201,9 +201,9 @@ dependencies = [
[[package]]
name = "cc"
version = "1.1.13"
version = "1.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72db2f7947ecee9b03b510377e8bb9077afa27176fdbff55c51027e976fdcc48"
checksum = "50d2eb3cd3d1bf4529e31c215ee6f93ec5a3d536d9f578f93d9d33ee19562932"
dependencies = [
"shlex",
]
@ -443,9 +443,9 @@ dependencies = [
[[package]]
name = "fastrand"
version = "2.1.0"
version = "2.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9fc0510504f03c51ada170672ac806f1f105a88aa97a5281117e1ddc3368e51a"
checksum = "e8c02a5121d4ea3eb16a80748c74f5549a5665e4c21333c6098f283870fbdea6"
[[package]]
name = "fb_procfs"
@ -599,9 +599,9 @@ checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55"
[[package]]
name = "libbpf-cargo"
version = "0.24.2"
version = "0.24.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "457ba100661d900812309eb2f79e1307af0a17cfba7f1e9ae382b071e0c6cace"
checksum = "3bf2872772e90abbf8a1e0aa5eb53b022a77deb48e7bec00468a610cfb422f58"
dependencies = [
"anyhow",
"cargo_metadata 0.15.4",
@ -617,9 +617,9 @@ dependencies = [
[[package]]
name = "libbpf-rs"
version = "0.24.2"
version = "0.24.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ecc6eb0468f4b691117c49673da2b833bd2fefbf05a4876a86ad93defd088f12"
checksum = "f52b1c660da083a765c10249e5b563d5adc12183870277d6a0d10820b222b025"
dependencies = [
"bitflags 2.6.0",
"libbpf-sys",
@ -866,6 +866,12 @@ version = "0.3.30"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d231b230927b5e4ad203db57bbcbee2802f6bce620b1e4a9024a07d94e2907ec"
[[package]]
name = "plain"
version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6"
[[package]]
name = "portable-atomic"
version = "1.7.0"
@ -880,9 +886,9 @@ checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391"
[[package]]
name = "prettyplease"
version = "0.2.20"
version = "0.2.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5f12335488a2f3b0a83b14edad48dca9879ce89b2edd10e80237e4e852dd645e"
checksum = "a909e6e8053fa1a5ad670f5816c7d93029ee1fa8898718490544a6b0d5d38b3e"
dependencies = [
"proc-macro2",
"syn",
@ -914,9 +920,9 @@ dependencies = [
[[package]]
name = "quote"
version = "1.0.36"
version = "1.0.37"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0fa76aaf39101c457836aec0ce2316dbdc3ab723cdda1c6bd4e6ad4208acaca7"
checksum = "b5b9d34b8991d19d98081b46eacdd8eb58c6f2b201139f7c5f643cc155a633af"
dependencies = [
"proc-macro2",
]
@ -990,6 +996,15 @@ version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a66a03ae7c801facd77a29370b4faec201768915ac14a721ba36f20bc9c209b"
[[package]]
name = "rlimit"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3560f70f30a0f16d11d01ed078a07740fe6b489667abc7c7b029155d9f21c3d8"
dependencies = [
"libc",
]
[[package]]
name = "rustc-hash"
version = "1.1.0"
@ -1030,6 +1045,117 @@ dependencies = [
"winapi-util",
]
[[package]]
name = "scx_bpfland"
version = "1.0.3"
dependencies = [
"anyhow",
"clap",
"crossbeam",
"ctrlc",
"libbpf-rs",
"log",
"rlimit",
"scx_stats",
"scx_stats_derive",
"scx_utils",
"serde",
"simplelog",
]
[[package]]
name = "scx_lavd"
version = "1.0.3"
dependencies = [
"anyhow",
"bitvec",
"clap",
"crossbeam",
"ctrlc",
"fb_procfs",
"hex",
"libbpf-rs",
"libc",
"log",
"nix 0.29.0",
"ordered-float 3.9.2",
"plain",
"rlimit",
"scx_stats",
"scx_stats_derive",
"scx_utils",
"serde",
"simplelog",
"static_assertions",
]
[[package]]
name = "scx_layered"
version = "1.0.3"
dependencies = [
"anyhow",
"bitvec",
"chrono",
"clap",
"crossbeam",
"ctrlc",
"fb_procfs",
"lazy_static",
"libbpf-rs",
"libc",
"log",
"scx_stats",
"scx_stats_derive",
"scx_utils",
"serde",
"serde_json",
"simplelog",
]
[[package]]
name = "scx_rlfifo"
version = "1.0.3"
dependencies = [
"anyhow",
"ctrlc",
"libbpf-rs",
"libc",
"plain",
"scx_rustland_core",
"scx_utils",
]
[[package]]
name = "scx_rustland"
version = "1.0.3"
dependencies = [
"anyhow",
"clap",
"ctrlc",
"fb_procfs",
"libbpf-rs",
"libc",
"log",
"ordered-float 3.9.2",
"plain",
"scx_rustland_core",
"scx_utils",
"simplelog",
]
[[package]]
name = "scx_rustland_core"
version = "1.0.3"
dependencies = [
"anyhow",
"libbpf-rs",
"libc",
"plain",
"scx_utils",
"tar",
"walkdir",
]
[[package]]
name = "scx_rusty"
version = "1.0.3"
@ -1115,18 +1241,18 @@ dependencies = [
[[package]]
name = "serde"
version = "1.0.208"
version = "1.0.209"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cff085d2cb684faa248efb494c39b68e522822ac0de72ccf08109abde717cfb2"
checksum = "99fce0ffe7310761ca6bf9faf5115afbc19688edd00171d81b1bb1b116c63e09"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.208"
version = "1.0.209"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24008e81ff7613ed8e5ba0cfaf24e2c2f1e5b8a0495711e44fcd4882fca62bcf"
checksum = "a5831b979fd7b5439637af1752d535ff49f4860c0f341d1baeb6faf0f4242170"
dependencies = [
"proc-macro2",
"quote",
@ -1135,9 +1261,9 @@ dependencies = [
[[package]]
name = "serde_json"
version = "1.0.125"
version = "1.0.127"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "83c8e735a073ccf5be70aa8066aa984eaf2fa000db6c8d0100ae605b366d31ed"
checksum = "8043c06d9f82bd7271361ed64f415fe5e12a77fdb52e573e7f06a516dea329ad"
dependencies = [
"itoa",
"memchr",
@ -1227,9 +1353,9 @@ checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
[[package]]
name = "syn"
version = "2.0.75"
version = "2.0.76"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f6af063034fc1935ede7be0122941bafa9bacb949334d090b77ca98b5817c7d9"
checksum = "578e081a14e0cefc3279b0472138c513f37b41a08d5a3cca9b6e4e8ceb6cd525"
dependencies = [
"proc-macro2",
"quote",

5
scheds/rust/Cargo.toml Normal file
View File

@ -0,0 +1,5 @@
[workspace]
members = ["scx_lavd", "scx_bpfland", "scx_rustland", "scx_rlfifo",
"scx_rusty",
"scx_layered"] # scx_mitosis temporarily excluded
resolver = "2"

View File

@ -1,31 +1,29 @@
# the previous scheduler in the compile sequence
sched = []
# Since meson and cargo tries build in parallel, this can cause significant load
# when meson tries to launch N instances of cargo and cargo tries to compile N files
# in parallel (N*N compiler instances in total).
#
# To prevent this from happening, we try to force meson to build them sequentially
# by making the "current" scheduler depend on another scheduler.
# To add a new scheduler, assign the output of your custom_target to sched
# and add sched as a dependency to your custom_target. For example:
#
# sched = custom_target('scx_mysched',
# ...
# depends: [mydep, sched],
# build_always_stale: true)
subdir('scx_layered')
#subdir('scx_mitosis') # Temporarily excluded until cgroup support lands in the kernel
subdir('scx_rusty')
subdir('scx_rustland')
subdir('scx_rlfifo')
subdir('scx_bpfland')
subdir('scx_lavd')
sched_deps = [libbpf, bpftool_target]
cargo_cmd = [cargo, 'build', '--manifest-path=@INPUT@', '--target-dir=@OUTDIR@',
cargo_build_args]
# the target to compile all rust schedulers
custom_target('rust_scheds',
input: 'meson.build',
output: '@PLAINNAME@.__PHONY__',
command: ['touch', '@PLAINNAME@.__PHONY__'],
depends: sched,
build_by_default: true)
input: 'Cargo.toml',
command: cargo_cmd,
env: cargo_env,
depends: sched_deps,
build_by_default: true,
build_always_stale: true)
# per-scheduler targets
rust_scheds = ['scx_lavd', 'scx_bpfland', 'scx_rustland', 'scx_rlfifo',
'scx_rusty',
'scx_layered'] # scx_mitosis temporarily excluded
foreach sched: rust_scheds
custom_target(sched,
output: sched + '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
command: cargo_cmd + ['-p', sched],
env: cargo_env,
depends: sched_deps,
build_by_default: false,
build_always_stale: true)
endforeach

File diff suppressed because it is too large Load Diff

View File

@ -1,9 +1,3 @@
if serialize
sched_deps = [libbpf, bpftool_target, sched]
else
sched_deps = [libbpf, bpftool_target]
endif
sched = custom_target('scx_bpfland',
output: '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
@ -11,4 +5,5 @@ sched = custom_target('scx_bpfland',
cargo_build_args],
env: cargo_env,
depends: sched_deps,
build_by_default: false,
build_always_stale: true)

View File

@ -1,2 +0,0 @@
src/bpf/.output
target

File diff suppressed because it is too large Load Diff

View File

@ -1,9 +1,3 @@
if serialize
sched_deps = [libbpf, bpftool_target, sched]
else
sched_deps = [libbpf, bpftool_target]
endif
sched = custom_target('scx_lavd',
output: '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
@ -11,4 +5,5 @@ sched = custom_target('scx_lavd',
cargo_build_args],
env: cargo_env,
depends: sched_deps,
build_by_default: false,
build_always_stale: true)

View File

@ -1,2 +0,0 @@
src/bpf/.output
target

File diff suppressed because it is too large Load Diff

View File

@ -1,9 +1,3 @@
if serialize
sched_deps = [libbpf, bpftool_target, sched]
else
sched_deps = [libbpf, bpftool_target]
endif
sched = custom_target('scx_layered',
output: '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
@ -11,4 +5,5 @@ sched = custom_target('scx_layered',
cargo_build_args],
env: cargo_env,
depends: sched_deps,
build_by_default: false,
build_always_stale: true)

View File

@ -1,2 +0,0 @@
src/bpf/.output
target

File diff suppressed because it is too large Load Diff

View File

@ -1,9 +1,3 @@
if serialize
sched_deps = [libbpf, bpftool_target, sched]
else
sched_deps = [libbpf, bpftool_target]
endif
sched = custom_target('scx_mitosis',
output: '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
@ -11,4 +5,5 @@ sched = custom_target('scx_mitosis',
cargo_build_args],
env: cargo_env,
depends: sched_deps,
build_by_default: false,
build_always_stale: true)

View File

@ -1,5 +1,3 @@
src/bpf/.output
intf.h
main.bpf.c
bpf.rs
target

File diff suppressed because it is too large Load Diff

View File

@ -1,9 +1,3 @@
if serialize
sched_deps = [libbpf, bpftool_target, sched]
else
sched_deps = [libbpf, bpftool_target]
endif
sched = custom_target('scx_rlfifo',
output: '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
@ -11,4 +5,5 @@ sched = custom_target('scx_rlfifo',
cargo_build_args],
env: cargo_env,
depends: sched_deps,
build_by_default: false,
build_always_stale: true)

View File

@ -1,5 +1,3 @@
src/bpf/.output
intf.h
main.bpf.c
bpf.rs
target

File diff suppressed because it is too large Load Diff

View File

@ -1,9 +1,3 @@
if serialize
sched_deps = [libbpf, bpftool_target, sched]
else
sched_deps = [libbpf, bpftool_target]
endif
sched = custom_target('scx_rustland',
output: '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
@ -11,4 +5,5 @@ sched = custom_target('scx_rustland',
cargo_build_args],
env: cargo_env,
depends: sched_deps,
build_by_default: false,
build_always_stale: true)

View File

@ -1,2 +0,0 @@
src/bpf/.output
target

View File

@ -1,9 +1,3 @@
if serialize
sched_deps = [libbpf, bpftool_target, sched]
else
sched_deps = [libbpf, bpftool_target]
endif
sched = custom_target('scx_rusty',
output: '@PLAINNAME@.__PHONY__',
input: 'Cargo.toml',
@ -11,4 +5,5 @@ sched = custom_target('scx_rusty',
cargo_build_args],
env: cargo_env,
depends: sched_deps,
build_by_default: false,
build_always_stale: true)