Now `python3 -m vmtest.manage -K --no-build` can be used to get the list
of latest kernel releases that need to be built. Also rename --dry-run
to --no-upload since --no-build is also a dry run in a way.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Slab cache merging can cause major confusion when debugging using slab
cache helpers. Add a helper to detect whether a slab cache is merged and
an explanation of the implications.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
These are ported from https://github.com/josefbacik/debug-scripts.
Signed-off-by: alexlzhu <alexlzhu@fb.com>
[Omar: make test case check for exception on SLOB]
Signed-off-by: Omar Sandoval <osandov@osandov.com>
GCC and Clang have 128-bit integer types on 64-bit targets: __int128 and
unsigned __int128. Clang additionally has N-bit integers of up to 2<<24
bits with _ExtInt(N), which was standardized in C23 as _BitInt(N).
Currently, we disallow creating objects with a >64-bit integer type. Jay
Kamat reported that this would cause errors when examining some
binaries. The reason we disallow this is that we don't have a way to
represent or do operations on >64-bit values. We could make use of a
bignum library like GMP to do this in the future.
However, for now, we can loosen this restriction and at least allow
reference and absent objects with big integer types. This requires
enforcing two things: that we never create a value object with a >64-bit
integer type, and that we never read the value of a reference object
with a >64-bit integer type.
Co-authored-by: Jay Kamat <jaygkamat@gmail.com>
Signed-off-by: Omar Sandoval <osandov@osandov.com>
This will be used for partial 128-bit object support. There are other
places that should probably be converted to use it.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
The tiny flavor was failing due to a bug fixed in Linux kernel commit
c12cd77cb028 ("mm/vmalloc: fix spinning drain_vmap_work after reading
from /proc/vmcore") (in v5.18-rc3). Now that the fix is in, we can add
5.18 with no changes required.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
The 32-bit and 64-bit variants have different register sizes, so they're
different architectures in drgn. For now, put them in the same file so
that they can share the relocation implementation. We'll need to figure
out how to handle registers later.
P.S. RISC-V has the weirdest relocations so far. /proc/kcore also
appears to be broken.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
The only relocation type I saw in Debian's kernel module debug info was
R_ARM_ABS32. R_ARM_REL32 is easy. The Linux kernel supports a bunch of
other ones that don't seem relevant to debug info.
Unfortunately, I wasn't able to test this because /proc/kcore doesn't
exist on Arm. This apparently goes all the way back to 2003:
https://lwn.net/Articles/45315/.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
The only relocation types I saw in Debian's kernel module debug info
were R_AARCH64_ABS64 and R_AARCH64_ABS32. R_AARCH64_ABS16,
R_AARCH64_PREL64, R_AARCH64_PREL32, and R_AARCH64_PREL16 are all easy.
The remaining types supported by the Linux kernel are for movw and
immediate instructions, which aren't relevant to debug info.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
The only relocation type I saw in Debian's kernel module debug info was
R_386_32. R_386_PC32 is easy. The Linux kernel also supports
R_386_PLT32, but that's the same story as R_X86_64_PLT32 in x86-64, so
we don't implement it for now.
I was torn between naming it i386, x86, or IA-32. x86 isn't immediately
clear whether x86-64 is included or not. No one other than Intel calls
it IA-32. i386 might incorrectly imply that it is strictly the original
i386 instruction set with no later extensions, but the more general
meaning is used frequently in the Linux world (e.g., Debian and QEMU
both call it i386), so I went with that in the end.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
One of the biggest things we depend on libdwfl for is applying
relocations on architectures other than x86-64. I'm exploring the
possibility of removing the libdwfl dependency, so I'm going to add
relocation implementations for more architectures, starting with ppc64.
R_PPC64_ADDR32 and R_PPC64_ADDR64 were the only ones I saw in Debian's
kernel module debug info. R_PPC64_REL32 and R_PPC64_REL64 are
straightforward. The Linux kernel also implements R_PPC64_TOC*, which
don't seem relevant to debugging information, and R_PPC64_REL24 and
R_PPC64_REL16*, which I'd prefer to have a real example of.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Implement R_X86_64_32S and R_X86_64_PC64. I haven't seen these for debug
info in the wild, but they're supported by the Linux kernel and they're
easy to support. The only other type of relocation currently supported
by the kernel is R_X86_64_PLT32, which is trickier. For kernel modules,
it's equivalent to R_X86_64_PC32 (see Linux kernel commit b21ebf2fb4cd
("x86: Treat R_X86_64_PLT32 as R_X86_64_PC32"), but that doesn't seem to
be true in general. It doesn't seem applicable to debug info sections,
so hopefully we don't need to worry about it.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
In preparation for supporting ELF relocations for more architectures,
generalize ELF relocations to handle SHT_REL sections/ElfN_Rel.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Check that size_t makes sense and make sure int_key_hash_pair() doesn't
get an integer type larger than it supports. I can't imagine either of
these failing in practice, but make our assumptions explicit.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We call hash_combine() with a uint64_t in
drgn_debug_info_module_key_hash_pair() and drgn_type_dedupe_hash_pair().
On 32-bit systems, this only uses the least-significant 32 bits. Use
hash_64_to_32() on 32-bit and hash_128_to_64() on 64-bit to ensure that
we use all bits if we're given a type larger than size_t, and sanity
check that we're not given anything larger than we support.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
If gh.download() in _download_kernel() raises an exception (e.g.,
because of an HTTP 403 error due to rate-limiting), then we will try to
exit the subprocess context managers for zstd and tar.
subprocess.Popen.__exit__() closes stdin if it is a pipe and then waits
for the subprocess. The context managers are exited in reverse order, so
first we'll try to wait for tar. But, that will never exit because its
stdin (piped from zstd) is still open. Fix it by always explicitly
closing zstd's stdin once the tar process is running.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
GitHub release assets don't allow "~" in filenames; they are changed
into ".". This means that we can't take advantage of the fact that ~
version sorts before anything, so add a sub-version to the localversion
which is 1 for the default flavor and 0 for everything else.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
The vmtest build workflow is running out of disk space because the build
directories are big, and we have a lot of them now. Default to deleting
them if uploading packages, default to keeping them in dry run mode, and
add options to choose one or the other explicitly.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
In order to test the upcoming slab helpers (#166) with the different
slab allocators, we need kernels built with SLAB and SLOB in addition to
the default SLUB. We also have some helpers which depend on SMP vs !SMP.
There will likely be other options to test in the future, like RCU
implementations.
A testing matrix of all of these options would be too big. Instead,
introduce the notion of "flavors": groups of tweaks to the vmtest kernel
configuration. For now, we have three flavors: "default" (which uses
SLUB), "alternative" (which uses SLAB), and "tiny" (which is !SMP,
!PREEMPT, and uses SLOB). As long as the options are orthogonal enough
and each one doesn't have many choices, we won't need many more flavors
than that.
I haven't decided yet exactly how to use these flavors. The easiest
thing to do is probably to run the entire Linux kernel test suite on
every flavor of every version. Since the total runtime is dominated by
boot and loading debugging information, this won't be much slower than
only running targeted tests. That will be sorted out later; for now,
this adds the build infrastructure.
While we're bumping the vmtest localversion, let's also add
CONFIG_CC_OPTIMIZE_FOR_SIZE=y.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
SLOB doesn't have /proc/slabinfo (and it can be disabled for SLUB and
SLAB on some kernel versions, too). Fall back to some known slab caches
if /proc/slabinfo doesn't exist.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
On !SMP kernels, crashing_cpu either doesn't exist or is always -1, so
drgn_program_crashed_thread() fails. Detect those cases and treat
crashing_cpu as 0.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
pre-commit/mirrors-mypy@d37f9c4f0c added
"from __future__ import annotations", which doesn't work on Python 3.6.
Python 3.6 is EOL, so it's probably time to drop it, but that should be
a different, intentional change.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Black 22.3.0 fixespsf/black#2964, which was breaking the CI with:
ImportError: cannot import name '_unicodefun' from 'click'
mypy v0.942 doesn't complain about anything new.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We currently log "Tests in VM returned $STATUS" when vmtest returns. To
find out what kernel version just returned, you have to scroll up to
find a previous log line that mentions the version. Make things more
convenient by adding the version to that message.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
tests/helpers/linux contains test cases for Linux kernel helpers and
test cases for core Linux kernel support. The latter don't make sense
there; move them to tests/linux_kernel instead, along with the
scaffolding.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
First, find the sample file relative to the test module so that tests
can be run from a different directory. Second, pass --force to zstd so
that it doesn't ignore symlinks, which is required for environments like
Buck that copy the test files as a symlinks.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
This will catch issues like the one fixed in commit 549cd7facc ("Add
tests/sample.core.zst to sdist").
Signed-off-by: Omar Sandoval <osandov@osandov.com>
If /proc/kcore is missing (e.g., because CONFIG_PROC_KCORE is not
enabled) or invalid (e.g., Docker mounts /dev/null over /proc/kcore),
then skip the tests.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Now that we have gen_strswitch.py, there's no reason to keep this AWK
script around. Replace it with a Python script that outputs a strswitch
file. This also gets rid of our gawk dependency.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
This adds an --enable-compiler-warnings flag that:
* Defines a canonical list of warnings that we enforce. For now, this is
-Wall -Wformat-overflow=2 -Wformat-truncation=2, but we can add to it
going forward.
* Enables warnings by default.
* Allows erroring on warnings. We recommend that developers use this and
use it for the CI.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Pre-read objects that are accessed multiple times to reduce reads from
/proc/kcore or the core dump. For both the lockless and old structured
variant on a live system with a full buffer, this is about 40-50%
faster.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
_get_printk_records_lockless() is guaranteed to terminate because the
loop control variable is a simple counter, but for
_get_printk_records_structured(), the log lengths essentially create a
linked list. This list could theoretically contain a cycle if the buffer
is corrupted. Add a simple check for this so we never get into an
infinite loop.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Knowing the thread or CPU that logged a specific message can be very
useful when investigating multithreaded bugs. Additionally, on kernels
since 5.10, the caller ID is always saved but only exposed to userspace
if CONFIG_PRINTK_CALLER is enabled, which distros don't currently seem
to do.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
get_printk_records() provides programmatic access to the kernel log
buffer. Add a shortcut for just getting the text with timestamps.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Previously iwyu.py overwrote the value of the `CFLAGS` environment
variable regardless of whether it was already set. This was very
frustrating when trying to set custom `CFLAGS` to be used in the
generation of compile_commands.json, so the script now only sets
`CFLAGS` if it is not already present in the environment.
Signed-off-by: Kevin Svetlitski <svetlitski@fb.com>
Get rid of the directions for running isort and black directly, since
that might run a different version than we have configured for
pre-commit. Instead, elaborate on pre-commit, including how to run
pre-commit manually.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
