This implements the existing thread API methods for live processes other
than drgn_thread_stack_trace(). It also doesn't yet add support for
full-blown tracing, but it at least brings live processes to feature
parity. This is taken from the non-ptrace parts of Kevin Svetlitski's
PR #142, with some modifications.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
drgn_thread_iterator_create(), drgn_thread_iterator_next(), and
drgn_program_find_thread() have big, divergent code paths for different
targets, and this would get worse once we add live processes. Split them
up into multiple functions.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We have a few places where we format a decimal number with sprintf() or
snprintf() to a buffer with an arbitrary size. Instead of this arbitrary
size, let's add a macro to get the exact number of characters required
to format a decimal number, use it in all of these places, and make all
of these places use snprintf() just to be safe. This is more verbose but
self-documenting. The max_decimal_length() macro is inspired by
https://stackoverflow.com/a/13546502/1811295 with some improvements.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
If a TID does not exist, then linux_helper_find_task() succeeds but
returns a null pointer object. Check for that instead of returning a
bogus thread.
Fixes: 301cc767ba ("Implement a new API for representing threads")
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Currently only supported for user-space crash dumps. E.g. no support for
live user-space application debugging or kernel debugging.
Closes#144.
Signed-off-by: Mykola Lysenko <mykolal@fb.com>
LeakSanitizer reported a leak of a Python object when running
tests.test_type.TestTypeTemplateParameter.test_callable. This one is
caused by a missing Py_DECREF() in an error case.
Fixes: 352c31e1ac ("Add support for C++ template parameters")
Signed-off-by: Omar Sandoval <osandov@osandov.com>
LeakSanitizer reported a leak of a drgn_thread_iterator when running the
unit tests. The root cause is that ThreadIterator_dealloc() isn't
freeing the underlying drgn_thread_iterator().
Fixes: 301cc767ba ("Implement a new API for representing threads")
Signed-off-by: Omar Sandoval <osandov@osandov.com>
When running under fakeroot (e.g., for an RPM mock build), geteuid()
returns 0, so LinuxHelperTestCase continues and fails with a
PermissionError either when attaching to /proc/kcore or opening
/dev/loop-control. Catch the PermissionError instead of checking the
EUID.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
GCC or binutils on Fedora Rawhide for ARM seems to have a bug where
c_keywords gets placed in the .data.rel.ro section (see
https://www.airs.com/blog/archives/189):
$ readelf -s .libs/libdrgnimpl_la-language_c.o | grep -w c_keywords
475: 00000000 16 OBJECT LOCAL DEFAULT 175 c_keywords
$ readelf -S .libs/libdrgnimpl_la-language_c.o | grep -F '[175]'
[175] .data.rel PROGBITS 00000000 051f90 000010 00 WA 0 0 4
$ readelf -s .libs/_drgn.so | grep -w c_keywords
9267: 0008e84c 16 OBJECT LOCAL DEFAULT 21 c_keywords.lto_priv.0
$ readelf -S .libs/_drgn.so | grep -F '[21]'
[21] .data.rel.ro PROGBITS 0008e018 07e018 000a10 00 WA 0 0 8
This results in a crash on startup when c_keywords_init() attempts to
populate c_keywords.
While this appears to be a compiler or linker bug, I've been meaning to
replace c_keywords with a static lookup function anyways. Now that we
have gen_strswitch.py, we can use it to generate the lookup function.
Add a script, gen_c_keywords_inc_strswitch.py, which generates an array
mapping token kind to spelling, and a memswitch mapping spelling to
token kind.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We have multiple places where we match an input string against several
cases:
* drgn_lexer_c() checks C identifiers against a runtime hash table of C
keywords.
* linux_kernel_object_find() has an if-else ladder of checks for object
names.
* drgn_debug_info_find_sections() loops over an array of ELF section
names to look for sections we need.
* libdrgn/build-aux/gen_arch.awk generates a compile-time trie using
nested switch statements to match register names.
This commit adds a script, gen_strswitch.py, that can hopefully be used
to replace all of these. gen_strswitch.py generalizes the compile-time
trie idea from gen_arch.awk in a few ways:
* It has syntax and semantics based on C switch statements.
* It supports both null-terminated strings and strings with an explicit
length.
* It compresses unique substrings to calls to strcmp(), strncmp(), or
memcmp() when appropriate.
In benchmarks, this approach is more performant than the above options
as well as a candidate based on gperf, while resulting in machine code
around the same size as the straightforward if-else ladder approach.
Future commits will convert the use cases above to use this script.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Currently, Python is only required to build the Python bindings. I
originally wanted to avoid having Python as a build dependency of
libdrgn, which is why gen_arch is an AWK script. However, I want to add
another code generation script which is harder to do in AWK.
Additionally, these days more people are familiar with Python than AWK,
so let's just bite the bullet and require Python to build. No one builds
libdrgn by itself anyways.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
ASAN is incredibly useful during development, especially when dealing
with non-deterministic behavior where re-running the code under a debugger
won't necessarily reproduce the bug each time. In order not to break any
existing workflows, building with ASAN is opt-in (via --enable-asan).
Signed-off-by: Kevin Svetlitski <svetlitski@fb.com>
Running the test suite with ASAN enabled revealed that when
the current target was a userspace core dump, the `crashed_thread`
member of `struct drgn_program` was being freed twice – once indirectly
via `drgn_thread_set_deinit`, and once explicitly in `drgn_prog_deinit`.
Signed-off-by: Kevin Svetlitski <svetlitski@fb.com>
libdrgn uses rint() for formatting floating-point numbers. rint() is
provided by libm, so we need to link with -lm.
This missing library has been masked for a couple of reasons:
1. Python is linked against libm, so the drgn Python bindings implicitly
have this dependency satisfied.
2. On x86-64, GCC has a builtin implementation of rint().
This can be reproduced on x86-64 by building examples/load_debug_info
with -fno-builtin.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Peilin Ye reported a couple of related crashes in drgn caused by Linux
kernel modules which had been processed with objcopy --only-keep-debug
(although he notes that since binutils-gdb commit 8c803a2dd7d3
("elf_backend_section_flags and _bfd_elf_init_private_section_data") (in
binutils v2.35), objcopy --only-keep-debug doesn't seem to work for
kernel modules).
If given an SHT_NOBITS section, elf_getdata() returns an Elf_Data with
d_buf = NULL and d_size set to the size in the section header, which is
often non-zero. There are a few places where this can cause us to
dereference a NULL pointer:
* In relocate_elf_sections() for the relocated section data.
* In relocate_elf_sections() for the symbol table section data.
* In get_kernel_module_name_from_modinfo().
* In get_kernel_module_name_from_this_module().
Fix it by checking the section type or directly checking Elf_Data::d_buf
everywhere that could potentially get an SHT_NOBITS section. This is
based on a PR from Peilin Ye.
Closes#145.
Reported-by: Peilin Ye <peilin.ye@bytedance.com>
Signed-off-by: Omar Sandoval <osandov@osandov.com>
relocate_elf_section() shouldn't need to deal with reading the sections.
Pull that logic out into relocate_elf_file() (which will be shared with
REL-style relocations when we support those) and rename
relocate_elf_section() to apply_elf_relas().
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Most of the places where we call elf_getdata() (via read_elf_section())
deal with SHT_PROGBITS sections. elf_getdata() always returns the
literal contents of the file for SHT_PROGBITS sections (usually straight
out of the mmap'd file).
The exceptions are the SHT_RELA and SHT_SYMTAB sections in
relocate_elf_section(). For these, if the byte order or alignment of the
file do not match the host, elf_getdata() allocates a new buffer and
converts the contents. relocate_elf_section() also handles unaligned
buffers and swaps the byte order, so it mistakenly ends up with the
original byte order of the file.
Rather than removing that from relocate_elf_section(), let's avoid the
extra allocation and use elf_rawdata(), which always returns the literal
contents of the file.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Sphinx 4.4.0 warns that kyber_stack_trace.rst could use an extlink for
the link to the Kyber source code instead of hard-coding the link to the
Linux repo.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Commit 7d7aa7bf7b ("libdrgn/python: remove Type == operator") removed
the substantial part of tests.TestCase.setUp() but didn't remove the
method.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We have several for_each_entry helpers that take a type as a str or
drgn.Type and pass that directly to container_of(). If a str is given,
then this looks up the type by name for every entry. We can optimize
this and only look up the type once. In a benchmark derived from
examples/linux/fs_inodes.py, iterating over ~900k entries was reduced
from ~2 seconds to ~1.6 seconds.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We indirectly test the linked list helpers via other helpers that use
them, but let's add some dedicated tests. We test against two lists:
* "modules", which is never empty in vmtest because we load the loop
module.
* "vmcore_list", which is always empty except in the kdump kernel.
Like the red-black tree tests, these tests are written generically so we
can use a different list if necessary.
There aren't any great candidates for hlists, so we'll have to make do
with indirectly testing them for now.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
The test cases use the VMA tree and cross-reference it with
/proc/$pid/maps, but they're written so it could easily be swapped out
for another tree if necessary.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We're passing the entry type to NULL instead of the entry pointer type,
which results in errors like:
File "/home/osandov/repos/drgn/drgn/helpers/linux/rbtree.py", line 210, in rb_find
return NULL(root.prog_, type)
TypeError: 'struct vm_area_struct' value must be dictionary or mapping
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Some helpers can accept either a str or a Type. If they want to always
work with a Type internally, they need to do something like:
if isinstance(type, str):
type = prog.type(type)
Instead, let's let Program.type() accept a Type and return the exact
same type, so those helpers can unconditionally do:
type = prog.type(type)
Signed-off-by: Omar Sandoval <osandov@osandov.com>
This is a common mistake:
$ drgn core_dump
Traceback (most recent call last):
File "/usr/bin/drgn", line 33, in <module>
sys.exit(load_entry_point('drgn==0.0.16', 'console_scripts', 'drgn')())
File "/usr/lib/python3.10/site-packages/drgn/internal/cli.py", line 133, in main
runpy.run_path(args.script[0], init_globals=init_globals, run_name="__main__")
File "/usr/lib/python3.10/runpy.py", line 268, in run_path
code, fname = _get_code_from_file(run_name, path_name)
File "/usr/lib/python3.10/runpy.py", line 242, in _get_code_from_file
code = compile(f.read(), fname, 'exec')
ValueError: source code string cannot contain null bytes
The user intends to debug the core dump, but they've actually specified
the core dump as a Python script to run. The error message from the
runpy internals does not make that clear. So, let's catch this earlier
by doing a quick-and-dirty test of the file magic to see if it looks
like a core dump or other ELF file. If so, we exit with a more helpful
message:
$ drgn core_dump
error: core_dump is a core dump
Did you mean "-c core_dump"?
$ drgn /usr/bin/ls
error: /usr/bin/ls is a binary, not a drgn script
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Currently we can lookup symbols by name or address, but this will only
return one symbol, prioritizing the global symbols. However, symbols may
share the same name, and symbols may also overlap address ranges, so
it's possible for searches to return multiple results. Add functions
which can return a list of multiple matching symbols.
Signed-off-by: Stephen Brennan <stephen@brennan.io>
Now that pre-commit is added, replace the manual commands for mypy,
isort, and black with equivalent pre-commit commands. This allows us to
avoid duplicating linter arguments. It also allows us to pin the linters
used in CI by way of the .pre-commit-config.yaml file, ensuring
reproducible lint errors.
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
`pre-commit run --all-files` results in the following minor
updates, which appear to be caused by my own failure to run linters.
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
During PRs, lint and mypy errors can show up in the CI tests, which is
useful, but can introduce unnecessary churn on the PR as small lint
fixes are pushed. This commit adds (optional) support for pre-commit, a
tool which can be configured to run as a git pre-commit hook, running
linters on all changed code to catch issues before you push your code.
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Previously, drgn had no way to represent a thread – retrieving a stack
trace (the only extant thread-specific operation) was achieved by
requiring the user to directly provide a tid.
This commit introduces the scaffolding for the design outlined in
issue #92, and implements the corresponding methods for userspace core
dumps, the live Linux kernel, and Linux kernel core dumps. Future work
will build on top of this commit to support live userspace processes.
Signed-off-by: Kevin Svetlitski <svetlitski@fb.com>
The thread API needs a way to iterate over all task_structs in the
kernel. Previously, we translated the existing for_each_task helper,
which supports iterating through specific PID namespaces by walking
through the PID radix tree or PID hashtable. However, we don't need
specific namespaces for the thread API, so we can instead use the much
simpler linked lists of thread groups and threads.
Signed-off-by: Kevin Svetlitski <svetlitski@fb.com>
This reverts commit 2b47583c73. After
Kevin had completed this, we realized that there is a simpler method for
iterating through tasks from libdrgn, which the next commit will
implement. Revert the translation, but keep the improved
tests.helpers.linux.test_pid.TestPid.test_for_each_task.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Disabling SMP is necessary to work around a bug in QEMU's handling of
the capture kernel, but makes the tests run much slower. However, this
bug only appears to manifest when KVM acceleration is disabled, so the
testing harness has been modified to only disable SMP when this is true.
[Omar: use an environment variable instead of touching a file]
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Signed-off-by: Kevin Svetlitski <svetlitski@fb.com>
The majority of test cases already inherited from drgn's TestCase class.
The few outliers that inherited directly from unittest.TestCase have
been brought in line with the other tests.
Signed-off-by: Kevin Svetlitski <svetlitski@fb.com>
Now that the vmtest kernel supports kdump, add a script that can be used
to crash and enter the kdump environment on demand. Use that to crash
after running the normal test suite so that we can run tests against
/proc/vmcore. vmcore tests live in their own directory; presently the
only test is a simple sanity check that ensures we can can attach to
/proc/vmcore.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Signed-off-by: Kevin Svetlitski <svetlitski@fb.com>
We can avoid the need for the kexec tool if we load the kdump kernel
ourselves, which is much easier with kexec_file_load(). Add the config
options to enable it.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We would like to test drgn against kernel core dumps (e.g., for #129).
One option would be to include some vmcore files in the repository and
test against those. But those can be huge, and we'd need a lot of them
to test different kernel versions. Instead, we can run vmtest, enable
kdump, and trigger a crash. To do that, we first need to enable a few
kernel config options.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Kernels built without multiprocessing support don't have
__per_cpu_offset; instead, per_cpu_ptr() is a no-op. Make the helper do
the same and update the test case to work on !SMP as well.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
I missed that the kernel has an idle_task() function which uses
cpu_rq()->idle instead of idle_threads; the latter is technically
architecture-specific. So, replace idle_thread() with idle_task(), which
is architecture-independent and more consistent with the kernel.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Add helpers to convert between sockets and inodes. As an example:
>>> file = fget(task, fd)
>>> sock = SOCKET_I(file.f_inode)
>>> sock.type.value_()
2
>>> import socket
>>> int(socket.SOCK_DGRAM)
2
>>> inode = SOCK_INODE(sock)
Also add tests for the new helpers to tests/helpers/linux/test_net.py.
Signed-off-by: Peilin Ye <peilin.ye@bytedance.com>
