Currently, repr() of structure and union types goes arbitrarily deep
(except for cycles). However, for lots of real-world types, this is
easily deeper than Python's recursion limit, so we can't get a useful
repr() at all:
>>> repr(prog.type('struct task_struct'))
Traceback (most recent call last):
File "<console>", line 1, in <module>
RecursionError: maximum recursion depth exceeded while getting the repr of an object
Instead, only print one level of structure and union types.
A couple of times, I've broken help(drgn) by formatting a function
signature in a way that the inspect module doesn't understand (namely,
it crashes on Enum default arguments). Let's add a simple test that the
documentation at least renders.
libdwfl is the elfutils "DWARF frontend library". It has high-level
functionality for looking up symbols, walking stack traces, etc. In
order to use this functionality, we need to report our debugging
information through libdwfl. For userspace programs, libdwfl has a much
better implementation than drgn for automatically finding debug
information from a core dump or PID. However, for the kernel, libdwfl
has a few issues:
- It only supports finding debug information for the running kernel, not
vmcores.
- It determines the vmlinux address range by reading /proc/kallsyms,
which is slow (~70ms on my machine).
- If separate debug information isn't available for a kernel module, it
finds it by walking /lib/modules/$(uname -r)/kernel; this is repeated
for every module.
- It doesn't find kernel modules with names containing both dashes and
underscores (e.g., aes-x86_64).
Luckily, drgn already solved all of these problems, and with some
effort, we can keep doing it ourselves and report it to libdwfl.
The conversion replaces a bunch of code for dealing with userspace core
dump notes, /proc/$pid/maps, and relocations.
The current array-based memory reader has a bug in the following
scenario:
prog.add_memory_segment(0xffff0000, 128, ...)
# This should replace a subset of the first segment.
prog.add_memory_segment(0xffff0020, 32, ...)
# This moves the first segment back to the front of the array.
prog.read(0xffff0000, 32)
# This finds the first segment instead of the second segment.
prog.read(0xffff0032, 32)
Fix it by using the newly-added splay tree. This also splits up the
virtual and physical memory segments into separate trees.
Currently, size_t and ptrdiff_t default to typedefs of the default
unsigned long and long, respectively, regardless of what the program
actually defines unsigned long or long as. Instead, make them refer the
whatever integer type (long, long long, or int) is the same size as the
word size.
Currently, programs can be created for three main use-cases: core dumps,
the running kernel, and a running process. However, internally, the
program memory, types, and symbols are pluggable. Expose that as a
callback API, which makes it possible to use drgn in much more creative
ways.
Similar to "libdrgn: make memory reader pluggable with callbacks", we
want to support custom type indexes (imagine, e.g., using drgn to parse
a binary format). For now, this disables the dwarf index tests; we'll
have a better way to test them later, so let's not bother adding more
test scaffolding.
I've been planning to make memory readers pluggable (in order to support
use cases like, e.g., reading a core file over the network), but the
C-style "inheritance" drgn uses internally is awkward as a library
interface; it's much easier to just register a callback. This change
effectively makes drgn_memory_reader a mapping from a memory range to an
arbitrary callback. As a bonus, this means that read callbacks can be
mixed and matched; a part of memory can be in a core file, another part
can be in the executable file, and another part could be filled from an
arbitrary buffer.
We need to set the value after we've reinitialized the object, otherwise
drgn_object_deinit() may try to free a buffer that we've already
overwritten. This also adds a test which triggers the crash.
There's a bug that we don't allow comparisons between void * and other
pointer types, so let's fix it by allowing all pointer comparisons
regardless of the referenced type. Although this isn't valid by the C
standard, GCC and Clang both allow it by default (with a warning).
The declaration file name of a DIE depends on the compilation directory,
which may not always be what the user expects. Instead, make the search
match as long as the full declaration file name ends with the given file
name. This is more convenient and more intuitive.
If libelf or libdw are statically linked or not in the usual search
path, ctypes won't be able to find them. Add the required functions to
the wrappers in the libdrgn bindings so that we can easily find them.
The current mixed Python/C implementation works well, but it has a
couple of important limitations:
- It's too slow for some common use cases, like iterating over large
data structures.
- It can't be reused in utilities written in other languages.
This replaces the internals with a new library written in C, libdrgn. It
includes Python bindings with mostly the same public interface as
before, with some important improvements:
- Types are now represented by a single Type class rather than the messy
polymorphism in the Python implementation.
- Qualifiers are a bitmask instead of a set of strings.
- Bit fields are not considered a separate type.
- The lvalue/rvalue terminology is replaced with reference/value.
- Structure, union, and array values are better supported.
- Function objects are supported.
- Program distinguishes between lookups of variables, constants, and
functions.
The C rewrite is about 6x as fast as the original Python when using the
Python bindings, and about 8x when using the C API directly.
Currently, the exposed API in C is fairly conservative. In the future,
the memory reader, type index, and object index APIs will probably be
exposed for more flexibility.
It's silly to have these trival factory methods. Instead, directly
construct with Object() and add a new NULL() function. We also
automatically import these in interactive mode.
The DIE generated by GCC for a zero-length array doesn't have a
DW_AT_upper_bound or DW_AT_count attribute; it looks identical to that
of a flexible array. See
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86985.
The semantics of zero-length and flexible arrays are slightly different
(see https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html), so there are
a few specific cases where replacing a zero-length array with a flexible
array creates an invalid type. We can tell when the type must have been
a zero-length array; in other cases, we will just assume the type is a
flexible array. This ambiguity will hopefully be fixed in a future
version of GCC.
And replace the Python implementation in ProgramObject.string_(). The
CoreReader implementation is still naive in that it reads byte-by-byte,
but this could always be improved in the future.
CompoundType.members() currently returns a list of member names;
sometimes, we actually want the type and offset. So, rename members() to
member_names(), and make members() return the type and offset, using a
newly added version of functools.partial() that caches the return value.
Currently, printing, e.g.,
typedef struct {
int counter;
} atomic_t
results in
typedef struct atomic_t
Which isn't very useful. Instead, print the former.
This is preparation for the next change, so that a Program can clean up
after itself. The Program will close the CoreReader, and the CoreReader
will close the underlying file.
For variables which are predeclared, GCC generates a DW_TAG_variable DIE
with DW_AT_name and DW_AT_declaration as well as a DW_TAG_variable DIE
without DW_AT_name but with DW_AT_specification pointing to the
declaration DIE. We should index the latter, not the former. This has a
couple of benefits: we can skip indexing variable declaration DIEs,
which contribute a lot of duplicate hash table insertions; and, we can
always get the address of a variable from DW_AT_location of the indexed
DIE instead of having to parse the symbol table.
A name and tag are not always enough to uniquely identify a type or
variable. For example, "struct workspace" in the Linux kernel can refer
to one of at least three types; fs/btrfs/{lzo,zlib,zstd}.c each have
their own struct workspace type. We can, however, also differentiate
DIEs on the file they were declared in.
The naive thing to do would be to include the file name as a string in
the hash table entry. However, that means we must allocate and
canonicalize each path in the line number program header and pay an
extra cache miss plus string comparison when adding a new entry.
We can get rid of the cache miss and string comparison if we instead map
the file name to a unique identifier. The foolproof way to do this would
be to create another big hash table of file names and use the hash table
entry index as the unique identifier. However, for this, we'd still need
to allocate and canoicalize each path as well as worry about another big
hash table.
Once we observe that we can get away with "almost certainly unique"
instead of "truly unique" identifiers, the next logical step is to just
use a hash of the file name as the identifier. With a 64-bit hash and
the ~50k files in the kernel, the probability of a collision is 1 in 10
billion. Even in the extremely unlikely event that there is a collision,
it only matters if the files with colliding names also have colliding
DIEs, which brings things pretty close to the realm of impossibility.
After this change, DwarfIndex.find() returns a list of DIEs matching the
name and tag. The callers will be updated to use the list in upcoming
changes.
ProgramObject.__dir__(), member_(), and container_of_() all check that
the relevant type is a struct or union, but they all need to allow
typedefs of structs or unions, as well.
We're currently also doing this check in __getattr__(), which is
unnecessary overhead in the common case. We can just check the exception
in the error case.
The standard library rlcompleter doesn't support expressions involving
an item lookup (e.g., x[0] or x['foo']). This is a pain for the drgn
CLI, because it's common to use prog['variable'] and want to
autocomplete it. Instead of using the standard library rlcompleter,
implement our own, cleaned up version of it with the ability to handle
expressions containing [key]. rlcompleter already allows for arbitrary
__getattr__() calls, and __getitem__() isn't any different.
This is a big change that makes EnumType have a compatible integer type
member instead of copying the fields, which ends up touching a lot of
stuff but also fixing a bunch of static typing errors.
Converting an lvalue to an operand has to do a little bit more than
remove qualifiers:
- Convert array types to pointer types
- Convert function types to pointer types
Some types don't actually have to go through find_dwarf_type(), so they
can be handled in the common code. This allows us to add a MockTypeIndex
to the tests.
The rules are really subtle and not completely specified, so hopefully
this covers all of the corner cases... This will be used for
ProgramObject operators.
Instead, take callbacks for looking up variables and reading memory.
While we're at it, get rid of TypeFactory and instead implement its
methods as functions taking a DwarfIndex.
Most of drgn.dwarf is not performance-sensitive, and the part that is
(DwarfIndex) can use some extra tuning which is easier to do in C rather
than Cython.
The lldwarf/drgn.dwarf split wasn't working out too well, and moving all
of drgn.dwarf into lldwarf (by rewriting it into C) would be way too
much work. Instead, use Cython, which results in a parser which is just
as fast but with much cleaner code overall. It also turns out lldwarf
wasn't doing GC right, so the switch also fixed that.
Resolving parameters, variables with function scope, and global
variables should work. This is just the variable resolution, no fetching
yet, but a bunch of refactors snuck in here so committing it all now.
The parsing library shouldn't really care about keeping track of these.
Instead, add __dict__ and getattr()/setattr() to all of the lldwarf
objects so higher layers can store the offset if they want.
DwarfDie is special in two ways:
1. We want to store an offset for the DIE so we know where to parse
sibling and children entries.
2. For some attributes, we need to store an offset so we know where to
find them later.
Both of these are changed to be relative to the CU rather than the
buffer.
I wrote all of this code a few months back and am just now getting
around to committing it. The low-level DWARF parsing library is pretty
solid, although it only implements a subset of DWARF so far. The CLI and
higher-level interface are experimental.
