As noted by commit 738261290f ("CI: temporarily disable vmtest"),
vmtest was generating too much traffic to the Dropbox shared folder that
hosted vmtest kernels. Instead, we can store kernel packages as GitHub
release assets. Update the code for downloading and uploading vmtest
assets, and also add a scheduled GitHub action to build new kernels
every Monday so I don't have to remember to do it manually.
This also drops vmtest support for 5.6-5.9, which now fail to build with
newer binutils due to the issue fixed in Linux kernel commit
1d489151e9f9 ("objtool: Don't fail on missing symbol table").
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Building drgn from an sdist currently requires autotools and gawk
because libdrgn in the sdist is more or less a git checkout. It's more
user-friendly to include the autotools output and generated code. Do
this by extending the sdist command to include a full libdrgn
distribution with `make distdir`.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We currently bundle a version of elfutils with patches to export
additional stack tracing functionality. This has a few drawbacks:
- Most of drgn's build time is actually building elfutils.
- Distributions don't like packages that bundle verions of other
packages.
- elfutils, and thus drgn, can't be built with clang.
Now that we've replaced the elfutils DWARF unwinder with our own, we
don't need the patches, so we can drop the bundled elfutils and fix
these issues.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
mypy 0.800 is stricter about reexports: "from foo import X as Y" is only
considered a reexport if X and Y are the same name (see
python/mypy#9515). mypy 0.800 fails with:
drgn/internal/cli.py:46: error: Module has no attribute "__version__"
Rename drgn.internal.version.version to __version__ so that
drgn/__init__.py can reexport it with import __version__ as __version__.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We get the version of drgn with pkg_resources.get_distribution() in two
places: setup.py (when using an sdist) and the CLI. The former causes
problems because in some cases, pip doesn't find the drgn distribution
that's currently being built. The latter adds significant latency to
startup. On my laptop, just importing pkg_resources takes 130 ms. We can
solve both of these problems by generating a file containing the version
instead.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
We currently build with CONFIG_MODULES=n for simplicity. However, this
means that we don't test kernel module support at all. Let's enable
module support. This requires changing how we distribute kernels. Now,
the /lib/modules/$(uname -r) directory (including the vmlinux and
vmlinuz) is bundled up as a tarball. We extract it, then mount it with
VirtFS, and do some extra setup for device nodes. (We lose the ability
to run kernel builds directly, but I've never actually used that
functionality.)
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Linux 5.5 built by GCC 10 doesn't boot due to new optimizations breaking
how the kernel sets up the stack canary
(https://lkml.org/lkml/2020/3/14/186 has the details). 5.5 has been EOL
since April, so the fix was never backported. We could backport it
ourselves for vmtest builds or build with GCC 9, but it's not worth the
extra effort to test an EOL kernel. Let's just drop it.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
Black was recently changed to treat a trailing comma as an indicator to
put each item/argument on its own line. We have a bunch of places where
something previously had to be split into multiple lines, then was
edited to fit on one line, but Black kept the trailing comma. Now this
update wants to unnecessarily split it back up. For now, let's get rid
of these commas. Hopefully in the future Black has a way to opt out of
this.
Signed-off-by: Omar Sandoval <osandov@osandov.com>
drgn was originally my side project, but for awhile now it's also been
my work project. Update the copyright headers to reflect this, and add a
copyright header to various files that were missing it.
Currently, vmtest runs a custom program as init in the guest that
communicates with the host via virtio-serial. This is a lot of C code
that turns out can be replaced with a shell script. In order to keep the
dependencies minimal, we use BusyBox.
I'm seeing some spurious failures on Travis where the tests pass in the
VM but setup.py reports that they failed. Log the returncode that
setup.py sees to narrow down the issue.
The current implementation of vmtest has a few issues:
1. Building drgn for each kernel version on Travis is slow, mostly
because they don't all run in parallel.
2. For local, incremental testing, recreating the filesystem image and
rebuilding drgn is slow, and syncing the code to the filesystem image
is brittle.
3. The filesystem image is the only communication channel, and reading
the exit status from the filesystem image is awkward and fragile.
4. Creating and accessing the filesystem image requires root.
This reworks vmtest to use the build on the host via VirtFS with a
simple agent on the guest that can execute arbitrary commands and return
the exit status. This has a few more moving parts but is faster and
saner overall.
I've been wanting to add type hints for the _drgn C extension for
awhile. The main blocker was that there is a large overlap between the
documentation (in docs/api_reference.rst) and the stub file, and I
really didn't want to duplicate the information. Therefore, it was a
requirement that the the documentation could be generated from the stub
file, or vice versa. Unfortunately, none of the existing tools that I
could find supported this very well. So, I bit the bullet and wrote my
own Sphinx extension that uses the stub file as the source of truth (and
subsumes my old autopackage extension and gen_docstrings script).
The stub file is probably incomplete/inaccurate in places, but this
should be a good starting point to improve on.
Closes#22.
According to PEP 440, developmental releases sort before their
corresponding final release. We want them to sort after the release, as
they are a previous release plus some number of commits. Move the commit
count to the local version, as well.
The "my_" prefix shows up as the command name in --help, which is a bit
confusing. Do what setuptools does and import the original command under
another name so the custom class can have the original name.
When investigating a reported bug, it's important to know which exact
version of drgn is being used. Let's include the git revision in the
version printed by the CLI in PEP440 format. This commit will also serve
as the 0.0.1 release.
Currently the drgn version number is defined in drgn.h.in, and configure
and setup.py both parse it out of there. However, now that we're
generating drgn.h anyways, it's easier to make configure.ac the source
of truth.
setuptools has a long-standing bug that if files are removed from the
list of sources but were included in a previous run of egg_info, they
remain in the generated list of sources (pypa/setuptools#436). This
affects egg_info and sdist. Let's work around this by removing the old
SOURCES.txt if we can recreate it from git.
In order to retrieve registers from stack traces, we need to know what
registers are defined for a platform. This adds a small DSL for defining
registers for an architecture. The DSL is parsed by an awk script that
generates the necessary tables, lookup functions, and enum definitions.
I was always doing the copy based on a comment in the equivalent
setuptools code [1]:
# Always copy, even if source is older than destination, to ensure
# that the right extensions for the current Python/platform are
# used.
On closer inspection, this isn't relevant as of PEP 3149 [2], since
extensions are tagged with the ABI version. Let's avoid the unnecessary
copies.
While we're here, let's restore self.inplace just to be safe.
1: e00f4a87ad
2: https://www.python.org/dev/peps/pep-3149/
Now that we have the bundled version of elfutils, build it from libdrgn
and link to it. We can also get rid of the elfutils version checks from
the libdrgn code.
Currently, we have a special Makefile target to output the files for a
libdrgn source tarball, and we use that for setuptools. However, the
next change is going to import elfutils, and it'd be a pain to add the
same thing for the elfutils sources. Instead, let's just use git
ls-files for everything. The only difference is that source
distributions won't have the autoconf/automake output.
autoreconf may successfully run autoconf but not automake, so we should
check that Makefile.in exists, not configure. Additionally, there also
seem to be some cases where configure fails but Makefile is still
generated. Make sure we delete the potentially-broken output if
autoreconf/configure failed.
I went back and forth on using setuptools or autotools for the Python
extension, but I eventually settled on using only setuptools after
fighting to get the two to integrate well. However, setuptools is kind
of crappy; for one, it rebuilds every source file when rebuilding the
extension, which is really annoying for development. automake is a
better designed build system overall, so let's use that for the
extension. We override the build_ext command to build using autotools
and copy things where setuptools expects them.
The current mixed Python/C implementation works well, but it has a
couple of important limitations:
- It's too slow for some common use cases, like iterating over large
data structures.
- It can't be reused in utilities written in other languages.
This replaces the internals with a new library written in C, libdrgn. It
includes Python bindings with mostly the same public interface as
before, with some important improvements:
- Types are now represented by a single Type class rather than the messy
polymorphism in the Python implementation.
- Qualifiers are a bitmask instead of a set of strings.
- Bit fields are not considered a separate type.
- The lvalue/rvalue terminology is replaced with reference/value.
- Structure, union, and array values are better supported.
- Function objects are supported.
- Program distinguishes between lookups of variables, constants, and
functions.
The C rewrite is about 6x as fast as the original Python when using the
Python bindings, and about 8x when using the C API directly.
Currently, the exposed API in C is fairly conservative. In the future,
the memory reader, type index, and object index APIs will probably be
exposed for more flexibility.
CompoundType.members() currently returns a list of member names;
sometimes, we actually want the type and offset. So, rename members() to
member_names(), and make members() return the type and offset, using a
newly added version of functools.partial() that caches the return value.
