Commit Graph

760 Commits

Author SHA1 Message Date
Omar Sandoval
2a31b052fa vmtest/manage: don't open-code dict.fromkeys() 2020-03-30 12:05:45 -07:00
Omar Sandoval
c55d199e0c vmtest/manage: always get dropbox token for --build-kernel-org
--build-kernel-org needs the token to get the available releases even if
we're not uploading.
2020-03-30 11:57:18 -07:00
Omar Sandoval
435a3440f3 vmtest/manage: chmod -x
This should just be executed as python3 -m vmtest.manage now.
2020-03-30 11:56:02 -07:00
Omar Sandoval
f73f4f4089 vmtest/manage: use util.nproc() instead of multiprocessing.cpu_count() 2020-03-30 10:52:30 -07:00
Omar Sandoval
b369ca0dd1 Split some utility functions out of setup.py
These will be shared with vmtest.
2020-03-30 10:51:04 -07:00
Omar Sandoval
0548926d96 vmtest/manage: add type annotations
And add docstrings to the functions with more complex signatures.
2020-03-30 10:12:39 -07:00
Omar Sandoval
32deb222f5 vmtest/manage: remove relocations from vmlinux
About half (!) of vmlinux comes from relocation sections (~130M out of
~250M). But, vmlinux is an ET_EXEC file, so relocations don't even apply
to it. We can massively shrink vmlinux by removing the unnecessary
sections.
2020-03-27 17:52:04 -07:00
Omar Sandoval
dda598068c vmtest/manage: move kernel config to separate file
Currently, the kernel config is embedded inside of manage.py. This is
inconvenient when building kernels manually for testing. Move it to a
separate file so that, e.g., it can be copied into the build tree
easily.
2020-03-27 17:43:21 -07:00
Omar Sandoval
739fe5686d vmtest: move to top-level directory 2020-03-27 17:13:44 -07:00
Serapheim Dimitropoulos
08193a97aa Support stack traces for running threads on kdumps 2020-03-27 16:12:03 -07:00
Omar Sandoval
d3f4f2b017 tests: create loop device for block helper tests
The upcoming vmtest rework won't have any block devices, so let's add a
loop device so that we always have a device to test with.
2020-03-27 10:09:41 -07:00
Omar Sandoval
2a0bbe7425 cli: don't fail if history can't be read or written
Instead, print a warning (unless in quiet mode).
2020-03-24 15:02:08 -07:00
Omar Sandoval
79f973007b libdrgn/python: fix reference counting on Object.type_
We need to keep the Program alive for its types to stay valid, not just
the objects the Program has pinned. (I have no idea why I changed this
in commit 565e0343ef ("libdrgn: make symbol index pluggable with
callbacks").)
2020-03-13 16:05:43 -07:00
Omar Sandoval
cae7336750 libdrgn: fix error when expecting identifier after tag in type name
We should be looking at the kind of the previous token, not the kind of
the unexpected token.

Closes #52.
2020-03-13 11:07:46 -07:00
Andrey Ignatov
618c7dcbc9 examples: Remove bpf.py
`examples/linux/bpf.py` was superseded by `tools/bpf_inspect.py` so no
reason to keep it around anymore. Remove it.

Signed-off-by: Andrey Ignatov <rdna@fb.com>
2020-03-07 11:36:11 -08:00
Andrey Ignatov
6cbdebc269 tools: Add drgn script to list progs/maps
Introduce bpf_inspect.py drgn script to list BPF programs and maps and
their properties unavailable to user space via kernel API.

The script was initially sent to kernel tree [1] but it was agreed that
drgn repo is a better place for it and it's a good idea to create
`tools/` directory in drgn to keep tools likes this. See [2] for
details.

The main use-case bpf_inspect.py covers is to show BPF programs attached
to other BPF programs via freplace/fentry/fexit mechanisms introduced
recently. There is no user-space API to get this info and, for example,
bpftool can show all BPF programs but can't show if program A replaces a
function in program B.

Example:

  % sudo tools/bpf_inspect.py p | grep test_pkt_access
     650: BPF_PROG_TYPE_SCHED_CLS          test_pkt_access
     654: BPF_PROG_TYPE_TRACING            test_main                        linked:[650->25: BPF_TRAMP_FEXIT test_pkt_access->test_pkt_access()]
     655: BPF_PROG_TYPE_TRACING            test_subprog1                    linked:[650->29: BPF_TRAMP_FEXIT test_pkt_access->test_pkt_access_subprog1()]
     656: BPF_PROG_TYPE_TRACING            test_subprog2                    linked:[650->31: BPF_TRAMP_FEXIT test_pkt_access->test_pkt_access_subprog2()]
     657: BPF_PROG_TYPE_TRACING            test_subprog3                    linked:[650->21: BPF_TRAMP_FEXIT test_pkt_access->test_pkt_access_subprog3()]
     658: BPF_PROG_TYPE_EXT                new_get_skb_len                  linked:[650->16: BPF_TRAMP_REPLACE test_pkt_access->get_skb_len()]
     659: BPF_PROG_TYPE_EXT                new_get_skb_ifindex              linked:[650->23: BPF_TRAMP_REPLACE test_pkt_access->get_skb_ifindex()]
     660: BPF_PROG_TYPE_EXT                new_get_constant                 linked:[650->19: BPF_TRAMP_REPLACE test_pkt_access->get_constant()]

It can be seen that there is a program test_pkt_access, id 650 and there
are multiple other tracing and ext programs attached to functions in
test_pkt_access.

For example the line:

     658: BPF_PROG_TYPE_EXT                new_get_skb_len                  linked:[650->16: BPF_TRAMP_REPLACE test_pkt_access->get_skb_len()]

means that BPF program new_get_skb_len, id 658, type BPF_PROG_TYPE_EXT
replaces (BPF_TRAMP_REPLACE) function get_skb_len() that has BTF id 16
in BPF program test_pkt_access, prog id 650.

Just very simple output is supported now but it can be extended in the
future if needed.

The script is extendable and currently implements two subcommands:
* prog (alias: p) to list all BPF programs;
* map (alias: m) to list all BPF maps;

Developer can simply tweak the script to print interesting pieces of
programs or maps.

More examples of output:

  % sudo tools/bpf_inspect.py p | shuf -n 3
      81: BPF_PROG_TYPE_CGROUP_SOCK_ADDR   tw_ipt_bind
      94: BPF_PROG_TYPE_CGROUP_SOCK_ADDR   tw_ipt_bind
      43: BPF_PROG_TYPE_KPROBE             kprobe__tcp_reno_cong_avoid

  % sudo tools/bpf_inspect.py m | shuf -n 3
     213: BPF_MAP_TYPE_HASH                errors
      30: BPF_MAP_TYPE_ARRAY               sslwall_setting
      41: BPF_MAP_TYPE_LRU_HASH            flow_to_snd

Help:

  % sudo tools/bpf_inspect.py
  usage: bpf_inspect.py [-h] {prog,p,map,m} ...

  drgn script to list BPF programs or maps and their properties
  unavailable via kernel API.

  See https://github.com/osandov/drgn/ for more details on drgn.

  optional arguments:
    -h, --help      show this help message and exit

  subcommands:
    {prog,p,map,m}
      prog (p)      list BPF programs
      map (m)       list BPF maps

[1] https://lore.kernel.org/bpf/20200228201514.GB51456@rdna-mbp/T/
[2] https://lore.kernel.org/bpf/20200228201514.GB51456@rdna-mbp/T/#mefed65e8a98116bd5d07d09a570a3eac46724951

Signed-off-by: Andrey Ignatov <rdna@fb.com>
2020-03-07 11:36:11 -08:00
Jay Kamat
fddabbfad0 editorconfig: Update styles for pyi and c files 2020-02-27 09:37:26 -08:00
Jay Kamat
3f870603fa libdrgn: add default language to drgn_program
For operations where we don't have a type available, we currently fall
back to C. Instead, we should guess the language of the program and use
that as the default. The heurisitic implemented here gets the language
of the CU containing "main" (except for the Linux kernel, which is
always C). In the future, we should allow manually overriding the
automatically determined language.
2020-02-26 19:55:42 -08:00
Jay Kamat
6c264b0eae libdrgn: add language to struct drgn_type
For types obtained from DWARF, we determine it from the language of the
CU. For other types, it can be specified manually or fall back to the
default (C). Then, we can use the language for operations where the type
is available.
2020-02-26 19:55:42 -08:00
Omar Sandoval
3d3c32f849 libdrgn/python: add Language to Python bindings 2020-02-26 19:55:42 -08:00
Omar Sandoval
9e2df9f217 libdrgn: put language definitions in one array
This way, languages can be identified by an index, which will be useful
for adding Python bindings for drgn_language and for adding a language
field to drgn_type.
2020-02-26 19:55:42 -08:00
Omar Sandoval
c1e1724c8e libdrgn: unify drgn_type_from_dwarf_child{,_internal}
The plain variant is a trivial wrapper around the internal variant, so
get rid of the wrapper and use the internal variant directly everywhere.
2020-02-26 19:55:42 -08:00
Omar Sandoval
efccc93e65 libdrgn: remove can_be_void from drgn_lazy_type_from_dwarf
We only lazily evaluate compound type members and function type
parameters, which are never void.
2020-02-26 19:55:42 -08:00
Omar Sandoval
fe42a71116 Add DW_LANG to generated dwarf.py
While we're here, make generate_dwarf_constants.py use the bundled
dwarf.h, generate code that black is happy with, and use the keyword
list from the standard library.
2020-02-26 19:55:42 -08:00
Omar Sandoval
8be7ae5299 drgndoc: handle forward references
String annotations (i.e., forward references) need to be parsed into an
ast node. Do it as a transformation step immediately after parsing the
source. We can also squash the constant node transformation into this
one.
2020-02-26 16:50:52 -08:00
Omar Sandoval
376979d25a Remove stray reference to gen_docstrings.py 2020-02-25 13:58:10 -08:00
Omar Sandoval
80c9fb35ff Add type hint stubs and generate documentation from them
I've been wanting to add type hints for the _drgn C extension for
awhile. The main blocker was that there is a large overlap between the
documentation (in docs/api_reference.rst) and the stub file, and I
really didn't want to duplicate the information. Therefore, it was a
requirement that the the documentation could be generated from the stub
file, or vice versa. Unfortunately, none of the existing tools that I
could find supported this very well. So, I bit the bullet and wrote my
own Sphinx extension that uses the stub file as the source of truth (and
subsumes my old autopackage extension and gen_docstrings script).

The stub file is probably incomplete/inaccurate in places, but this
should be a good starting point to improve on.

Closes #22.
2020-02-25 13:39:06 -08:00
Omar Sandoval
1b7e683930 setup.py: use contextlib.suppress(FileNotFoundError) for SOURCES.txt 2020-02-21 11:09:04 -08:00
Omar Sandoval
52e9b2f8d8 drgn 0.0.3 2020-02-21 10:37:02 -08:00
Serapheim Dimitropoulos
e3789512ab Fix leak in kdump code
prog->kdump_ctx is never really initialized, and the kdump_ctx struct
allocated in drgn_program_set_kdump() is leaked.
2020-02-20 15:49:44 -08:00
Omar Sandoval
3b6494bf8c docs: fix Program.symbol() raises description
We missed updating this in commit 31d544949f ("libdrgn: Add
find_symbol_by_name to look up ELF symbols").
2020-02-20 15:25:18 -08:00
Omar Sandoval
9246094cdc libdrgn: use dwfl_frame_register() instead of dwfl_frame_eval_expr()
I thought I'd be able to avoid adding a separate API for register values
and reuse dwfl_frame_eval_expr(), but this doesn't work if the frame is
missing debug information but has known register values (e.g., if the
program crashed with an invalid instruction pointer).
2020-02-20 14:13:08 -08:00
Omar Sandoval
016189f477 Update elfutils with improved stack frame interface
Rebase on master, add the improved
dwfl_frame_module/dwfl_frame_dwarf_frame patch, and add the
dwfl_frame_register patch.

Based on:

889edd912 PR25365: debuginfod-client: restrict cleanup to client-pattern files

With the following patches:

configure: Add --disable-programs
configure: Add --disable-shared
libdwfl: add interface for attaching to/detaching from threads
libdwfl: add interface for getting Dwfl_Module and Dwarf_Frame for Dwfl_Frame
libdwfl: export __libdwfl_frame_reg_get as dwfl_frame_register
libdwfl: add interface for evaluating DWARF expressions in a frame
2020-02-20 13:49:10 -08:00
Omar Sandoval
a5cd92f24e libdrgn: make vmcoreinfo accessible before loading debug info
UTS_RELEASE is currently only accessible once debug info is loaded with
prog.load_debug_info(main=True). This makes it difficult to get the
release, find the appropriate vmlinux, then load the found vmlinux. We
can add vmcoreinfo_object_find as part of set_core_dump(), which makes
it possible to do the following:

  prog = drgn.Program()
  prog.set_core_dump(core_dump_path)
  release = prog['UTS_RELEASE'].string_()
  vmlinux_path = find_vmlinux(release)
  prog.load_debug_info([vmlinux_path])

The only downside is that this ends up using the default definition of
char rather than what we would get from the debug info, but that
shouldn't be a big problem.
2020-02-19 12:11:45 -08:00
Omar Sandoval
cc18d9e502 libdrgn: add UTS_RELEASE to vmcoreinfo_object_find
The osrelease is accessible via init_uts_ns.name.release, but we can
also get it straight out of vmcoreinfo, which will be useful for the
next change. UTS_RELEASE is the name of the macro defined in the kernel.
2020-02-19 12:11:20 -08:00
Omar Sandoval
da26e814c2 docs: add missing mentions of class types
Commit 0df2152307 ("Add basic class type support") missed updating the
API reference.
2020-02-12 15:40:19 -08:00
Omar Sandoval
26ef465007 libdrgn/python: add proper type for members and parameters
This continues the conversion from the last commit. Members and
parameters are basically the same, so we can do them together. Unlike
enumerators, these don't make sense to unpack or access as sequences.
2020-02-12 15:40:19 -08:00
Omar Sandoval
7c70a1a384 libdrgn/python: add proper type for enumerators
Currently, type members, enumerators, and parameters are all represented
by tuples in the Python bindings. This is awkward to document and
implement. Instead, let's replace these tuples with proper types,
starting with the easiest one, TypeEnumerator. This one still makes
sense to treat as a sequence so that it can be unpacked as (name,
value).
2020-02-12 15:37:41 -08:00
Omar Sandoval
bda1426ae9 python: use tuple instead of list for __all__
It makes more sense semantically for __all__ to be immutable (except for
drgn.helpers.linux.__all__, which is built dynamically). While we're
here, sort the names, too.
2020-02-12 14:19:07 -08:00
Jay Kamat
23c7d34099 helpers: Add get_config helper for getting kconfig map 2020-02-12 14:06:49 -08:00
Jay Kamat
31d544949f libdrgn: Add find_symbol_by_name to look up ELF symbols 2020-02-12 14:06:49 -08:00
Jay Kamat
054cb54a01 libdrgn: Rename find_symbol to find_symbol_by_address 2020-02-12 14:06:49 -08:00
Omar Sandoval
9de2cc8410 libdrgn/python: make Object.__index__() TypeError message clearer
Currently, we print:

>>> prog.symbol(prog['init_task'])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: cannot convert 'struct task_struct' to index

It's not obvious what it means to convert to an index. Instead, let's
use the error message raised by operator.index():

TypeError: 'struct task_struct' object cannot be interpreted as an integer
2020-02-11 09:19:53 -08:00
Omar Sandoval
4adc691622 helpers: add helpers for finding user_structs 2020-02-10 18:11:25 -08:00
Omar Sandoval
71ce81ecbf travis.yml: add 5.6 to vmtest kernels and remove 5.3
v5.6-rc1 was released today, and 5.3 was recently EOL'd. (It might be
worthwhile to continue testing EOL kernels, but for now I don't want
Travis runs to take forever).
2020-02-10 12:43:25 -08:00
Omar Sandoval
b5232d944d tests: fix skipping Linux kernel tests if missing debug info
I forgot to name the caught exception.
2020-02-07 14:45:34 -08:00
Omar Sandoval
401a310fed python: add missed return type annotation for execscript() 2020-02-07 14:31:32 -08:00
Omar Sandoval
43277f59f2 docs: fix OutOfBoundsError exception directive
Commit ad82e9623a ("Introduce OutOfBoundsError") missed a space in the
documentation for OutOfBoundsError. Fix it.
2020-02-07 14:29:14 -08:00
Omar Sandoval
6d03d87978 docs: document that sizeof() argument is positional-only 2020-02-07 14:28:14 -08:00
Jay Kamat
ccc17cb846 .editorconfig: Add editorconfig for configuring style 2020-02-07 14:26:27 -08:00