If a class inherits from more than one base class Clang Parser currently
calculates incorrect offsets for everything but the first base class.
This is owing to the fact that TypeGraph needs offsets in bits but
ClangParser is providing them in bytes.
Currently there is no testing for ClangTypeParser even though it's used in
production. This is because adding integration tests is very hard: they require
testing the build time behaviour at runtime, or else they'd be build failures
intead of test failures. There's a PR available for integration tests but it's
incomplete.
In contrast ClangTypeParser can be sort of unit tested. This follows the
structure of `test/test_drgn_parser.cpp` with some differences. There is a
tonne of boilerplate for setting up the Clang tool, and this set of testing
operates on type names instead of OID functions. The new tests are also
incredibly slow as they compile the entire `integration_test_target.cpp` (which
is huge) for every test case. I don't think this is avoidable without
compromising the separation of the tests somewhat due to the way Clang tooling
forces the code to be structured.
Currently I can't run these tests locally on a Meta devserver due to some
weirdness with the internal build and the `compile_commands.json` file. They
run in the CI and on any other open source machine though so I'm happy to merge
it - it's still useful. I'm going to close the PR to change the devserver build
given I'll be unable to follow up if it ends up being bad.
Test plan:
- CI
Summary:
Correct identification of packing.
Tests were modified to reflect the new behaviour. One test was removed as it was bogus - the flattener pass runs before the alignmentcalc pass and therefore the layout in the test could never happen (i.e. it has a hierarchy).
Reviewed By: JakeHillion
Differential Revision: D53815661
Summary: Adding support for the nullptr clang type (clang::BuiltinType::NullPtr). While there I augmented the exception message to include the type name that is missing.
Differential Revision: D53272742
Summary:
OI was previously using `std::regex_match` to match container names. This was bad because `libstdc++`'s implementation of regex is awful. In the case of limited inlining it was causing a stack overflow when running CodeGen for large types (I think types with large names but I never got to the bottom of it).
Replace this with the competent `boost::regex_match` that we already have a dependency on.
Reviewed By: ajor
Differential Revision: D53002752
Previously AlignmentCalc calculates the alignment and sets packing for every
type except a member with explicit alignment. Change this to check whether an
alignment has been previously set for a type before calculating it. Use this in
ClangTypeParser where the full alignment of the type is available.
Remove explicitly aligning members by the type because that was previously
reserved for members with explicit alignment. AlignmentCalc will correctly
align a member to the underlying type without this. Explicit member alignment
is still missing, as before this change.
Test plan:
- CI
- Too little. Gets further into a production type than without this change.
A previous change enabled running OIL tests with specific features enabled.
This highlighted that pointer code generation under TreeBuilder-v2 was very
broken. This change updates pointer code generation to work and enables the
skipped tests. All enabled tests need `expected_json_v2` added to them due to
formatting differences.
Reformatted and rewrote the basic type handler that handles primitives and
pointers. Removed the reliance on `features` to decide whether to generate for
TreeBuilder-v2 as the intermediate features have been removed. Pointers are
treated as containers with a capacity of 1 and a length of 0 if null/a cycle
and 1 if followed. This holds for void pointers where, although they aren't
followed, the length is still set.
There were a couple of other changes needed to enable these tests on TBv2 that
aren't worth their own issues and PRs, I sneaked them in here.
Extra changes:
- Added `Pointer` and `Reference` to TopoSorter so they generate
`NameProvider` instances. It might be worth visiting the graph differently
for `NameProvider` as it requires so many instances that others generators do
not. Will consider that in the future.
- Follow typedefs when calculating exclusive size for a type.
Closes#458.
Test plan:
- CI
- Enabled previously disabled tests.
Some of the logic that makes Thrift isset work for TreeBuilder-v2 in DrgnParser
(JIT OIL) wasn't ported to ClangTypeParser meaning it doesn't work in
Ahead-of-Time (AoT) OIL.
Add the template parameter name reconstruction for enum values to
ClangTypeParser.
Test plan:
- Tested with Thrift isset enabled on an internal type. Doesn't build before,
does build after.
Support the capture-thrift-isset feature with TreeBuilder-v2. Fairly minor
changes here except the type of the Enum in a template parameter now matters.
We follow the previous behaviour of capturing a value for each field in a
struct that has an `isset_bitset`. This value is a VarInt captured before the
C++ contents of the member. It has 3 values: 0 (not set), 1 (set), and 2
(unavailable). These are handled by the processor and represented in the output
as `false`, `true`, and `std::nullopt_t` respectively.
Changes:
- Add a simple Thrift isset processor before any fields that have Thrift isset.
- Store the fully qualified names of enum types in DrgnParser - it already
worked out this information anyway for naming the values and this is
consistent with classes.
- Forward all enum template parameters under their input name under the
assumption that they will all be policy type things like `IssetBitsetOption`.
This could turn out to be wrong.
Test plan:
- CI (doesn't test thrift changes but covers other regressions)
- Updated Thrift enum tests for new format.
- `FILTER='OilIntegration.*' make test` - Thrift tests failed before, succeed
after.
Array members are currently being named "TODO" (whoops). Include arrays in
TopoSorter so each one can have a `NameProvider` generated in CodeGen. Then
pass array elements through `make_field`.
Test plan:
- CI
- Add array member names to an array test.
Summary:
We have a good type representation in the Type Graph of an incomplete type and
the underlying type that represents. However, this incomplete type still ends
up in the generated code as `void` which loses information. For example, a
container that can't contain void may fail to compile because it was
initialised with `void` but really its because the type it was supposed to be
initialised with (say, `Foo`) had incomplete debug information.
This change identifies that a type is incomplete in the output by generating it
as an incomplete type `struct Incomplete<struct Foo>`. This allows us to name
the type correctly in the TreeBuilder output and filter for incomplete types,
as well as getting appropriate compiler errors if it mustn't be incomplete.
Test Plan:
- CI
- Added a unit test to namegen.
- Enabled and added an extra pointers_incomplete test.
This change is tricky to test because it isn't really user visible. The types
still use their `inputName` which is unchanged in any successful output - this
change is used so the compiler fails with a more detailed error.
Attempting to complete a type which can't be completed currently fails oilgen.
For incomplete arrays, which we know are not possible to complete, return false
deliberately.
`requireCompleteType` likely needs to not fail in all cases in the future. For
now this works.
Test plan:
- `std::unique_ptr<long[]>` used to fail the generation. Now it can
successfully codegen.
Unlike DWARF, the Clang AST is capable of correctly calculating the alignment
for each member. If we do this then AlignmentCalc doesn't traverse into the
member to attempt to calculate the alignment.
This check might be wrong if the field has explicit alignment. That case can be
covered when we have proper integration testing and a repro.
Test plan:
- Without this lots of static asserts occur. With this it's okay.
Previously ClangTypeParser assumed all RecordTypes were structs. This is fine
for structs and classes but completely incorrect for unions. Check which type
it is and give type graph the correct one.
Test plan:
- Unions static assert without this change because their size/alignment is
wrong.
Summary:
oilgen: migrate to source parsing
Using debug information generated from partial source (that is, not the final
binary) has been insufficient to generally generate OIL code.
A particular example is pointers to templates:
```cpp
#include <oi/oi.h>
template <typename T>
struct Foo {
T t;
};
template <typename T>
struct Bar {
Foo<T>& f;
};
void foo(const Bar<int>& b) {
oi::introspect(b);
}
```
The pointer/reference to `Foo<int>` appears in DWARF with
`DW_AT_declaration(true)` because it could be specialised before its usage.
However, with OIL, we are creating an implicit usage site in the
`oi::introspect` call that the compiler is unable to see.
This change reworks OILGen to work from a Clang command line rather than debug
information. We setup and run a compiler on the source, giving us access to an
AST and Semantic Analyser. We then:
- Find the `oi::introspect` template.
- Iterate through each of its callsites for their type.
- Run `ClangTypeParser::parse` on each type.
- Run codegen.
- Compile into an object file.
Having access to the semantic analyser allows us to forcefully complete a type,
as it would be if it was used in the initial code.
Test Plan:
hope
`buck2 run fbcode//mode/opt fbcode//object-introspection/oil/examples/compile-time:compile-time`
Reviewed By: tyroguru
Differential Revision: D51854477
Pulled By: JakeHillion
We previously moved container identification later in CodeGen in order
to preserve information for AlignmentCalc.
However, Flattener needs to know if a class is a container in order to
apply its special handling for this case.
This new approach moves container identification in front of Flattener,
but has Container own a type node, representing its layout. This
underlying type node can be used for calculating a container's
alignment in a later pass.
MutationTracker could only store Type nodes, while ResultTracker is
templated on the result type so can store anything.
Template the Visitor base class on the return type of visit() functions.
This sets us up for allowing visitors to return different results from
their visit() functions in the future.
This will be used in a future commit introducing DrgnExporter, where we
cache drgn_type* results while walking the type graph.
They must not appear in the final generated code as we'd end up with
invalid types with void members, e.g.:
struct Foo {
int a;
void myIncompleteMember;
int c;
};
Removing them from the type graph early also ensures that padding is
calculated correctly.
Adds the range-v3 library which supports features that otherwise wouldn't be
available until C++23 or C++26. I caught a couple of uses that suit it but this
will allow us to use more in future.
Test Plan:
- CI
Not all containers have 8-byte alignment, so if we want to avoid lots of
manual logic for calculating container alignment on a case-by-case
basis, we must calculate alignment from the member variables before the
Class nodes have been replaced by Container nodes.
Leave it to the new mutator pass IdentifyContainers to replace Class
nodes with Container nodes where appropriate.
This will allow us to run passes over the type graph before identifying
containers, and therefore before we have lost information about the
internal details of the container (e.g. alignment of member variables).
For the containers which are allowed to be declared with incomplete
types, it is only the contained types which are allowed to be
incomplete. Other template parameters (e.g. allocators) must always be
defined before use.
Lots of places rely on reference stability of ContainerInfo objects
(CodeGen's deduplication, Container nodes' containerInfo_ member).
In the key capture work, we need to be able to append to this list,
which would invalidate references before this change.
Dummy and DummyAllocator nodes had been changed to use NodeIds, but
were still printed out in full when visited for a second time.
[[nodiscard]] prevents future bugs of this type by turning them into
compilation errors.
Example of the now-fixed bug:
[1] Container: std::map<int32_t, int32_t, DummySizedOperator<0, 0, 8>, std::allocator<std::pair<int32_t const, int32_t>>>
Param
Primitive: int32_t
Param
Primitive: int32_t
Param
[2] Dummy [less<int>]
Param
...
[3] Container: std::map<int32_t, int32_t, DummySizedOperator<0, 0, 8>, std::allocator<std::pair<int32_t const, int32_t>>>
Param
Primitive: int32_t
Param
Primitive: int32_t
Param
[2]
Dummy [less<int>]
Param
...
With this patch, the second "Dummy" line will not be printed.
We only want to do the extra work if it's explicitly requested.
chaseRawPointers is already explicitly requested whenever it's needed
and readEnumValues currently isn't needed at all.
Summary:
Update to clang-15 compiler and libraries as clang-12 is ancient.
The changes to oilgen are necessary because the new internal toolchain is being more picky about linking PIC to PIC. In certain modes we build with PIC, but try to link a non-PIC oilgen artifact. Add the ability to build the oilgen artifacts with PIC which sorts this.
Reviewed By: ttreyer
Differential Revision: D46220858
Types within containers were previously named TODO. This sorts it out so
they're named as their most resolved type. The current implementation
skips Typedef names.
The TypeGraph class should only be responsible for storing Type nodes.
Traversing the graph and tracking which nodes have been visited should
not be included there.
Passes now take a NodeTrackerHolder as an input parameter, which
provides access to a zeroed-out NodeTracker.
