Summary:
Update `OIGenerator` and out BUCK stuff for compile time OIL with OIL v2. Main things:
- Switch `OIGenerator` from the `getObjectSize` call to the new `introspect` call.
- Switch from looking at template parameters to looking at function parameters, as this was exposing a bug in our elfutils/drgn and this way it's the same as OID.
- Migrate `OIGenerator` to CodeGen v2 and update CodeGen v2 to accept a linkage name.
- Update the compile time example to be the same as the JIT example, using the new interface and the JSON exporter.
- Clean up the `ObjectIntrospection.h` header.
Differential Revision: D48687728
fbshipit-source-id: 2c3c041fd1b6499c5e02eb5e2082a977bfa529d7
Types within containers were previously named TODO. This sorts it out so
they're named as their most resolved type. The current implementation
skips Typedef names.
The TypeGraph class should only be responsible for storing Type nodes.
Traversing the graph and tracking which nodes have been visited should
not be included there.
Passes now take a NodeTrackerHolder as an input parameter, which
provides access to a zeroed-out NodeTracker.
Type Graph deduplicates and modifies names to better fit the generated
code, for example `int32_t[4]` becomes `OIArray<int32_t, 4>` and `struct
MyStruct` might become `struct MyStruct_0`.
Add an `inputName` which better represents the original input code which
can be used when building the tree.
This removes Printer's legacy behaviour of generating an ID for each
node as it gets printed. This old method meant that if new nodes were
added to or removed from a graph, every ID after the new/removed node
would change.
Now IDs are stable so it is easier to follow specific nodes through
multiple transformation passes in CodeGen.
Names which were generated on-demand are now stored in member variables,
which are set during the ctor and can be regenerated when required (by
NameGen).
We previously only marked as packed if there was no tail padding, which
was not a sufficient condition.
The new AlignmentCalcTest.PackedMembers test case is an example which
would previously not have been marked as packed.
CodeGen v1 does not record anything for pointers to incomplete types.
Not even the address, as is done for other pointers.
Introduce a new Primitive type "Incomplete". This behaves identically to
"Void", but allows us to tell whether a type was defined as void or if
it ended up like that because of incomplete DWARF information.
This extracts the compatibility logic from AddPadding, which allows for it to be
simplified and will make it easier to extend and eventually remove in the
future. No functional changes.
This lets us remove fields from types when they are no longer needed,
speeding up later passes.
A secondary benefit of pruning unused types means that we sometimes
remove types for which we can't generate correct C++ code. This can
allow us to CodeGen for complex types which reference these broken types
without actually requiring them (e.g. as template parameters).
Add a new feature flag "prune-type-graph" to control this pass. It makes
sense to prune most of the time, but for testing CodeGen functionality
on a wider range of types, it will be useful to have the option to not
prune.
In general, we can't tell which member is active in a union so it is not
safe to try and measure any of them.
Explicitly set the alignment of unions (and structs/classes) in CodeGen
if it is available, as the C++ compiler can no longer infer it from the
members.
Also reshuffle CodeGen's passes to fix an alignment bug with removed
members.
Change RemoveMembers to actually remove members instead of replacing
them with padding. AddPadding must be run afterwards to fill in the
gaps.
The underlying type of bitfield is important to the size of a struct:
struct Foo { int64_t bitfield : 1; };
struct Bar { int8_t bitfield : 1; };
sizeof(Foo) = 8;
sizeof(Bar) = 1;
Previously this code would not have removed all members which it was
supposed to.
Also remove some now-redundant code from TypeIdentifier. RemoveIgnored
will take over the responsibility of removing members from classes.
TypeGraphParser parses a textual type graph, as emitted by Printer.
It also doubles as a way of ensuring that Printer displays all
information about a type graph, to aid with debugging.
Convert Flattener unit tests over to this new framework as a first step.
Summary:
Stubbing tuple without alignment info can cause failures. I added a test
case for this which failed before this fix but works now.
Test Plan:
Ran the test.
Summary:
If a container has 0 template params (e.g. IOBuf, IOBufQueue), it wasn't
being added to containerTypeMap. This causes the deserialization to go wrong
as TreeBuilder doesn't treat the types as container
Test Plan:
make test
Use fully qualified names to determine if a class is really the child of
our type. It may be that it is the child of another type with an
identical name in another namespace.
visit(Type&) and visit(Type*) were helper functions than did cycle
detection and provided nicer syntax for the type.accept(*this) calls.
However, because they overloaded the visit() function, it was easy to
accidentally call a concrete visit method (e.g. visit(Class&)) instead
of the virtual-dispatching visit(Type&).
By changing the name of this wrapper, it will make it much more obvious
when code is introduced which bypasses the cycle detection.
Pass-through-types represent classes to be turned into containers. We
don't want these to turn these containers into Dummy's on a second run
of TypeIdentifier.
This will eventually be used to enable running with Tree Builder v2.
For now, when it is disabled it puts CodeGen v2 into compatibility mode,
disabling features which weren't present in CodeGen v1 so that its
output can be understood by Tree Builder v1.
These aren't used for anything yet, but should be useful for stable IDs
when printing nodes before and after passes and for faster cycle
detection than the current map of pointers.
As we now store ContainerInfo objects in OICodeGen::Config, we can not
copy it any more. Change all places that took copies to take const
references instead.
The copy in OICodeGen modified membersToStub, the contents of which form
part of OICache's hash. However, as OICache also previously had its own
copy, it would not have been OICodeGen's modifications.
- Change member and parent offsets to work in bits, not bytes
- Printer still displays offsets in bytes, with decimals when using
bitfields
- AddPadding: Don't pad bitfields
- CodeGen: Emit code for bitfields
For std::vector and std::list, template parameters are not required to
be defined before they can be used. Delay sorting them until the end.
Also fix a CodeGen bug where we were defining typedefs in the middle of
the forward declarations. They only need to be defined when other types
are defined.
These represent types which don't store any interesting data for us to
measure, but which are required by a real container so can not be
replaced with our own generated class types.
std::allocator often has bad DWARF, so it must be replaced after the
DWARF is fixed up in Flattener. The others could be replaced earlier in
the transformation process if desired, but I've left them all together
for simplicity for now.
This fixes the folly::fbstring tests.
The worry about doing this earlier was performance, but on a further
reading of the code, both legacy OICodeGen and new DrgnParser have been
calling into drgn_type_fully_qualified_name() for every class for a long
time already.
We want to use the fully qualified name for scoped enums to keep the C++
compiler happy. When a parameter expects an enum value, we must supply
an enum value and not its underlying integer value.
Before:
isset_bitset<1, 0>
After:
isset_bitset<1, apache::thrift::detail::IssetBitsetOption::Unpacked>
Multi dimensional arrays are not flattened into 1-D arrays when using
TypeGraph. Update TreeBuilder to account for this.
By not flattening arrays, we are able to produce more descriptive
results.
The disadvantage is that we must now recurse inside arrays
containing only primitives. A better solution to requiring flattening
would be the planned work to not recurse into any static types (not just
primitives). This would also apply to multi-dimensional arrays of
primtivies.
When we were previously removing allocators, we were only able to work
with containers whose allocators appeared as their last template
parameter.
Now we can replace allocators in the middle of a parameter list.
This fixes tests for folly::sorted_vector_set.
This is necessary when replacing the allocator of a map type, for
example.
`std::map<int, int>` will need an allocator which allocates elements of
type `std::pair<const int, int>>`
std::basic_string takes three template parameters:
1. CharT
2. Traits
3. Allocator
The Traits parameter was causing issues, as it requires a type which
exposes certain things, e.g. `Traits::value_type`.
We have a few options to resolve this:
1. Remove this parameter, as we do for allocators
Cons: removing a template parameter doesn't work if other
parameters appear after it
2. Stub this parameter, as we do for hashers/comparators
Cons: we need to hardcode an implementation that satisfies the
`Traits::value_type` requirements
3. Leave the parameter as-is
Cons: will not work if a non-standard Traits is used
By using the real implementation of this Traits parameter
(normally `std::char_traits<CharT>`), we get one that we know will
work as long as it is defined in a stdlib header.
Option 3 is what we use in this patch. Instead of adding more
configuration options to the container TOML file format (e.g.
`params_to_keep = [1]`), we add `std::char_traits` as a dummy
container type. Now, whenever `std::char_traits` appears, it will be
left as-is, i.e. not removed, replaced or reverse-engineered.
This is the same approach previously used for Thrift's isset_bitset.
It was previously easy to miss modules and I haven't been able to notice
any difference in behaviour when setting the VLOG level globally
(despite what the old comment says).
Previously we had an `R"(` string in `OITraceCode.cpp` which allowed us
to include the file as a string. Instead, keep `OITraceCode.cpp` a fully
formed C++ file and utilise the build system to turn it into a string.
This will be used for more header files that are needed both as valid
headers and as strings for JIT compilation in the Typed TreeBuilder
work.
