Summary:
Changes jitlog to use a memfd, an anonymous in memory file descriptor, rather
than a file on disk. Also clean up this fd at the end of an OID run rather than
leaving it in the hope it's valid next time.
A previous attempt to land this used a `char*` from the OID process space in
the remote target syscall. Somehow this works with our integration test target,
but not globally. Changed to use the previous behaviour of putting the syscall
arg in the empty text segment. In doing this I noticed that the text segment
wouldn't be initialised at this point on a fresh process, so we were copying
into effectively an uninitialised address. Move the jit log fd setup to after
the segment setup accordingly.
Test plan:
- CI
- Tested on an integration test target as before. Works.
- Created a new target that definitely doesn't have this string in (simple for
loop). Failed before, works now.
Example:
```sh
$ OID_TEST_ARGS='-fjit-logging' stest OidIntegration.simple_struct
...
I1121 02:57:36.136890 500897 OIDebugger.cpp:269] Outputting JIT logs:
I1121 02:57:36.136896 500897 OIDebugger.cpp:272] JITLOG: SimpleStruct
@00007ffc639be180
I1121 02:57:36.136899 500897 OIDebugger.cpp:272] JITLOG: a @00007ffc639be180
I1121 02:57:36.136901 500897 OIDebugger.cpp:272] JITLOG: obj @00007ffc639be180
I1121 02:57:36.136904 500897 OIDebugger.cpp:272] JITLOG: b @00007ffc639be184
I1121 02:57:36.136905 500897 OIDebugger.cpp:272] JITLOG: obj @00007ffc639be184
I1121 02:57:36.136907 500897 OIDebugger.cpp:272] JITLOG: c @00007ffc639be188
I1121 02:57:36.136909 500897 OIDebugger.cpp:272] JITLOG: obj @00007ffc639be188
I1121 02:57:36.136911 500897 OIDebugger.cpp:278] Finished outputting JIT logs.
...
```
CodeGen v1 and CodeGen v2 must be in sync in order for CodeGen v2 and
TreeBuilder v1 to be compatible. This change updates CodeGen v1 to use
the same set of containers as CodeGen v2.
This specifically fixes "-ftype-graph -Ftree-builder-v2 -Fcapture-thrift-isset"
for Thrift types.
The integration test "thrift_isset_no_capture" covers this, but this bug
was missed as the Thrift tests do not run in CI.
Summary:
tbv2: add dynamic context passed through all functions
Previously for we had some shared state between all requests, noticeably the
pointers set. This change adds a by reference value to all requests which can
hold additional mutable state. The pointers set is moved into this mutable
state for OIL, which means each concurrent request will have its own pointer
set. Doing things this way allows more features to be added in the future
without such a big code modification.
Closes https://github.com/facebookexperimental/object-introspection/issues/404
Pull Request resolved: https://github.com/facebookexperimental/object-introspection/pull/410
Test Plan: - CI
Differential Revision: D51394035
Pulled By: JakeHillion
fbshipit-source-id: 55d2ba9b5e056148a29dc821020cfc3d94e5175a
Summary:
tbv2: replace DB template param with Ctx
TreeBuilder v2 adds a DB template parameter to every function. This is used as
part of the static type to decide what type of DataBuffer is being used:
currently `BackInserterDataBuffer<std::vector<uint8_t>>` for OIL and it would
be `DataSegmentDataBuffer` for OID.
This change replaces the `DB` template parameter with a more general `Ctx`. Due
to issues with dependent naming it also adds a `using DB` to each `TypeHandler`
which has the same function as before. This allows us to add more "static
context" (typedefs and constants) to functions without changing this signature
again, because changing the signature of everything is a massive pain.
Currently this change achieves nothing because Ctx contains only DB in a static
wrapper. In the next change I'm going to pass a reference of type Ctx around to
add a "dynamic context" to invocations which will contain the pointer array. In
future we'll then be able to add either static or dynamic context without any
signature adjustments.
Test Plan:
- CI
---
Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/facebookexperimental/object-introspection/pull/409).
* https://github.com/facebookexperimental/object-introspection/issues/410
* __->__ https://github.com/facebookexperimental/object-introspection/issues/409
Reviewed By: ajor
Differential Revision: D51352092
Pulled By: JakeHillion
Summary:
Currently there are two features between CodeGen v2 (TypeGraph) and TreeBuilder
v2. These are TypedDataSegment and TreeBuilderTypeChecking. Each of these
features currently has a full set of tests run in the CI and each have specific
exclusions.
Collapse these features into TreeBuilder v2. This allows for significantly
simplified testing as any OIL tests run under TreeBuilder v2 and any OID tests
run under TreeBuilder v1.
The reasoning behind this is I no longer intend to partially roll out this
feature. Full TreeBuilder v2 applies different conditions to containers than
the intermediate states, and writing these only to have them never deployed is
a waste of time.
Test Plan:
- it builds
- CI
Adds the range-v3 library which supports features that otherwise wouldn't be
available until C++23 or C++26. I caught a couple of uses that suit it but this
will allow us to use more in future.
Test Plan:
- CI
Not all containers have 8-byte alignment, so if we want to avoid lots of
manual logic for calculating container alignment on a case-by-case
basis, we must calculate alignment from the member variables before the
Class nodes have been replaced by Container nodes.
Leave it to the new mutator pass IdentifyContainers to replace Class
nodes with Container nodes where appropriate.
This will allow us to run passes over the type graph before identifying
containers, and therefore before we have lost information about the
internal details of the container (e.g. alignment of member variables).
For the containers which are allowed to be declared with incomplete
types, it is only the contained types which are allowed to be
incomplete. Other template parameters (e.g. allocators) must always be
defined before use.
Summary:
Spin around our open/closed source checks. Previously we defined `OSS_ENABLE` in open source builds. This change defines `OI_META_INTERNAL` instead. This is nicer, as external users don't have to do anything special to get a working build.
Use this new macro to define a boolean constant in a new header `Portability.h`. This is inspired by Folly, and makes the internal build easier - definitions in Buck2 have to propagate up from a dependency instead of down from one. Annoyingly we can't use `if constexpr` for a lot of the previous `#ifdef` blocks as we conditionally include the headers. Longer term we could fix this by exposing a header interface but no source, allowing these to build but not be compiled in. For now I did something weird: I defined a function style macro in `Portability.h` based on the compile time macro. This forces you to have included `Portability.h` before using it to ensure the definition is everywhere. Open to feedback as I haven't seen anyone else do this.
Reviewed By: tyroguru
Differential Revision: D50000454
Summary:
More moving code out of the `cea` subdirectory. This is moving the GOBS cache code which is contained in the 'internal' subdirectory. I'm hoping I get some auto generated diffs to push to GitHub for the third party source changes...
Reviewed By: JakeHillion
Differential Revision: D50366591
Summary:
Extend the multiple config files system to OILGen, the piece it was originally designed for. This allows for specifying additional configs which say which keys of maps to capture.
Reviewed By: ajor
Differential Revision: D50105138
Summary:
Adds the option for required features to container definitions. These cause the container not to be passed to `DrgnParser` if that feature is not enabled during code generation. The thrift isset type does not currently work with `tree-builder-v2` and only provides benefit with `capture-thrift-isset`. This change makes sure the container is ignored if it won't be useful, allowing code generation under `tree-builder-v2`.
Test Plan: - CI
Differential Revision: D49960512
Pulled By: JakeHillion
Summary:
Previously OID/OIL required exactly one configuration file. This change makes it so you can supply 0 or more configuration files. 0 is useful if you have pre-generated the cache or use some sort of remote generation system. 1 is useful for the common case, where you have a configuration file that describes your entire source and use just that. More are useful if you have supplemental bits of config you wish to apply/override - see the changes to the integration test framework where we do exactly this.
Test Plan:
This isn't super well tested. It works for the test cases which add features via the config or enable `codegen.ignore`.
- CI
Reviewed By: ajor
Differential Revision: D49758032
Pulled By: JakeHillion
Lots of places rely on reference stability of ContainerInfo objects
(CodeGen's deduplication, Container nodes' containerInfo_ member).
In the key capture work, we need to be able to append to this list,
which would invalidate references before this change.
Dummy and DummyAllocator nodes had been changed to use NodeIds, but
were still printed out in full when visited for a second time.
[[nodiscard]] prevents future bugs of this type by turning them into
compilation errors.
Example of the now-fixed bug:
[1] Container: std::map<int32_t, int32_t, DummySizedOperator<0, 0, 8>, std::allocator<std::pair<int32_t const, int32_t>>>
Param
Primitive: int32_t
Param
Primitive: int32_t
Param
[2] Dummy [less<int>]
Param
...
[3] Container: std::map<int32_t, int32_t, DummySizedOperator<0, 0, 8>, std::allocator<std::pair<int32_t const, int32_t>>>
Param
Primitive: int32_t
Param
Primitive: int32_t
Param
[2]
Dummy [less<int>]
Param
...
With this patch, the second "Dummy" line will not be printed.
We only want to do the extra work if it's explicitly requested.
chaseRawPointers is already explicitly requested whenever it's needed
and readEnumValues currently isn't needed at all.
Summary:
Previously on large types OIL would have problems with corrupting the `std::stack<exporter::inst::Inst>` that is passed to the processors. This change hides the implementation of the stack from the processors by wrapping the call to emplace in a `std::function` written by the non-generated code, which solves the test case I've seen for this crashing. It also allows us to easily change the stack implementation in future - I plan to change it to a `std::stack<T, std::vector<T>>` in a follow up.
Reviewed By: tyroguru
Differential Revision: D49273116
Summary:
Update to clang-15 compiler and libraries as clang-12 is ancient.
The changes to oilgen are necessary because the new internal toolchain is being more picky about linking PIC to PIC. In certain modes we build with PIC, but try to link a non-PIC oilgen artifact. Add the ability to build the oilgen artifacts with PIC which sorts this.
Reviewed By: ttreyer
Differential Revision: D46220858
Summary:
The iterator was incremented without checking it in the JSON exporter. This caused an assertion to trigger on the last run in debug mode (weirdly no crashes). This change should fix that by checking the iterator at the increment site and not just when the loop rolls around.
Differential Revision: D49151482
Summary:
Update `OIGenerator` and out BUCK stuff for compile time OIL with OIL v2. Main things:
- Switch `OIGenerator` from the `getObjectSize` call to the new `introspect` call.
- Switch from looking at template parameters to looking at function parameters, as this was exposing a bug in our elfutils/drgn and this way it's the same as OID.
- Migrate `OIGenerator` to CodeGen v2 and update CodeGen v2 to accept a linkage name.
- Update the compile time example to be the same as the JIT example, using the new interface and the JSON exporter.
- Clean up the `ObjectIntrospection.h` header.
Differential Revision: D48687728
fbshipit-source-id: 2c3c041fd1b6499c5e02eb5e2082a977bfa529d7
Types within containers were previously named TODO. This sorts it out so
they're named as their most resolved type. The current implementation
skips Typedef names.
The TypeGraph class should only be responsible for storing Type nodes.
Traversing the graph and tracking which nodes have been visited should
not be included there.
Passes now take a NodeTrackerHolder as an input parameter, which
provides access to a zeroed-out NodeTracker.
Type Graph deduplicates and modifies names to better fit the generated
code, for example `int32_t[4]` becomes `OIArray<int32_t, 4>` and `struct
MyStruct` might become `struct MyStruct_0`.
Add an `inputName` which better represents the original input code which
can be used when building the tree.
This removes Printer's legacy behaviour of generating an ID for each
node as it gets printed. This old method meant that if new nodes were
added to or removed from a graph, every ID after the new/removed node
would change.
Now IDs are stable so it is easier to follow specific nodes through
multiple transformation passes in CodeGen.
Names which were generated on-demand are now stored in member variables,
which are set during the ctor and can be regenerated when required (by
NameGen).
We previously only marked as packed if there was no tail padding, which
was not a sufficient condition.
The new AlignmentCalcTest.PackedMembers test case is an example which
would previously not have been marked as packed.
CodeGen v1 does not record anything for pointers to incomplete types.
Not even the address, as is done for other pointers.
Introduce a new Primitive type "Incomplete". This behaves identically to
"Void", but allows us to tell whether a type was defined as void or if
it ended up like that because of incomplete DWARF information.
This extracts the compatibility logic from AddPadding, which allows for it to be
simplified and will make it easier to extend and eventually remove in the
future. No functional changes.
This lets us remove fields from types when they are no longer needed,
speeding up later passes.
A secondary benefit of pruning unused types means that we sometimes
remove types for which we can't generate correct C++ code. This can
allow us to CodeGen for complex types which reference these broken types
without actually requiring them (e.g. as template parameters).
Add a new feature flag "prune-type-graph" to control this pass. It makes
sense to prune most of the time, but for testing CodeGen functionality
on a wider range of types, it will be useful to have the option to not
prune.
In general, we can't tell which member is active in a union so it is not
safe to try and measure any of them.
Explicitly set the alignment of unions (and structs/classes) in CodeGen
if it is available, as the C++ compiler can no longer infer it from the
members.
Also reshuffle CodeGen's passes to fix an alignment bug with removed
members.
Change RemoveMembers to actually remove members instead of replacing
them with padding. AddPadding must be run afterwards to fill in the
gaps.
The underlying type of bitfield is important to the size of a struct:
struct Foo { int64_t bitfield : 1; };
struct Bar { int8_t bitfield : 1; };
sizeof(Foo) = 8;
sizeof(Bar) = 1;
Previously this code would not have removed all members which it was
supposed to.
Also remove some now-redundant code from TypeIdentifier. RemoveIgnored
will take over the responsibility of removing members from classes.
