Commit Graph

462 Commits

Author SHA1 Message Date
Jon Haslam
af7800e661
std_align_t fix (#475) 2024-02-05 13:10:43 +00:00
Jonathan Haslam
2e8edd78b5 stub clang::BuiltinType::NullPtr primitive type
Summary: Adding support for the nullptr clang type (clang::BuiltinType::NullPtr). While there I augmented the exception message to include the type name that is missing.

Differential Revision: D53272742
2024-01-31 10:22:54 -08:00
Jake Hillion
db289c1a1a tbv2: use std::decay_t with smart pointers
CodeGen v2 permits template parameters to be qualified. This means that if we
call `make_field` with a template parameter it will be qualified. However, we
don't qualify the types when generating meta functions such as `NameProvider`
and `TypeHandler`. This means these qualified types don't match up with the
expected type.

Use `std::decay_t` when forwarding the type to `NameProvider` and `TypeHandler`
so they're always the base type that they were generated with. Most of this is
covered by `make_field`, but there are direct references to `TypeHandler<Ctx,
T>` in a lot of `TypeHandler::type` fields. Fix the problematic types manually
for now, there may need to be a better solution with meta functions for this in
the future.

Test Plan:
- CI
- Added a test for `std::unique_ptr<const uint64_t>` to exercise this. Failed
  before, passes after.
- Added a test for `std::unique_ptr<const std::vector<uint64_t>>` to test a
  non-primitive type. Failed before, passes after.
2024-01-31 17:27:04 +00:00
Jon Haslam
31bf9e7b59
Make KeyCapture work with nested typedefs (#473) 2024-01-31 17:03:05 +00:00
dependabot[bot]
617493ddf5
build(deps): bump postcss from 8.4.21 to 8.4.33 in /website (#470) 2024-01-31 11:50:19 +00:00
dependabot[bot]
ecc114ebba
build(deps): bump follow-redirects from 1.15.2 to 1.15.5 in /website (#471) 2024-01-31 11:49:49 +00:00
dependabot[bot]
8c7529e214
build(deps): bump @babel/traverse from 7.21.3 to 7.23.9 in /website (#472)
Bumps [@babel/traverse](https://github.com/babel/babel/tree/HEAD/packages/babel-traverse) from 7.21.3 to 7.23.9.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.23.9/packages/babel-traverse)

---
updated-dependencies:
- dependency-name: "@babel/traverse"
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-31 11:49:10 +00:00
Janeczko Jakub
cc401c1e5b
fix string type sso computation (#469)
* fix string type sso computation

* fix rest of sbo/sso calculation

* make placement of uintptr_t cast consistent

* separate check-inline into function
2024-01-31 11:19:24 +00:00
Jake Hillion
b5b94ed236 container_info: switch to boost::regex (#465)
Summary:

OI was previously using `std::regex_match` to match container names. This was bad because `libstdc++`'s implementation of regex is awful. In the case of limited inlining it was causing a stack overflow when running CodeGen for large types (I think types with large names but I never got to the bottom of it).

Replace this with the competent `boost::regex_match` that we already have a dependency on.

Reviewed By: ajor

Differential Revision: D53002752
2024-01-23 10:58:58 -08:00
Thierry Treyer
1c65ed8ec7 Implement Container V2 for std::deque 2024-01-23 19:43:56 +01:00
Jake Hillion
7de35863f5 tbv2: fix Thrift isset lookups with padding
Thrift isset was failing with a SEGV if the struct contained padding. This is
because we indexed the `isset_indexes` data structure using our field index
rather than the index of the field in Thrift. This then gave a rubbish index
for any exceeding which happens if we have added padding in the middle of the
struct, and this index was looked up in the bitset which can cause a SEGV.

Track a new index `thriftFieldIdx` which is only incremented if we've looked up
a Thrift index.

Namespaced the generated Thrift structs while I was there. This isn't necessary
anymore but cleans things up.

Test plan:
- Added a test case with lots of padding. These don't run in the CI but it
  passes locally.
- `FILTER='OilIntegration.*' make test` - no failures
- `FILTER='OidIntegration.*' make test` - no new failures
2024-01-19 19:16:46 +00:00
Jake Hillion
7eebee2bf7 type_graph: avoid overwriting explicitly set alignment
Previously AlignmentCalc calculates the alignment and sets packing for every
type except a member with explicit alignment. Change this to check whether an
alignment has been previously set for a type before calculating it. Use this in
ClangTypeParser where the full alignment of the type is available.

Remove explicitly aligning members by the type because that was previously
reserved for members with explicit alignment. AlignmentCalc will correctly
align a member to the underlying type without this. Explicit member alignment
is still missing, as before this change.

Test plan:
- CI
- Too little. Gets further into a production type than without this change.
2024-01-18 16:44:12 +00:00
Jake Hillion
31ba8659f0 tbv2: fix pointer codegen
A previous change enabled running OIL tests with specific features enabled.
This highlighted that pointer code generation under TreeBuilder-v2 was very
broken. This change updates pointer code generation to work and enables the
skipped tests. All enabled tests need `expected_json_v2` added to them due to
formatting differences.

Reformatted and rewrote the basic type handler that handles primitives and
pointers. Removed the reliance on `features` to decide whether to generate for
TreeBuilder-v2 as the intermediate features have been removed. Pointers are
treated as containers with a capacity of 1 and a length of 0 if null/a cycle
and 1 if followed. This holds for void pointers where, although they aren't
followed, the length is still set.

There were a couple of other changes needed to enable these tests on TBv2 that
aren't worth their own issues and PRs, I sneaked them in here.

Extra changes:
- Added `Pointer` and `Reference` to TopoSorter so they generate
  `NameProvider` instances. It might be worth visiting the graph differently
  for `NameProvider` as it requires so many instances that others generators do
  not. Will consider that in the future.
- Follow typedefs when calculating exclusive size for a type.

Closes #458.

Test plan:
- CI
- Enabled previously disabled tests.
2024-01-18 16:22:18 +00:00
Jake Hillion
819914beca tbv2: fix thrift isset with ClangTypeParser
Some of the logic that makes Thrift isset work for TreeBuilder-v2 in DrgnParser
(JIT OIL) wasn't ported to ClangTypeParser meaning it doesn't work in
Ahead-of-Time (AoT) OIL.

Add the template parameter name reconstruction for enum values to
ClangTypeParser.

Test plan:
- Tested with Thrift isset enabled on an internal type. Doesn't build before,
  does build after.
2024-01-17 14:58:51 +00:00
Jake Hillion
40af807d8b tbv2: support capture-thrift-isset
Support the capture-thrift-isset feature with TreeBuilder-v2. Fairly minor
changes here except the type of the Enum in a template parameter now matters.

We follow the previous behaviour of capturing a value for each field in a
struct that has an `isset_bitset`. This value is a VarInt captured before the
C++ contents of the member. It has 3 values: 0 (not set), 1 (set), and 2
(unavailable). These are handled by the processor and represented in the output
as `false`, `true`, and `std::nullopt_t` respectively.

Changes:
- Add a simple Thrift isset processor before any fields that have Thrift isset.
- Store the fully qualified names of enum types in DrgnParser - it already
  worked out this information anyway for naming the values and this is
  consistent with classes.
- Forward all enum template parameters under their input name under the
  assumption that they will all be policy type things like `IssetBitsetOption`.
  This could turn out to be wrong.

Test plan:
- CI (doesn't test thrift changes but covers other regressions)
- Updated Thrift enum tests for new format.
- `FILTER='OilIntegration.*' make test` - Thrift tests failed before, succeed
  after.
2024-01-16 19:09:46 +00:00
Jake Hillion
4975b6e9fa test: add features field to integration tests
Previously we tested different feature flags by using the `cli_options` field
in the test `.toml`. This works for OID but doesn't work for JIT OIL and won't
work for AoT OIL when those tests get added.

This change adds a new higher level `features` field to the test `.toml` which
adds the features to the config file as a prefix. This works with all methods
of generation.

Change the existing `cli_options` features to `features` except for where
they're testing something specific. Enable tests that were previously disabled
for OIL but only didn't work because of not being able to enable features.
Change pointer tests that are currently broken for OIL from `oil_disable` to
`oil_skip` - they can work, but codegen is broken for them at the minute.

Test plan:
- CI
- `make test` is no worse
2024-01-16 16:23:21 +00:00
Jake Hillion
4c047b5f91 tbv2: add is_primitive to output
C++ has a concept of Primitive which holds in the type graph. However we don't
currently expose this information to the end user. Expose this from the OIL
iterator to allow future features like primitive rollups.

This affects containers like maps which have a fake `[]` element with no type.
They use this to group together the key/value in a map and to account for any
per element storage overhead. Currently the decision is to make the fake `[]`
element a primitive if all of its children are primitives. This allows for more
effective primitive rollups if that is implemented. This implementation detail
may be changed in future.

Test Plan:
- CI
- Updated simple tests.
2024-01-16 11:14:13 +00:00
Jake Hillion
16fcba20bc tbv2: name array member types correctly
Array members are currently being named "TODO" (whoops). Include arrays in
TopoSorter so each one can have a `NameProvider` generated in CodeGen. Then
pass array elements through `make_field`.

Test plan:
- CI
- Add array member names to an array test.
2024-01-15 16:22:28 +00:00
Jake Hillion
0e72947786 tbv2: add support for std::reference_wrapper
Closes #307

Test plan:
- CI
- Updated and enabled tests.
2024-01-15 16:20:26 +00:00
Jake Hillion
08e2faa90e tbv2: correctly account for list overhead
`std::list` has per element overhead for the individual heap allocations. This
was already calculated in the container implementation but not used. Allocate
the overhead of each element in the `std::list` to the `std::list` itself as
with other sequential containers.

Test Plan:
- CI
- Updated test cases
2024-01-11 15:41:48 +00:00
Jake Hillion
a9afb25248 tbv2: update tuple test
This test is a bit odd, but this change adds the full set of size/member checks
for the hierarchy for TreeBuilder v2 and enables it.

Closes #304

Test Plan:
- CI
2024-01-11 13:39:19 +00:00
Thierry Treyer
d232a5d2fb Add namespaces to integration tests' "typeName"
Some integration tests have not been updated since drgn started
reporting type names with all their namespaces, and were failing.
2024-01-11 14:13:51 +01:00
Thierry Treyer
cf8fe64d5d Enable test arrays_member_int0 2024-01-10 19:13:41 +01:00
Thierry Treyer
91ff9fceb9 Fix TreeBuilder processing of zero-length array
TreeBuilder did not consider a zero-length array like a container and
never read the array's sizeof stored in the data buffer, leading to a
mismatch between bytes written vs read out of the buffer.

Now, `TreeBuilder::isContainer` does consider zero-length array like
a container and properly consume all the object sizes in the buffer.
2024-01-10 19:13:41 +01:00
Thierry Treyer
fba0d527fd Fix static_assert failure for zero-length array
The recursive template implemented for `validate_size` does not support
incomplete types, like zero-length array.

By splitting the `validate_size` struct in two parts:
1. `validate_size_eq` that does the actual size check, and
2. `validate_size` that preserves the previous interface and inherit
   from `validate_size_eq`,
we get the same interface and feature than previously, but without the
recursive template that doesn't support incomplete types.
2024-01-10 19:13:41 +01:00
Jake Hillion
cbeafba9bb tbv2: fix type names for std::optional
Type names of optional elements were accidentally left as todo. Update
`std::optional` to use `make_field` and correctly name its elements.

Test Plan:
- CI
- Updated the integration tests to test the names.
2024-01-09 15:09:24 +00:00
Jake Hillion
db93feb180 incomplete: name type in compiler errors
Summary:

We have a good type representation in the Type Graph of an incomplete type and
the underlying type that represents. However, this incomplete type still ends
up in the generated code as `void` which loses information. For example, a
container that can't contain void may fail to compile because it was
initialised with `void` but really its because the type it was supposed to be
initialised with (say, `Foo`) had incomplete debug information.

This change identifies that a type is incomplete in the output by generating it
as an incomplete type `struct Incomplete<struct Foo>`. This allows us to name
the type correctly in the TreeBuilder output and filter for incomplete types,
as well as getting appropriate compiler errors if it mustn't be incomplete.

Test Plan:
- CI
- Added a unit test to namegen.
- Enabled and added an extra pointers_incomplete test.

This change is tricky to test because it isn't really user visible. The types
still use their `inputName` which is unchanged in any successful output - this
change is used so the compiler fails with a more detailed error.
2024-01-09 15:08:25 +00:00
Jake Hillion
71e734b120 tbv2: calculate total memory footprint
Add the option to calculate total size (inclusive size) by wrapping the
existing iterator. This change provides a new iterator, `SizedIterator`, which
wraps an existing iterator and adds a new field `size` to the output element.

This is achieved with a two pass algorithm on the existing iterator:
1. Gather metadata for each element. This includes the total size up until that
   element and the range of elements that should be included in the size.
2. Return the result from the underlying iterator with the additional
   field.

This algorithm is `O(N)` time on the number of elements in the iterator and
`O(N)` time, storing 16 bytes per element. This isn't super expensive but is a
lot more than the current algorithm which requires close to constant space.
Because of this I've implemented it as a wrapper on the iterator rather than on
by default, though it is now on in every one of our integration test cases.

Test plan:
- Added to the integration tests for full coverage.
2024-01-04 09:21:35 +00:00
Jake Hillion
6c90103278 circleci: clean up codegen v1 runs
Remove the CodeGen v1 sections of the CI config because both OID and OIL use
CodeGen v2.

We were missing running any test that wasn't `Oi{d,l}Integration.*` before.
This now runs the unit tests again and requires a minor fix to one unit test.

Test plan:
- CI
2024-01-03 17:29:59 +00:00
Jake Hillion
beb404e41c clangparser: mark incomplete arrays as incomplete without failing
Attempting to complete a type which can't be completed currently fails oilgen.
For incomplete arrays, which we know are not possible to complete, return false
deliberately.

`requireCompleteType` likely needs to not fail in all cases in the future. For
now this works.

Test plan:
- `std::unique_ptr<long[]>` used to fail the generation. Now it can
  successfully codegen.
2023-12-20 16:31:52 +00:00
Jake Hillion
c5ecb9aaa2 clangparser: provide alignment info for members
Unlike DWARF, the Clang AST is capable of correctly calculating the alignment
for each member. If we do this then AlignmentCalc doesn't traverse into the
member to attempt to calculate the alignment.

This check might be wrong if the field has explicit alignment. That case can be
covered when we have proper integration testing and a repro.

Test plan:
- Without this lots of static asserts occur. With this it's okay.
2023-12-20 16:15:12 +00:00
Jake Hillion
20cd48ac63 clangparser: provide correct kind for classes/unions
Previously ClangTypeParser assumed all RecordTypes were structs. This is fine
for structs and classes but completely incorrect for unions. Check which type
it is and give type graph the correct one.

Test plan:
- Unions static assert without this change because their size/alignment is
  wrong.
2023-12-20 16:15:02 +00:00
Jake Hillion
6d898bed95 tbv2: account for duplicate types when emitting name providers
ClangTypeParser has emitted a duplicate type for `std::allocatr<int8_t>`.
Rather than fixing this, add the same check the compiler will do for the
duplicate templates that `addNames` emits. That is, `template<>
NameProvider<Foo>` will collide if `Foo` is used twice. We can do this by
adding a set of these strings for now. If this shows up regularly it will
likely make sense to deduplicate the type graph with a deduplication pass.

Test plan:
- Fixes the issue in prod. This change is quite logical.
2023-12-20 16:14:48 +00:00
Jake Hillion
5a2ca8b059 tbv2: implement folly::IOBuf
folly::IOBuf does not have TreeBuilder v2 container support. Add it.

The implementation is a direct clone of v1. It still lacks tests.

Test Plan:
- It codegens on a prod type.
- No runtime testing... Bad form, I know.
- Issue created to add integration tests: https://github.com/facebookexperimental/object-introspection/issues/436
2023-12-20 16:13:50 +00:00
Jake Hillion
55989a9156 oilgen: migrate to source parsing (#421)
Summary:
oilgen: migrate to source parsing

Using debug information generated from partial source (that is, not the final
binary) has been insufficient to generally generate OIL code.

A particular example is pointers to templates:
```cpp
#include <oi/oi.h>
template <typename T>
struct Foo {
  T t;
};
template <typename T>
struct Bar {
  Foo<T>& f;
};
void foo(const Bar<int>& b) {
  oi::introspect(b);
}
```

The pointer/reference to `Foo<int>` appears in DWARF with
`DW_AT_declaration(true)` because it could be specialised before its usage.
However, with OIL, we are creating an implicit usage site in the
`oi::introspect` call that the compiler is unable to see.

This change reworks OILGen to work from a Clang command line rather than debug
information. We setup and run a compiler on the source, giving us access to an
AST and Semantic Analyser. We then:
- Find the `oi::introspect` template.
- Iterate through each of its callsites for their type.
- Run `ClangTypeParser::parse` on each type.
- Run codegen.
- Compile into an object file.

Having access to the semantic analyser allows us to forcefully complete a type,
as it would be if it was used in the initial code.


Test Plan:
hope

`buck2 run fbcode//mode/opt fbcode//object-introspection/oil/examples/compile-time:compile-time`

Reviewed By: tyroguru

Differential Revision: D51854477

Pulled By: JakeHillion
2023-12-19 13:26:25 -08:00
Jake Hillion
37b89d789d codegen: remove reliance on drgn type for top level name
Currently we rely on `SymbolService::getTypeName` for getting the hash that's
included in the generated function's name. The value of this must stay the same
to match with the value expected by OIDebugger - changing it causes failure to
relocate when attaching with OID and JIT OIL.

Calculate this name in the `codegenFromDrgn` method and pass it through where
appropriate rather than passing the `drgn_type` itself through.

We don't need to name the type like that when using AoT OIL. Let's
hash the linkage name instead as that is more unique.

Test Plan:
- CI
2023-12-19 15:35:59 +00:00
Alastair Robertson
2060a0491e CodeGen v2: Enable independent running without CodeGen v1
Create DrgnExporter to translate Type Graph "Type" nodes into drgn_type
structs, suitable for use in OICache and TreeBuilder.
2023-12-15 14:57:24 +00:00
Alastair Robertson
a0164e5cc7 TypeGraph: Make Class types use fully qualified names as their input names
This will ensure we continue to get fully qualified names in
user-visible output when we switch to CodeGen v2.
2023-12-15 14:45:01 +00:00
Alastair Robertson
688d483c0c TypeGraph: Fix handling for classes which inherit from containers
We previously moved container identification later in CodeGen in order
to preserve information for AlignmentCalc.

However, Flattener needs to know if a class is a container in order to
apply its special handling for this case.

This new approach moves container identification in front of Flattener,
but has Container own a type node, representing its layout. This
underlying type node can be used for calculating a container's
alignment in a later pass.
2023-12-14 18:02:45 +00:00
Alastair Robertson
aa87c3f2d1 NameGen: Override inputName for anonymous members 2023-12-14 17:42:48 +00:00
Alastair Robertson
c874f72ae2 OID: Make CodeGen v2 (TypeGraph) the default 2023-12-14 17:42:03 +00:00
Jake Hillion
35afd15ee4 capture_keys: store dynamic type path components more efficiently 2023-12-14 16:05:33 +00:00
952d3e75c8 ci: move formatting checks to nix 2023-12-14 15:31:07 +00:00
Alastair Robertson
8bf7dbae9f Type Graph: Replace MutationTracker with the more general ResultTracker
MutationTracker could only store Type nodes, while ResultTracker is
templated on the result type so can store anything.

Template the Visitor base class on the return type of visit() functions.

This sets us up for allowing visitors to return different results from
their visit() functions in the future.

This will be used in a future commit introducing DrgnExporter, where we
cache drgn_type* results while walking the type graph.
2023-12-14 13:43:19 +00:00
Jon Haslam
8193d271a8
Really, really make sure arrays have a name (#430) 2023-12-13 16:28:31 +00:00
Thierry Treyer
d79b55cfd2 Update integration tests 2023-12-13 11:59:21 +00:00
Thierry Treyer
33b67e6caf Update drgn to Omar's branch 2023-12-13 11:59:21 +00:00
Thierry Treyer
9047f69db4 Check bounds when processing unique_ptr 2023-12-13 11:59:21 +00:00
Alastair Robertson
7cc7aa8882 CodeGen: Remove Incomplete members from Classes
They must not appear in the final generated code as we'd end up with
invalid types with void members, e.g.:
  struct Foo {
    int a;
    void myIncompleteMember;
    int c;
  };

Removing them from the type graph early also ensures that padding is
calculated correctly.
2023-12-12 18:50:15 +00:00
Alastair Robertson
6b780add4a Integration Tests: Fix thrift_unions tests
With a recent Thrift update, we now must also define a destructor for
this type.
2023-12-12 18:33:52 +00:00