object-introspection/test/integration/README.md

# Integration Tests

This directory contains test definition files for OI's integration tests.

## Running tests

There are a number of ways to run these integration tests.

1. Run the `integration_test_runner` executable directly. This provides some
additional options to aid debugging:
    - `--verbose` Verbose output
    - `--preserve` Do not clean up files generated by OID after tests are finished
    - `--force` Run tests that have been marked as "skipped"

    e.g.
    ```sh
    build/test/integration/integration_test_runner --gtest_filter=OidIntegration.primitives_int --verbose
    ```

1. Run a number of the integration tests in parallel:

    ```sh
    ctest --test-dir build/test/integration -j$(nproc) [--tests-regex <regex>]
    ```

## Adding tests

1. Create a new test definition file in this directory and populate it as needed. See [Test Definition Format](#test-definition-format) for details.
1. Add your new definition file to the `INTEGRATION_TEST_CONFIGS` list in [`CMakeLists.txt`](CMakeLists.txt)

## Test Definition Format

Test definitions are stored in the [TOML](https://toml.io/) file format.

Example:
```toml
includes = ["vector", "unordered_map"]
definitions = '''
  struct Foo {
    std::vector<int> v;
  };
  using Bar = std::unordered_map<int, int>;
'''
[cases]
  [cases.my_first_test_case]
    param_types = ["const Foo&", "const Bar&"]
    setup = '''
      Foo foo;
      foo.v = {4,5,6};
      Bar bar;
      bar[2] = 3;
      return {foo, bar};
    '''
    expect_json = '{"staticSize":4,"dynamicSize":32}'
  [cases.another_test_case]
    param_types = ["int"]
    setup = 'return 123;'
```

### Details

- `includes`

  Header files required for this test.

  Example:
  ```
  includes = ["vector", "unordered_map"]
  ```

- `definitions`

  C++ type definitions required for a test can be defined here.

  Anything defined in this section will be automatically wrapped in a namespace
  and will be private to this test.

  Example:
  ```
  definitions = '''
    struct Foo {
      std::vector<int> v;
    };
    using Bar = std::unordered_map<int, int>;
  '''
  ```

- `thrift_definitions`

  Thrift type definitions can be specified here. These will be passed to the
  Thrift compiler which will generate C++ code from them.

  **CAUTION**: Generated Thrift types are not wrapped in a namespace, so type
  names must be globally unique between all tests.

  Example:
  ```
  thrift_definitions = '''
    struct MyThriftStruct {
      1: optional i32 a;
      2: optional i32 b;
    }
  '''
  ```

- `raw_definitions`

  This section allows specifying of arbitrary C++ code which will be directly
  copied into the target program's source without being wrapped in a namespace.

  It should not be used for most tests. The purpose is to allow defining code
  required for a specific test to compile, avoiding the need to add new
  dependencies to the build system for one-off tests.

- `cases` **Required**

  A list of individual test cases, each with their own setup, OI probe
  definition and expected results, but sharing any type definitions created in
  this test file.

  Test cases should be grouped into related areas and put into shared test files.

  - `param_types` **Required**

    Paramter types of the function to probe.

    oid does not have complete support for probing pass-by-value parameters, so
    it is recommended to define all parameters as reference or pointer types.

    Example:
    ```
    param_types = ["const std::vector<int>&", "const Foo&"]
    ```

  - `arg_types`

    Types of the arguments being passed to the probed function. Defaults to
    `param_types` with const, volatile and references removed.

    It is only necessary to specify `arg_types` when they will differ from the
    parameter types expected by the probed function. This can be useful for
    testing inheritance.

    Example:
    ```
    param_types = ["BaseClass *"]
    arg_types = ["DerivedClass"]
    ```

  - `setup` **Required**

    A snippit of C++ code which creates and returns values to be passed to the
    function being probed as a part of this test case. The returned value should
    be a tuple of `param_types`, although the curly brakcets/braces can be
    omitted in most cases when there is only a single value in the tuple.

    Example:
    ```
    setup = '''
      std::vector<int> ret = {1,2,3};
      return {ret, Foo(1)};
    '''
    ```

  - `type`

    OI probe type. Defaults to `entry`.

    Example:
    ```
    type = "return"
    ```

  - `args`

    Comma separated list of arguments to introspect. Defaults to `arg0`.

    Example:
    ```
    args = "arg0,arg1"
    ```

  - `target_function`

    Symbol of the target function to be traced, when the auto-generated target
    function is not sufficient.

    The fields `param_types` and `setup` are no longer required if
    `target_function` is defined.

    Example:
    ```
    target_function = "my_function"
    ```

    Implies `oil_disable`.

  - `cli_options`

    Additional command line arguments passed to oid.

    Example:
    ```
    cli_options = ["--chase-raw-pointers"]
    ```

  - `skip`, `oid_skip`, `oil_skip`

    Skip running this test for oid and/or oil.

    Pass in a string to provide a reason for skipping the test.

    Example:
    ```
    oid_skip = "oid support not implemented yet"
    ```

  - `oil_disable`

    Disable this test for oil.

    When a test doesn't make sense for oil, disable it rather than skipping it,
    to avoid it showing up in the aggregate test results entirely.

    Example:
    ```
    oil_skip = "oil can't chase raw pointers safely"
    ```

  - `expect_oid_exit_code`, `expect_oil_exit_code`

    Exit code expected from OI. Defaults to 0.

    Example:
    ```
    expect_oid_exit_code = 6
    ```

  - `expect_json`

    JSON expected to match results from OI.

    Only keys which appear in these expected results are used for comparison.
    This means that irrelevant or non-reproducable keys can be omitted and they
    will be ignored. Missing keys in the actual results will still cause test
    failures.

    Example:
    ```
    expect_json = '{"staticSize":4,"dynamicSize":0}'
    ```

    To ensure that a given key does not appear in the results, the special
    **NOT** key can be used, with the value set to the undesired key's name.

    This example checks that the JSON result does not contain the key "members":
    ```
    expect_json = '{"NOT":"members"}'
    ```

    The **NOT** key can also be used to check that a given key's value is not
    equal to some expected value.

    This example checks that the result has a key named `pointer`, but that its
    value is not equal to 0:
    ```
    expect_json = '{"NOT":{"pointer":0}}'
    ```

  - `expect_stdout`

    Regex expected to match OI's stdout.

    Example:
    ```
    expect_stdout = ".*SUCCESS.*"
    ```

  - `expect_stderr`

    Regex expected to match OI's stderr.

    Example:
    ```
    expect_stderr = ".*Successfully detached from pid.*"
    ```

  - `expect_not_stdout`

    Regex expected to not match OI's stdout.

    Example:
    ```
    expect_not_stdout = "ABC"
    ```

  - `expect_not_stderr`

    Regex expected to not match OI's stderr.

    Example:
    ```
    expect_not_stderr = ".*ERROR.*"
    ```