object-introspection/test/integration/README.md

Integration Tests

This directory contains test definition files for OI's integration tests.

Running tests

There are a number of ways to run these integration tests.

1. Run the integration_test_runner executable directly. This provides some additional options to aid debugging:

   • --verbose Verbose output
   • --preserve Do not clean up files generated by OID after tests are finished
   • --force Run tests that have been marked as "skipped"

   e.g.

   build/test/integration/integration_test_runner --gtest_filter=OidIntegration.primitives_int --verbose
   

2. Run a number of the integration tests in parallel:

   ctest --test-dir build/test/integration -j$(nproc) [--tests-regex <regex>]
   

Additional command line arguments can be passed to oid by setting the OID_TEST_ARGS environment variable, e.g.:

OID_TEST_ARGS="-fmy-new-feature" build/test/integration/integration_test_runner
OID_TEST_ARGS="-Fold-feature" ctest --test-dir build/test/integration

Adding tests

Create a new test definition file in this directory and populate it as needed. See Test Definition Format for details. It will be automatically picked up by CMake on your next build.

Test Definition Format

Test definitions are stored in the TOML file format.

Example:

includes = ["vector", "unordered_map"]
definitions = '''
  struct Foo {
    std::vector<int> v;
  };
  using Bar = std::unordered_map<int, int>;
'''
[cases]
  [cases.my_first_test_case]
    param_types = ["const Foo&", "const Bar&"]
    setup = '''
      Foo foo;
      foo.v = {4,5,6};
      Bar bar;
      bar[2] = 3;
      return {foo, bar};
    '''
    expect_json = '{"staticSize":4,"dynamicSize":32}'
  [cases.another_test_case]
    param_types = ["int"]
    setup = 'return 123;'

Details

• includes

  Header files required for this test.

  Example:

  includes = ["vector", "unordered_map"]
  

• definitions

  C++ type definitions required for a test can be defined here.

  Anything defined in this section will be automatically wrapped in a namespace and will be private to this test.

  Example:

  definitions = '''
    struct Foo {
      std::vector<int> v;
    };
    using Bar = std::unordered_map<int, int>;
  '''
  

• thrift_definitions

  Thrift type definitions can be specified here. These will be passed to the Thrift compiler which will generate C++ code from them.

  CAUTION: Generated Thrift types are not wrapped in a namespace, so type names must be globally unique between all tests.

  Example:

  thrift_definitions = '''
    struct MyThriftStruct {
      1: optional i32 a;
      2: optional i32 b;
    }
  '''
  

• raw_definitions

  This section allows specifying of arbitrary C++ code which will be directly copied into the target program's source without being wrapped in a namespace.

  It should not be used for most tests. The purpose is to allow defining code required for a specific test to compile, avoiding the need to add new dependencies to the build system for one-off tests.

• cases Required

  A list of individual test cases, each with their own setup, OI probe definition and expected results, but sharing any type definitions created in this test file.

  Test cases should be grouped into related areas and put into shared test files.

  • param_types Required

    Paramter types of the function to probe.

    oid does not have complete support for probing pass-by-value parameters, so it is recommended to define all parameters as reference or pointer types.

    Example:

    param_types = ["const std::vector<int>&", "const Foo&"]
    

  • arg_types

    Types of the arguments being passed to the probed function. Defaults to param_types with const, volatile and references removed.

    It is only necessary to specify arg_types when they will differ from the parameter types expected by the probed function. This can be useful for testing inheritance.

    Example:

    param_types = ["BaseClass *"]
    arg_types = ["DerivedClass"]
    

  • setup Required

    A snippit of C++ code which creates and returns values to be passed to the function being probed as a part of this test case. The returned value should be a tuple of param_types, although the curly brakcets/braces can be omitted in most cases when there is only a single value in the tuple.

    Example:

    setup = '''
      std::vector<int> ret = {1,2,3};
      return {ret, Foo(1)};
    '''
    

  • type

    OI probe type. Defaults to entry.

    Example:

    type = "return"
    

  • args

    Comma separated list of arguments to introspect. Defaults to arg0.

    Example:

    args = "arg0,arg1"
    

  • target_function

    Symbol of the target function to be traced, when the auto-generated target function is not sufficient.

    The fields param_types and setup are no longer required if target_function is defined.

    Example:

    target_function = "my_function"
    

    Implies oil_disable.

  • features

    Append this list of features to the configuration. This works for all types of test.

    Example:

    features = ["chase-raw-pointers"]
    

  • cli_options

    Additional command line arguments passed to oid.

    Example:

    cli_options = ["-Ftype-graph"]
    

  • skip, oid_skip, oil_skip

    Skip running this test for oid and/or oil.

    Pass in a string to provide a reason for skipping the test.

    Example:

    oid_skip = "oid support not implemented yet"
    

  • oil_disable

    Disable this test for oil.

    When a test doesn't make sense for oil, disable it rather than skipping it, to avoid it showing up in the aggregate test results entirely.

    Example:

    oil_skip = "oil can't chase raw pointers safely"
    

  • expect_oid_exit_code, expect_oil_exit_code

    Exit code expected from OI. Defaults to 0.

    Example:

    expect_oid_exit_code = 6
    

  • expect_json

    JSON expected to match results from OI.

    Only keys which appear in these expected results are used for comparison. This means that irrelevant or non-reproducable keys can be omitted and they will be ignored. Missing keys in the actual results will still cause test failures.

    Example:

    expect_json = '{"staticSize":4,"dynamicSize":0}'
    

    To ensure that a given key does not appear in the results, the special NOT key can be used, with the value set to the undesired key's name.

    This example checks that the JSON result does not contain the key "members":

    expect_json = '{"NOT":"members"}'
    

    The NOT key can also be used to check that a given key's value is not equal to some expected value.

    This example checks that the result has a key named pointer, but that its value is not equal to 0:

    expect_json = '{"NOT":{"pointer":0}}'
    

  • expect_stdout

    Regex expected to match OI's stdout.

    Example:

    expect_stdout = ".*SUCCESS.*"
    

  • expect_stderr

    Regex expected to match OI's stderr.

    Example:

    expect_stderr = ".*Successfully detached from pid.*"
    

  • expect_not_stdout

    Regex expected to not match OI's stdout.

    Example:

    expect_not_stdout = "ABC"
    

  • expect_not_stderr

    Regex expected to not match OI's stderr.

    Example:

    expect_not_stderr = ".*ERROR.*"