# Integration Tests This directory contains test definition files for OI's integration tests. ## Running tests There are a number of ways to run these integration tests. 1. Run the `integration_test_runner` executable directly. This provides some additional options to aid debugging: - `--verbose` Verbose output - `--preserve` Do not clean up files generated by OID after tests are finished - `--force` Run tests that have been marked as "skipped" e.g. ```sh build/test/integration/integration_test_runner --gtest_filter=OidIntegration.primitives_int --verbose ``` 1. Run a number of the integration tests in parallel: ```sh ctest --test-dir build/test/integration -j$(nproc) [--tests-regex ] ``` ## Adding tests 1. Create a new test definition file in this directory and populate it as needed. See [Test Definition Format](#test-definition-format) for details. 1. Add your new definition file to the `INTEGRATION_TEST_CONFIGS` list in [`CMakeLists.txt`](CMakeLists.txt) ## Test Definition Format Test definitions are stored in the [TOML](https://toml.io/) file format. Example: ```toml includes = ["vector", "unordered_map"] definitions = ''' struct Foo { std::vector v; }; using Bar = std::unordered_map; ''' [cases] [cases.my_first_test_case] param_types = ["const Foo&", "const Bar&"] setup = ''' Foo foo; foo.v = {4,5,6}; Bar bar; bar[2] = 3; return {foo, bar}; ''' expect_json = '{"staticSize":4,"dynamicSize":32}' [cases.another_test_case] param_types = ["int"] setup = 'return 123;' ``` ### Details - `includes` Header files required for this test. Example: ``` includes = ["vector", "unordered_map"] ``` - `definitions` C++ type definitions required for a test can be defined here. Anything defined in this section will be automatically wrapped in a namespace and will be private to this test. Example: ``` definitions = ''' struct Foo { std::vector v; }; using Bar = std::unordered_map; ''' ``` - `thrift_definitions` Thrift type definitions can be specified here. These will be passed to the Thrift compiler which will generate C++ code from them. **CAUTION**: Generated Thrift types are not wrapped in a namespace, so type names must be globally unique between all tests. Example: ``` thrift_definitions = ''' struct MyThriftStruct { 1: optional i32 a; 2: optional i32 b; } ''' ``` - `raw_definitions` This section allows specifying of arbitrary C++ code which will be directly copied into the target program's source without being wrapped in a namespace. It should not be used for most tests. The purpose is to allow defining code required for a specific test to compile, avoiding the need to add new dependencies to the build system for one-off tests. - `cases` **Required** A list of individual test cases, each with their own setup, OI probe definition and expected results, but sharing any type definitions created in this test file. Test cases should be grouped into related areas and put into shared test files. - `param_types` **Required** Paramter types of the function to probe. oid does not have complete support for probing pass-by-value parameters, so it is recommended to define all parameters as reference or pointer types. Example: ``` param_types = ["const std::vector&", "const Foo&"] ``` - `arg_types` Types of the arguments being passed to the probed function. Defaults to `param_types` with const, volatile and references removed. It is only necessary to specify `arg_types` when they will differ from the parameter types expected by the probed function. This can be useful for testing inheritance. Example: ``` param_types = ["BaseClass *"] arg_types = ["DerivedClass"] ``` - `setup` **Required** A snippit of C++ code which creates and returns values to be passed to the function being probed as a part of this test case. The returned value should be a tuple of `param_types`, although the curly brakcets/braces can be omitted in most cases when there is only a single value in the tuple. Example: ``` setup = ''' std::vector ret = {1,2,3}; return {ret, Foo(1)}; ''' ``` - `type` OI probe type. Defaults to `entry`. Example: ``` type = "return" ``` - `args` Comma separated list of arguments to introspect. Defaults to `arg0`. Example: ``` args = "arg0,arg1" ``` - `cli_options` Additional command line arguments passed to oid. Example: ``` cli_options = ["--chase-raw-pointers"] ``` - `oid_skip`, `oil_skip` Skip running this test. Defaults to false. Example: ``` oid_skip = true ``` - `expect_oid_exit_code`, `expect_oil_exit_code` Exit code expected from OI. Defaults to 0. Example: ``` expect_oid_exit_code = 6 ``` - `expect_json` JSON expected to match results from OI. Only keys which appear in these expected results are used for comparison. This means that irrelevant or non-reproducable keys can be omitted and they will be ignored. Missing keys in the actual results will still cause test failures. Example: ``` expect_json = '{"staticSize":4,"dynamicSize":0}' ``` To ensure that a given key does not appear in the results, the special **NOT** key can be used, with the value set to the undesired key's name. This example checks that the JSON result does not contain the key "members": ``` expect_json = '{"NOT":"members"}' ``` The **NOT** key can also be used to check that a given key's value is not equal to some expected value. This example checks that the result has a key named `pointer`, but that its value is not equal to 0: ``` expect_json = '{"NOT":{"pointer":0}}' ``` - `expect_stdout` Regex expected to match OI's stdout. Example: ``` expect_stdout = ".*SUCCESS.*" ``` - `expect_stderr` Regex expected to match OI's stderr. Example: ``` expect_stderr = ".*Successfully detached from pid.*" ``` - `expect_not_stdout` Regex expected to not match OI's stdout. Example: ``` expect_not_stdout = "ABC" ``` - `expect_not_stderr` Regex expected to not match OI's stderr. Example: ``` expect_not_stderr = ".*ERROR.*" ```