# Automated Tests for dv-runtime and DV Modules

The tests located in this folder make use of dv-python in order to run integration tests for `dv-runtime` and `DV`
modules.

\[\[_TOC_\]\]

## Running Existing Tests

The tests can be executed either using the makefiles generated by `cmake` or by calling `runtime-test.py` directly.

### Using make

The easiest way to run the tests is by using `make`

1. Build dv-runtime: `make`
1. Execute the tests: `make tests <options>`

Command line options:

- `TESTS=<tests>`
  - `all`: execute all tests
  - `minimal`: execute minimal tests only (00 - 03)
  - `noperf`: execute all tests except performance tests (default)
  - `perf`: execute performance tests only
  - `<ids>`: list of test IDs to be executed, e.g. `TESTS=05,08,09_01`
- `PERF_DATA_APPEND=<perf_data_append>` (For details see [Performance Tests](#performance-tests))
  - `1`: Append to existing performance baseline
  - `0`: Don't append to existing performance baseline

### Using `runtime-test.py` Directly

1. Build dv-runtime: `make`
1. Execute the tests: `python3 <path_to_test_runtime.py> <options>`

Command line options:

- `--env-vars <env_vars>`
- `--runtime-exec <RUNTIME_EXEC>`
- `--control-exec <CONTROL_EXEC>`
- `--modules-path <modules_path>`
- `--tests <tests>`
  - `all`: execute all tests
  - `minimal`: execute minimal tests only (00 - 03)
  - `noperf`: execute all tests except performance tests (default)
  - `perf`: execute performance tests only
  - `<ids>`: list of test IDs to be executed, e.g. `--tests 05,08,09_01`
- `--perf-data-append <perf_data_append>` (For details see [Performance Tests](#performance-tests))
  - `1`: Append to existing performance baseline
  - `0`: Don't append to existing performance baseline

## Creating New Tests

### The Anatomy of a Test

Each test file should contain a single test. Tests are to be named according to the following pattern:
`test_<test_id>_<sub_id>_<name>`

Tests for a single module shall have a common `test_id`, for example tests `test_05_01` to `test_05_05` all test the
accumulator module.

As an example let's look at test `05_03`

```python
from test_framework._support import *
from test_framework._test_params import *


@dvxplorer_input
@output_hash("331bc6cf66b2f2da63ae3a7c2cd9c6952c9a37e68e72db4f448d2d2efde288fe")
def test_05_03_accumulatorStep(self):
    module_parameters = [["decayFunction", "Step"]]
    module = dv_module("dv_accumulator", [["events", "input[events]"]], ["frames"], config_options=module_parameters)
    test_modules_with_io(self, [module])
```

This test makes use of the dvxplorer input file, creates the `dv_accumulator` module as the module under test, and
specifies the configuration of this module. Lastly, the function `test_modules_with_io` is called in order to execute
the test.

The function decorators above the test function are used to specify test parameters. This example selects the dvxplorer
input file and sets the hash of the produced output file to an expected value. The provided options for the
configuration of test parameters are shown [below](#test-parameter-configuration).

A module under test is represented by the `dv_module` class. This module encapsulates the inputs, outputs and
configration options of a module.

The module configuration for this example is explained below:

- Configuration options:
  - A single option is specified: `["decayFunction", "Step"]`.
  - Multiple options can be passed in an array: `[["option1": "value1"], ["option2", "value2"]]`
- Inputs:
  - A single input is specified: `["events", "input[events]"]`.
  - Multiple options can be passed in an array: `[["name1": "source1"], ["name2", "source2"]]`
- Outputs
  - A single output is specified: `"frames"`.
  - Multiple options can be passed in an array: `["name1", "name2"]`

The call to `test_modules_with_io` is then used to trigger the test. More information on the available test functions is
explained [below](#test-functions)

### Test Parameter Configuration

The configuration of test parameters is implemented using python decorators. The following configuration options are
provided:

#### Setting the Input File

The input files that are used in the GitLab CI are located in the repository `infra/docker-files/test_data`. If you want
to run the existing tests locally, you need to check out this repository and modify the corresponding file path in
`test_framework/_test_params.py`.

```python
@davis_input
@dvxplorer_input
@dvxplorer_performance_input
@aedat2_3_input
@aedat3_0_input
```

#### Setting the Expected Output Hash

```python
@output_hash("foo") # A single output file and the same hash on all platforms
@output_hash(["foo", "bar"]) # A single output file but different platforms yield different outputs
@output_hashes([["output0_hash", "foo"],
["output1_hash", "bar"], ... ]) # Multiple output files, same hash on all platforms
@output_hashes([["output0_hash", ["foo", "bar"]],
["output1_hash", "baz"], ... ]) # Multiple output files, different hashes on different platforms for output0, but same hash on all platforms on output1
```

#### Setting the Expected Output File Size

```python
@output_size(42) # A single output file and the same size on all platforms
@output_size([42, 24]) # A single output file but different platforms yield different outputs
@output_sizes([["output0_size", 42],
["output1_size", 24], ... ]) # Multiple output files, same size on all platforms
@output_sizes([["output0_size", [42, 24]],
["output1_size", 9000], ... ]) # Multiple output files, different sizes on different platforms for output0, but same size on all platforms on output1
```

#### Setting the Expected Output Written Data Size

```python
@output_written_data_size(42) # A single output file and the same size on all platforms
@output_written_data_size([42, 24]) # A single output file but different platforms yield different outputs
@output_written_data_sizes([["output0_written_data_size", 42],
["output1_written_data_size", 24], ... ]) # Multiple output files, same size on all platforms
@output_written_data_sizes([["output0_written_data_size", [42, 24]],
["output1_written_data_size", 9000], ... ]) # Multiple output files, different sizes on different platforms for output0, but same size on all platforms on output1
```

#### Enabling Perfomance Testing

```python
@test_performance
```

#### Disabling a Test

```python
@disable
```

#### Marking a Test as Expected to Fail

```python
@expected_to_fail
```

#### Ignoring Failure of a Test

```python
@ignore_fail
```

#### Marking a Test for Which a Connected Camera is Required

```python
@camera_required
```

#### Marking a Test for Which a Calibration File is Required

```python
@calibration_required
```

#### Initialising a Test Which Requires no Configuration Parameters

As each test needs certain implicitly defined configurations, such as the name of the test, the standard parameters must
be configured. This happens automatically when parameters are specified. In case no parameters need to be specified, the
test must be marked with the following decorator:

```python
@standard_params
```

### Test Functions

To execute the test, several functions are provided in `test_framework/_support.py`. These functions set up the required
module network, execute the test and verify the resulting data.

#### Create the File I/O for a Module and Run the Test

This function creates one file input module, one file output for any output of any module specified in `modules`, and
then executes the test. If any input- or output parameters are required they can be passed to `input_parameters` or
`output_parameters`.

```python
test_modules_with_io(self, modules, input_parameters=None, output_parameters=None)
```

#### Test the File I/O Directly

This function creates one file input module, one file output for any output of any element specified in `output_inputs`,
and then executes the test. The argument `output_inputs` takes a list of strings, each element therein contains an
output of the module `input`, e.g. `output_inputs = ["events", "imu", "frames"]`. If any input- or output parameters are
required they can be passed to `input_parameters` or `output_parameters`.

```python
test_io_with_no_module(self, output_inputs, input_parameters=None, output_parameters=None)
```

#### Create the File Input for a Module and Run the Test

This function creates one file input module, no output, and then executes the test. If any input parameters are required
they can be passed to `input_parameters` or `output_parameters`.

```python
test_modules_with_input(self, modules, input_parameters=None)
```

#### Create the File Outputs for a Module and Run the Test

This function creates no file input module, one file output for any output of any module specified in `modules`, and
then executes the test. If any input- or output parameters are required they can be passed to `input_parameters` or
`output_parameters`.

```python
test_modules_with_outputs(self, modules, output_parameters=None)
```

#### Don't Create any I/O and Run the Test for a Module

This function does not create any input or output modules, it only executes the test.

```python
test_modules(self, modules)
```

### Performance Tests

Performance tests provide the functionality to compare two versions of a module with respect to the following
performance metrics:

- throughput
  - events/s, frames/s, ...
- cycle time
  - time spent in `run()`

In order to measure the performance, the test can be marked with the `@test_performance` decorator.

To run a performance test, we need some data to compare the performance of your code to. For this purpose we need to
measure a performance baseline. The test script automatically creates such a baseline if there is none yet present. The
baseline will be put into the folder `perf_data/` and named after your test (`perf_<test_name>.json`). If this data is
present, it will be used and the measured performance statistics will be compared to this data.

The test criteria are that the throughput be at least as high as:

```math
\mu_{tp} - \sigma_{tp}
```

, the cycle time be no more than:

```math
\mu_{ct} + \sigma_{ct}
```

, and the cpu time no more than:

```math
\mu_{cput} + \sigma_{cput}
```

As a visualization, the data is plotted to histograms, along with the minimum and maximum values explained above, as
well as the measured value. An example plot is shown below. This plot shows the comparison of the performance of the
`dv_export_csv` after optimisation to a baseline taken before said optimisation.

![Example Performance Plot](doc/perf_example.png)

If you require to add more data to the baseline, you can do so by specifying `--perf-data-append 1` in the command line
for the python script, or the equivalent `PERF_DATA_APPEND=1` for the makefile.

**TLDR**; To quantify whether your changes improved the performance of a module, do the following

1. Build the state before optimisation: `make`
1. Run the performance test `make tests TESTS=<test_id>`
1. Build the new state after optimisation: `make`
1. Run the performance test again: `make tests TESTS=<test_id>`
1. Look at the log output and the plots in `perf_data/`
