Table of Contents
Autoware.Auto uses the ament_cmake framework to build and run tests. The same framework is also used to analyze the test results.
ament_cmake provides several convenience functions to make it easy to register tests in a CMake-based package and to ensure that JUnit-compatible result files are generated. It currently supports a few different testing frameworks like pytest, gtest, and gmock.
See below for an example of using ament_cmake_gtest with colcon test. All other tests follow a similar pattern.
This example assumes that the package my_cool_pkg is generated with autoware_auto_create_package.
Create a unit test with gtest
In my_cool_pkg/test, create the gtest code file test_my_cool_pkg.cpp:
For more examples of gtest features, see the gtest repo.
In package.xml, add the following line:
Next add an entry under BUILD_TESTING in the CMakeLists.txt to compile the test source files:
This automatically links the test with the default main function provided by gtest. The code under test is usually in a different CMake target (${PROJECT_NAME} in the example) and its shared object for linking and include directories need to be added.
To register a new gtest item, wrap the test code with the macro TEST (). TEST () is a predefined macro that helps generate the final test code, and also registers a gtest item to be available for execution. Test case name should be in CamelCase. This is as gtest inserts an underscore between the fixture name and the class case name when creating the test executable.
gtest/gtest.h also contains predefined macros of gtest like ASSERT_TRUE(condition), ASSERT_FALSE(condition), ASSERT_EQ(val1,val2), ASSERT_STREQ(str1,str2), EXPECT_EQ(), etc. ASSERT_* will abort the test if the condition is not satisfied, while EXPECT_* will mark the test as failed but continue to next test condition. More information about gtest can be found in the gtest repo.
In the demo CMakeLists.txt, ament_add_gtest is a predefined macro in ament_cmake_gtest that helps simplify adding gtest code. Details can be viewed in ament_add_gtest.cmake.
Build test
By default, all necessary test files (ELF, CTesttestfile.cmake, etc.) are compiled by colcon:
Test files are generated under ~/workspace/build/my_cool_pkg.
Run test
To run test on a specific package, call:
The test command output contains a brief report of all the test results.
To get job-wise information of all executed tests, call:
Look in the ~/workspace/log/test_<date>/<package_name> directory for all the raw test commands, std_out, and std_err. There's also the ~/workspace/log/latest_*/ directory containing symbolic links to the most recent package-level build and test output.
To print the tests' details while the tests are being run, use the --event-handlers console_cohesion+ option to print the details directly to the console:
Coverage
Loosely described, a coverage metric is a measure of how much of the program code has been exercised (covered) during testing.
In Autoware.Auto the lcov tool (which uses gcov internally) is used to measure:
- Statement coverage
- Function coverage
- Branch coverage
lcov also collects the results and generates html to visualize the coverage information.
Coverage for the latest successful CI run on the master branch is here.
Use the commands below to generate coverage information for my_cool_pkg:
- Note
package_coverage.shprompts to deletebuild,install, andlogdirectories, if present. Answer withyto delete, or clean your build before generating the coverage report.
This produces the high-level coverage report and also generates a coverage folder with an index.html file in it assuming the build and tests passed successfully. The resulting lcov/index.html will have a similar form to the following:
In Autoware.Auto, there is a separate "coverage" job as part of the CI pipeline that measures and reports the test coverage of a merge request:
and the summary statistics are printed near the end of the log output:
