Effective Quality in C++

Urs Fässler

In the ever-evolving landscape of software development, maintaining high-quality and reliable software is paramount. To achieve this, rigorous testing practices and robust development methodologies are essential. This article delves into various testing strategies and tools specific to C++, emphasizing the importance of automated regression tests, test coverage, compiler warnings, code formatting, static code analysis, sanitizers, and continuous integration (CI). By adopting these practices, teams can ensure their software remains reliable, maintainable, and bug-free. We offer comprehensive support to help teams implement these tools and practices effectively, enabling them to achieve optimal results.

Tests

Testing is an integral part of software development. Automated regression tests instill confidence in the software’s correctness, ensuring consistent behavior, and allow to resolve bugs efficiently. Rigorous automated testing validates that our software functions as intended and remains reliable over time.

Various types of tests form a comprehensive testing strategy:

  • Unit Tests: Focus on individual units in isolation to verify their correctness.

  • Component Tests: Examine interactions between integrated units to ensure they work together as expected.

  • Integration Tests: Evaluate the complete system’s interactions and workflows, ensuring all parts of the system function harmoniously.

  • Hardware Tests: Involve actual hardware (when applicable) to ensure the business logic is correctly wired to the hardware and operating system.

  • Acceptance Tests: Confirm that the software meets the required specifications and performs its intended functions.

Effective tests possess certain key properties:

  • Isolated: Operate independently of other tests and the environment, ensuring reliable results.

  • Fast: Execute quickly to maintain efficiency, with all unit tests ideally running in under one second.

  • Deterministic: Yield consistent results on every run regardless of the machine or time of day.

  • Focused: Each test should address a single aspect to provide clear and specific feedback on failures.

Adopting Test Driven Development (TDD) facilitates the creation of testable software. TDD promotes the development of small, independent units by writing tests before writing the actual code. Adhering to clean code principles is also crucial for maintaining modularity and testability. These principles guide developers in writing code that is easy to understand, maintain, and test, ensuring long-term reliability and efficiency.

Test Coverage

Measuring test coverage ensures that the code is thoroughly tested. It highlights potentially risky areas that require extra caution during review and provides feedback on the adequacy of tests for new code.

Tools like gcov can be used to measure test coverage. Enable coverage by compiling and linking with the --coverage flag. After running the tests, generate the coverage report with the following command:

gcovr build/ --html-details index.html --xml cobertura.xml

For more information, see GCC Program Instrumentation Options.

Monitoring Number of Tests

The absolute number of tests does not provide much insight, but it is motivating to see the number increasing. However, the number of tests and test coverage should never be targets in themselves. According to Goodhart’s law, "when a measure becomes a target, it ceases to be a good measure". Instead, these metrics should help the team monitor their effort and make informed decisions.

Compiler Warnings

Always use strict compiler warnings to catch potential issues early. The minimum recommended flags are -Wall -Wextra -Wpedantic. Numerous other options are available; see the GCC Options to Request or Suppress Warnings for more details.

Enable these warnings for all parts of the code, including the main application, custom libraries, tests, and helper tools.

Only when compiling in Continuous Integration, treat warnings as errors using the -Werror flag. This approach allows developers to experiment freely during development while enforcing strict standards for committed code. Additionally, it facilitates easy compilation with different compilers or versions that might report warnings.

Code Formatting

Enforcing a consistent code style is crucial for readability and maintainability. Consistent formatting reduces cognitive load and makes code reviews easier. Tools like clang-format can automate this process.

Configuration can be defined in a .clang-format file:

Language: Cpp
ColumnLimit: 100
IndentWidth: 4
SpaceBeforeAssignmentOperators: true

To check code formatting:

files=$(find . -type d -o -regex '.*\.\(cpp\|hpp\)' -print)
clang-format --style=file --Werror --dry-run --verbose ${files}
echo $?

Agree on a common style once and adhere to it, as consistency is more important than personal preference.

Static Code Analysis

Static code analyzers like clang-tidy help detect potential issues in the code. To use it, generate a compilation database and run the analyzer:

cmake build/ \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=1
clang-tidy -p build/ old.cpp \
  --warnings-as-errors=\* -checks= \
    modernize-redundant-void-arg, \
    readability-implicit-bool-conversion
echo $?

For example, given the code:

void bar(void) {}

void foo(int* ptr)
{
  if (ptr) {
    bar();
  }
}

We get the report:

old.cpp:1:10: warning: redundant void argument list in function definition [modernize-redundant-void-arg]
void bar(void) {}
         ^~~~

old.cpp:5:7: warning: implicit conversion 'int *' -> bool [readability-implicit-bool-conversion]
  if (ptr) {
      ^
          != nullptr

Sanitizers

Sanitizers are tools designed to detect various types of bugs and issues in software programs. They work by instrumenting code to identify problems such as memory errors, data races, and undefined behavior during runtime. Common types of sanitizers include Address Sanitizer (for memory errors), Thread Sanitizer (for data races), and many more (see -fsanitize= in GCC Program Instrumentation Options).

We use sanitizers because they have proven invaluable during development. These tools have detected invalid memory access in complex applications on multiple occasions. They can identify subtle and infrequent bugs, saving us from difficult and lengthy bug-hunting sessions.

Address Sanitizer

The address sanitizer (ASan) is a tool that helps detect memory errors such as out-of-bounds access and use-after-free bugs in programs. ASan is particularly useful for debugging and improving the reliability of software by identifying problematic memory usage patterns. It can be enabled with the compiler flag -fsanitize=address.

Given we compile the snippet use-after-free.c with Asan:

#include <stdlib.h>
int main() {
  char *x = (char*)malloc(10 * sizeof(char*));
  free(x);
  return x[5];
}

When we run the application or test, then we get the report:

==9901==ERROR: AddressSanitizer: heap-use-after-free on ...
READ of size 1 at 0x60700000dfb5 thread T0
    #0 0x45917a in main use-after-free.c:5
freed by thread T0 here:
    #1 0x45914a in main use-after-free.c:4
previously allocated by thread T0 here:
    #1 0x45913f in main use-after-free.c:3
SUMMARY: AddressSanitizer: heap-use-after-free use-after-free.c:5 main

Thread Sanitizer

The thread sanitizer (TSan) is a tool designed to detect data race bugs in programs. It instruments memory access instructions to identify these issues, helping developers improve the reliability and correctness of their multi-threaded applications. As it is not compatible with the Address Sanitizer, the application or test needs to be compiled and run a second time. It can be enabled with the compiler flag -fsanitize=thread.

Given we compile the snippet simple_race.cc with TSan:

int Global;

void Thread1() {
  Global++;
}

void Thread2() {
  Global--;
}

When we run the application or test, then we get the report:

WARNING: ThreadSanitizer: data race (pid=26327)
  Write of size 4 at 0x7f89554701d0 by thread T1:
    #0 Thread1() simple_race.cc:4 (exe+0x000000006e66)
  Previous write of size 4 at 0x7f89554701d0 by thread T2:
    #0 Thread2() simple_race.cc:8 (exe+0x000000006ed6)

Continuous Integration

Continuous Integration (CI) is a development practice where developers integrate code into a shared repository frequently, usually several times a day. This allows teams to detect problems early and maintain a consistently shippable codebase. CI is the a standard pillar in our strategy for building robust software, ensuring that each code change is validated through rigorous processes.

A CI server runs all automated tests to ensure code quality and functionality. It also executes the discussed additional checks, such as static code analysis and code formatting. Building all applications verifies proper compilation and functionality. Compiling different configurations with various compilers helps catch portability issues.

Effective CI practices include running CI jobs on every branch and push to catch issues early. Keeping the CI pipeline short, ideally under 10 minutes, allows developers to stay focused on the same task, preventing task switches and reducing mental load. Failed CI jobs prevent merges, maintaining code integrity. For more insights, refer to Martin Fowler’s article on Continuous Integration.

Conclusion

Implementing a thorough testing strategy and leveraging the right tools is crucial for building high-quality C++ software. Automated tests, rigorous test coverage, strict compiler warnings, consistent code formatting, and static code analysis all contribute to the reliability and maintainability of the codebase. Additionally, sanitizers and continuous integration enhance the development process by catching subtle bugs and ensuring code integrity. By embracing these practices, C++ development teams can create robust software that meets high standards of quality and performance.

We understand that integrating these tools and practices can be challenging. That’s why we offer comprehensive support to assist teams in adopting and effectively using these methodologies. Our support ensures a smooth implementation process, helping teams achieve the best possible outcomes for their C++ projects.

Modernisierung

Ich modernisiere Ihre Software, damit sie zukunftssicher wird. Dies umfasst in der Regel eine modularere Gestaltung des Source-Codes und die Einführung automatisierter Tests, die in einer Continuous-Integration-Umgebung ausgeführt werden. Gleichzeitig implementiere ich gemeinsam mit dem Team neue Features.

read more