tl;dr: Make an equivalent test runner to boost with simpler macros and tighter coupling to bitcoin-specific code.
Motivation
There are a number of problems with using dependencies in general. Bugs must either be promptly upstreamed or patched, projects may be abandoned, and they are not tailored to a particular use case. This PR removes boost for writing and running unit tests. In the case of Boost.Test, there are a number of issues which are listed below.
My problems with Boost.Test
Macros
When using boost, there are 3 ways of writing the same thing, each with different outcomes
BOOST_CHECK(expr1 == expr2): failures show the expression, but no values forexpr1andexpr2BOOST_CHECK_EQUAL(expr1, expr2): failures show the expression and values, but requires developers rememberCHECK_EQUALand does not use usual operandsBOOST_TEST(expr1 == expr2): failures show the expression and values, and developers may use familiar operands (==,!=)
All three are preserved to maintain backwards compatibility - presumably, but this results in all 3 being used throughout the repository. As a bonus there is a 4th way, with added context
BOOST_CHECK_MESSAGE(expr1 == expr2, "the context"): failures show the expression and message, but no values forexpr1andexpr2
To make debugging and review easier, all of these should be unified to a single macro. Test sites use operands, print helpful messages - values are always included, and do not vary in approach (deciding between BOOST_TEST, CHECK, and BOOST_CHECK_EQUAL is unintuitive)
<details> <summary>Example boost outputs</summary>
BOOST_AUTO_TEST_CASE(mustfail)
{
auto a{1};
auto b{2};
BOOST_CHECK(a == b);
BOOST_CHECK_EQUAL(a, b);
BOOST_TEST(a == b);
COutPoint outpoint_1{Txid{"0000000000000000000000000000000000000000000000000000000000000100"}, 0};
COutPoint outpoint_2{Txid{"0000000000000000000000000000000000000000000000000000000000000100"}, 1};
BOOST_CHECK(outpoint_1 == outpoint_2);
// Doesn't even compile
// BOOST_TEST(outpoint_1 == outpoint_2);
// BOOST_CHECK_EQUAL(outpoint_1, outpoint_2);
}
Running 1 test case...
test/argsman_tests.cpp(31): error: in "argsman_tests/mustfail": check a == b has failed
test/argsman_tests.cpp(32): error: in "argsman_tests/mustfail": check a == b has failed [1 != 2]
test/argsman_tests.cpp(33): error: in "argsman_tests/mustfail": check a == b has failed [1 != 2]
test/argsman_tests.cpp(36): error: in "argsman_tests/mustfail": check outpoint_1 == outpoint_2 has failed
</details>
<details> <summary>New outputs</summary>
TEST_CASE(mustfail)
{
auto a{1};
auto b{2};
CHECK(a == b);
COutPoint outpoint_1{Txid{"0000000000000000000000000000000000000000000000000000000000000100"}, 0};
COutPoint outpoint_2{Txid{"0000000000000000000000000000000000000000000000000000000000000100"}, 1};
CHECK(outpoint_1 == outpoint_2);
}
Running 1 test cases...
[FAIL]: test/argsman_tests.cpp:30: CHECK(a == b)
1 == 2
[FAIL]: test/argsman_tests.cpp:33: CHECK(outpoint_1 == outpoint_2)
COutPoint(0000000000, 0) == COutPoint(0000000000, 1)
[FAIL] mustfail (2/2 checks failed)
</details>
No repository context
Many types implement ToString, but boost has no way of meaningfully using these representations (will change w/ future std::format). With a repository-specific test runner we can use these today. See example above.
Not extensible
With the test runner in this repository we can add things such as #35139, implement string representations for more types, and implement ideas from #8670.
Other issues
More issues with boost are raised in #34666, #8670. This PR does not resolve all of them, but one it does address immediately is banning comparisons of integers with different signs, previously allowed by boost. Using a test runner within the source also makes IWYU easier to work with.
High level changes
For those running tests, not a tremendous amount has changed in this PR. For instance, the doc page remains valid. Changes to the runner options include anything that is not help, run_test, log_level, list_content. This includes catch_system_errors which is always no. The log_levels have been reduced to 5 levels.
Changes in writing tests
CHECKto compare two values with any==,!=,>, etcCHECK(a == b, "my message")to append a messageREQUIRE, same asCHECK, but will fail the test immediatelyCHECK_EQUAL_RANGES(it1, it2)to compare two rangesTEST_CASE(name)to add a testFIXTURE_TEST_CASE(name, Fixture)to add a test with a fixtureTEST_SUITE_BEGIN/ENDto declare a suite, optionally with fixture for each test in the suite
Migration
Boost.Test is removed in this PR, but the macros are aliased by BOOST_* counterparts so there are no merge conflicts. The idea would be to migrate test files to the new macros when there are no/low number of conflicts on that file.
Commits
The addition of the framework:
- test: Add header-only framework to util
Migration script (majority of the file changes):
- scripted-diff: Migrate tests to header-only framework
Low usage count macro removals:
- test: Remove use of
BOOST_FAIL - test: Remove use of
BOOST_TEST_INFO_* - test: Remove use of
BOOST_CHECK_CLOSE
Required for expression decomposition:
- crypto: Use constexpr to define output size
- util: Use constexpr in LockedPool definitions
- net: Make AddressPosition comparitor const
- test: Unroll
&&conditions in macros - test: Wrap
||expressions in macros - test: Wrap bitwise
&expressions in macros
To pass CI
- test: Use lambda for CRecipient to avoid gcc12 uninitialized warning
- test: Include memory in result_tests for IWYU
- test: Initialize variables in dbwrapper_tests
Build and config
- depends: drop test from Boost libraries
- cmake: drop vcpkg Boost Test check
- vcpkg: drop boost-test dependency
