RFC: Build feature set shouldn't affect the total unit test list #35741

Currently, the build system includes source files with unit tests depending on the actual build feature set, including options such as ENABLE_WALLET, ENABLE_IPC and BUILD_GUI_TESTS.

This leads to confusion when test results from different builds are aggregated together, for example, in https://my.cdash.org/index.php?project=bitcoin-core. It raises questions about why the total number of tests differs across builds.

The functional tests take a better approach by simply disabling tests for unavailable features.

I believe the unit tests should adopt the same approach.

In light of the ongoing modernization of the test framework (https://github.com/bitcoin/bitcoin/pull/33483 and #35713), I wonder whether those efforts are compatible with this idea.

Related: #31576.

I think it would be tedious to (let's say) compile a empty dummy test suite for every wallet test suite, even when the wallet is disabled. I think it is normal and expected for tests to not be counted when not compiled?

This leads to confusion when test results from different builds are aggregated together,

What build results are aggregated together and why? CDash displays test deltas (-1, +1) based on the previous build with the same build name and site name.

We should make sure that ENABLE_WALLET, ENABLE_IPC and BUILD_GUI_TESTS should remain consistent. Don't submit test results with different values of ENABLE_WALLET and identical build name. A naming convention for build names that encodes those settings helps. You seem to already do that with build names like "Alpine, depends, no GUI".

This leads to confusion when test results from different builds are aggregated together,

What build results are aggregated together and why? CDash displays test deltas (-1, +1) based on the previous build with the same build name and site name.

We should make sure that ENABLE_WALLET, ENABLE_IPC and BUILD_GUI_TESTS should remain consistent. Don't submit test results with different values of ENABLE_WALLET and identical build name. A naming convention for build names that encodes those settings helps. You seem to already do that with build names like "Alpine, depends, no GUI".

That's exactly right. However, because some features are platform-dependent, it's much more convenient to see exactly which tests are disabled for a specific environment when switching platforms locally to build and test.