Support gathering code coverage data for RPC tests with lcov #6813

Bitcoin Core has a nice way to gather code coverage data, though it only runs the unit tests and (optionally) the BitcoinJ tests.

With this update, the RPC tests (via qa/pull-tester/rpc-tests.py) are also executed, when gathering code coverage data with make cov.

When cleaning, there were several leftovers, and only the coverage related files in src/ were removed, while the ones in the other dirs remained. The leftovers from tests are now also removed, whereby qa/tmp/ is related to the BitcoinJ tests, and cache/ is related to RPC tests.

Because Python is used to run the RPC tests, it is explicitly checked, whether Python is available.

The configuration option --enable-extended-rpc-tests may be used to enable extended RPC tests, and the configuration option --enable-comparison-tool-reorg-tests may be used to enable extended tests via BitcoinJ. Note that the extended tests can take some time.

How to:

Generating coverage data requires lcov, which may be installed with:

sudo apt-get install lcov

There are a few configuration options, but no further setup is necessary.

To include the BitcoinJ tests, get the test tool:

TOOL_URL=https://github.com/theuni/bitcoind-comparisontool/raw/master/pull-tests-8c6666f.jar
TOOL_HASH=a865332b3827abcde684ab79f5f43c083b0b6a4c97ff5508c79f29fee24f11cd
wget $TOOL_URL -O ./share/BitcoindComparisonTool.jar
echo "$TOOL_HASH  ./share/BitcoindComparisonTool.jar" | shasum --algorithm 256 --check

The coverage data can then be gathered with:

./autogen.sh
./configure --enable-lcov --with-comparison-tool=./share/BitcoindComparisonTool.jar
make
make cov

It runs the tests and generates two HTML reports:

• test_bitcoin.coverage/index.html
• total.coverage/index.html

Example report:

• http://bitwatch.co/uploads/lcov-total-5e75bae983/

(only line and function coverage, no branch coverage, without extended tests)

Concept ACK - how does this relate to #6804 ?

It's not yet clear to me where #6804 is going, so this may be complementary. #6804 seems like a very easy and fast way to get an understanding of what's covered by the RPC tests, while this PR is more heavy. Given that the lcov testing already exists, extending it to include RPC tests seems nevertheless reasonable in my opinion.

Agree that this is compatible with #6804; I think this sort of "industrial strength" coverage report is absolutely worth having, but an expedient and simple view into which (if any) RPC functions are going untested is valuable also. Lcov certainly provides a more exact notion of what is tested, but it takes a bit more work to tease out which RPC functions are untested; the simple binary coverage provided in #6804 answers that question immediately and unambiguously, despite being primitive.

#6804 may also make it easier to gate builds down the road based on RPC coverage -- if the list of untested RPC functions calculated is non-empty, we fail. I think this would be nice to do once we have tests in place for each RPC function, It's not immediately apparent to me how we'd do this with just lcov.

So again, I think both are valuable and compatible. What do you think, @dexX7 @laanwj?

Concept ACK. What about linking Travis builds with Coveralls (example: https://coveralls.io/github/jonasschnelli/libbtc)?

Not saying that high test coverage stands for quality. But it is a good indicator for people writing new code that they should write unit tests...

@jamesob: based on what you posted in #6804, the list of uncovered RPCs is pretty large, and I agree, it's nice to have a quick overview. At some point it probably becomes less useful, if a binary outcome doesn't provide further insight (e.g. once all RPCs are rudimentary covered). @jonasschnelli: this is an interesting idea. Some time ago I actually tried to combine make cov and cpp-coveralls, but I kinda failed, and unfortunally didn't dig deeper.

On a slightly related note: it looks like coveralls is run two times in your repo (Linux+clang, Linux+gcc), and if this is an issue for you, you may look into a custom script deployment, which supports expressive conditionals (I used it to deploy docs).

One question that arrises is about the extra build time, if coverage data were gathered via Travis + Coveralls. I haven't timed it, but I have the impressionen it takes significantly longer to go though the process, so this would be something to consider. I'd like, if it were nicely integrated though.

@dexX7 the console output itself certainly becomes less useful after all RPCs are covered, but I think the lasting value is the ability to easily fail a build if someone introduces a new RPC without writing tests that exercise it as a client.

Anyway, I'll fully defer to you and @laanwj as to whether or not we should proceed with #6804 as well. You guys have much more experience in this ecosystem than I do!

ACK. Works for me.

@jonasschnelli wrote:

What about linking Travis builds with Coveralls

I just noticed #11680 which reminded me of this. Is it worth making an issue / PR?

I am running nightly reports: https://marcofalke.github.io/btc_cov/