Embedded test data in bitcoin_test #2985

This is the first in a line of PRs to address some of the concerns laid out in #2979.

Rather than adding more trickery to deduce where test data should be found, I believe that this is the more reliable solution. This uses the same mechanism as QT to include their assets in the resulting binary rather than reading it from the disk at runtime. Untested as-yet on OSX, but I believe it should work fine there.

Copied from the main commit:

Advantages:
- Tests become distributable.
- Cross-compile friendly. Build on one machine and execute in an arbitrary location on another.
- Easier testing for backports. Users can verify that tests pass without having to track down corresponding test data.
- More trustworthy test results and easier quality assurance as tests make fewer assumptions about their environment.
- Tests could theoretically run at client/daemon startup and exit on failure.

Disadvantages:
- Required 'hexdump' build-dependency. This is a standard bsd tool that should be usable everywhere. It is likely already installed on all build-machines.
- Tests can no longer be fudged after build by altering test-data.

I'm a bit divided on this; on one hand, this is very useful if the tests are to be distributed/installed. On the other hand is that a common use case? I always run them within the build directory.

What if one wants to edit the test data while testing? One of the advantages of data-driven testing is not having to recompile every time after changing the test data.

@laanwj It's common for me, as I'm usually doing cross/gitian builds. This would encourage devs to use tests when interacting with users, as-in: "please post the output of the tests", which can now be easily run by users and trusted by the bug-hunter.

But you could also do that by making the tests expect the data in a subdirectory of the executable, and then send a package with tests + data together.

In any case I'm not against this, but in my own projects I usually do testing with a python harness and having to compile as little as possible while designing tests is a great advantage (faster test cycle). So some way I feel this is a step in the wrong direction.

But let's see what the other devs think.

Edit: a compromise would also be possible, allow overriding the data directory on the command line and default to built-in data. The qt resource system allows this as well IIRC.

I think this is a step in the wrong direction. I have never wanted to give an end-user unit tests, but found it incredibly helpful to be able to rapidly run tests by just editing a .json file and then re-running ./test_bitcoin.

Fair enough. Different perspectives I suppose, and ease of use by the devs trumps everything else for sure. But just to throw out a data-point from my desktop:

#change the test
cory@cory-i7:~/dev/bitcoin/src/test(included-tests)$ touch data/base58_encode_decode.json

#make check will rebuild as necessary and run
cory@cory-i7:~/dev/bitcoin/src/test(included-tests)$ time make check
Generated data/base58_encode_decode.json.h
make  check-am
make[1]: Entering directory `/home/cory/dev/bitcoin/src/test'
  CXX    test_bitcoin-base58_tests.o
  CXXLD  test_bitcoin
make  check-TESTS
make[2]: Entering directory `/home/cory/dev/bitcoin/src/test'
Running 95 test cases...

*** No errors detected
PASS: test_bitcoin
=============
1 test passed
=============
make[2]: Leaving directory `/home/cory/dev/bitcoin/src/test'
make[1]: Leaving directory `/home/cory/dev/bitcoin/src/test'

real    0m2.895s

With such small build times, I don't really care about it. I don't think this is really a problem, but avoiding the absolute paths mess is certainly an advantage.

Data is a very good way of making me change my mind.

Recompiles are fast enough, so ACK after rebase and making pull-tester happy.

@gavinandresen Could you please install 'bsdmainutils' in the pull-tester? After that i'll rebase on master and push for a new test.

I installed bsdmainutils in the pull-tester's chroot environment.

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/152e51c7af2624831cc4796e06bf3b72787cc85f for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

Where are we on updating documentation for autotools? The new bsdmainutils apt-get dependency needs to be documented.

FYI: /usr/bin/hexdump seems to be standard on OSX.

I think after the debug/release stuff gets fixed up, that marks a good time to do the thorough docs overhaul.

Ack on OSX, it seems to be standard (and compliant for once, woohoo!) from at least 10.6-10.8.

ACK; seems to build and run fine.