[tests] Make comp test framework more debuggable #11468

We should remove the comparison test framework entirely (see #10603).

Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging.

Before this PR:

→ ./p2p-fullblocktest.py 
2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr
2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975
2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS
2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS
2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS
2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main
    self.run_test()
  File "./p2p-fullblocktest.py", line 72, in run_test
    self.test.run()
  File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run
    assert test_number != 4
AssertionError
2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes
2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr
2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log

With this PR:

→ ./p2p-fullblocktest.py 
2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf
2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783
2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184
2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193
2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205
2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208
2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main
    self.run_test()
  File "./p2p-fullblocktest.py", line 72, in run_test
    self.test.run()
  File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run
    assert test_number != 4
AssertionError
2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes
2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf
2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log

ACK https://github.com/bitcoin/bitcoin/pull/11468/commits/7481579131e3a330300d2c7f9a8f93aa67be33e1

utACK 7481579.

I like seeing Test X: PASS and I in the same line Test X: FAILED. How about wrapping in try/catch instead?

I like seeing Test X: PASS and I in the same line Test X: FAILED. How about wrapping in try/catch instead?

I don't think there's need for that. The problem this is solving is not having a way to tell where in the test script execution has reached when there's a failure. I've done that with the smallest change I could.

We should stop using the comparison test framework entirely for these static tests (see https://botbot.me/freenode/bitcoin-core-dev/2017-06-15/?msg=87309124&page=3) so I don't intend to make bigger changes than is necessary.

Yes, that should be a bigger patch. Just read the IRC log 👍

Obvious utACK 7481579. Sorry that I missed this.