Retire the comparison test framework #10603

1. 
   jnewbery commented at 5:35 pm on June 15, 2017: member

   The comparison test framework was introduced in #5981 as a successor to the Java comparison tool, following discussion in #4545. The Java comparison tool was finally removed in #8504.

   The python comparison test framework was originally intended to start multiple bitcoind nodes on different versions, send various blocks/transactions to those nodes, and compare that the nodes reach the same state. However, the tests are now usually run through the test_runner.py framework with only one node being run. The comparison part of the comparison test framework is not even used.

   Tests written in the ‘comparison’ style make use of TestManager and TestInstance classes and yield statements throughout the test. The framework is overly complex and not flexible enough to be really useful. In addition, they’re difficult to write, understand, update and troubleshoot. There are only 7 tests which use the comparison test style - we should rewrite them in the regular BitcoinTestFramework style.

   subtasks

   • Convert bip65-cltv-p2p.py to use the BitcoinTestFramework (#10695)
   • Convert bipdersig-p2p.py to use the BitcoinTestFramework (#10695)
   • Convert invalidtxrequest.py to use the BitcoinTestFramework (#11771)
   • Convert p2p_invalid_block.py to use the BitcoinTestFramework (#11772)
   • Convert feature_block.py to use the BitcoinTestFramework (#11773)
   • Convert feature_csv_activation.py to use the BitcoinTestFramework (#11817)
   • Delete redundant feature_bip9_softforks.py (#11818)
   • Remove the ComparisonTestFramework code (#11818)
2. fanquake added the label Tests on Jun 16, 2017
3. MarcoFalke deleted a comment on Jun 18, 2017
4. 
   MarcoFalke commented at 8:32 pm on June 18, 2017: member

   I’d be surprised if our tests run with a version of bitcoind older than a couple of weeks. Usually our test framework is using all “bleeding edge” features of the master branch and breaks if those features are not included in the binary. E.g. the binary of the last release.

   We would probably need to enforce compatibility, i.e. run the comparison tests as part of travis.

5. 
   TheBlueMatt commented at 2:46 pm on June 20, 2017: member

   It would be nice for the test to support running against an old bitcoind, but I see no reason why it should need to run multiple bitcoinds at once (we already support providing a path to a bitcoind, or, obviously, you can just leave an old binary in the tree).
6. MarcoFalke referenced this in commit 76b3349159 on Nov 10, 2017
7. laanwj referenced this in commit 2dbc4a4740 on Feb 13, 2018
8. MarcoFalke referenced this in commit 0630974647 on Mar 13, 2018
9. laanwj referenced this in commit f0f9732d05 on Mar 29, 2018
10. 
    jnewbery commented at 3:41 pm on March 29, 2018: member

    #11773 merged. Next at bat: #11817.
11. MarcoFalke referenced this in commit 2a09a78c08 on Apr 1, 2018
12. 
    jnewbery commented at 5:45 pm on April 2, 2018: member

    #11817 is merged. Next up: #11818
13. MarcoFalke referenced this in commit 9a2db3b3d5 on Apr 5, 2018
14. 
    fanquake commented at 0:50 am on April 5, 2018: member

    Closing now that #11818 is merged.
15. fanquake closed this on Apr 5, 2018

    ---

16. 
    jnewbery commented at 2:26 pm on April 5, 2018: member

    You’re too fast @fanquake - I was looking forward to closing this myself :cry:
17. jnewbery reopened this on Apr 5, 2018

    ---

18. jnewbery closed this on Apr 5, 2018

    ---

19. 
    jnewbery commented at 2:26 pm on April 5, 2018: member

    That’s better
20. 
    MarcoFalke commented at 2:29 pm on April 5, 2018: member

    :ok_hand:
21. codablock referenced this in commit e3a7aeab7b on Sep 30, 2019
22. codablock referenced this in commit 21040f773f on Sep 30, 2019
23. codablock referenced this in commit 22b9bfea33 on Sep 30, 2019
24. codablock referenced this in commit 8b88ad62b5 on Sep 30, 2019
25. codablock referenced this in commit 33c77c4039 on Sep 30, 2019
26. codablock referenced this in commit c32b271410 on Sep 30, 2019
27. codablock referenced this in commit 9895e2b20e on Oct 2, 2019
28. codablock referenced this in commit e720d73770 on Oct 2, 2019
29. codablock referenced this in commit 79d6ae2922 on Oct 2, 2019
30. codablock referenced this in commit dc335cdee3 on Oct 2, 2019
31. codablock referenced this in commit d40f2e5825 on Oct 2, 2019
32. codablock referenced this in commit 4bf88d56ce on Oct 2, 2019
33. UdjinM6 referenced this in commit 6902ac1f2b on Oct 3, 2019
34. UdjinM6 referenced this in commit beed5cf1f2 on Oct 3, 2019
35. UdjinM6 referenced this in commit 400674c412 on Oct 3, 2019
36. codablock referenced this in commit 01c138e86b on Oct 3, 2019
37. codablock referenced this in commit 3e72a09c16 on Oct 3, 2019
38. codablock referenced this in commit f7d1974e3d on Oct 3, 2019
39. PastaPastaPasta referenced this in commit 128ed7f408 on Dec 27, 2019
40. PastaPastaPasta referenced this in commit 4227b6d585 on Jan 2, 2020
41. PastaPastaPasta referenced this in commit 12fda46dfd on Jan 4, 2020
42. PastaPastaPasta referenced this in commit ce5b846a0f on Jan 12, 2020
43. PastaPastaPasta referenced this in commit 107a86efe7 on Jan 12, 2020
44. PastaPastaPasta referenced this in commit 2f4948fc71 on Jan 12, 2020
45. PastaPastaPasta referenced this in commit 2d6af79199 on Jan 12, 2020
46. PastaPastaPasta referenced this in commit 1fea156fa6 on Jan 12, 2020
47. PastaPastaPasta referenced this in commit e2185c9914 on Jan 12, 2020
48. barrystyle referenced this in commit 3598c04df5 on Jan 22, 2020
49. barrystyle referenced this in commit 2b7f14e055 on Jan 22, 2020
50. MarcoFalke locked this on Sep 8, 2021

Contributors
jnewbery MarcoFalke TheBlueMatt fanquake

Labels
Tests