Replacing Boost Test Framework #8670

1. 
   JeremyRubin commented at 8:37 pm on September 6, 2016: contributor

   The unit tests are pretty slow (especially under wine) which can cause timeouts in travis.

   Furthermore, it’s another boost dependency we can probably get rid of.

   This issue is to discuss what features people would like in a new test framework.

   To start a list of features that people might like:

   • printing the expression, value, and line number on failure
   • checkpointing in case of segfaults
   • regex filter tests
   • progress meter/output currently running test
   • test benchmarking info/ability to regression test performance

   Please chime in with anything else that you think should go into it.

2. fanquake added the label Tests on Sep 7, 2016
3. 
   Christewart commented at 8:50 pm on September 7, 2016: contributor

   It would be nice to have some sort of support for property based testing, as per #8469. Conveniently the framework I integrate in that pull request supports integration with Boost.
4. 
   JeremyRubin commented at 10:40 pm on September 7, 2016: contributor

   @Christewart we’re trying to get rid of boost.
5. 
   maflcko commented at 10:03 am on September 11, 2016: member

   There is a bunch of BOOST_CHECK()s in the code( where BOOST_CHECK_EQUAL()s should be used instead) causing at least one assignment error (#8673). So would it make sense to have BOOST_CHECK() fail on non-boolean values, if it is possible at all?
6. 
   JeremyRubin commented at 7:11 pm on September 13, 2016: contributor

   @MarcoFalke I have no clue how to do that; short of a compile time regex (http://cpptruths.blogspot.com/2011/07/compile-time-regex-matcher-using.html).
7. 
   JeremyRubin commented at 7:12 pm on September 13, 2016: contributor

   I also like that GDB can attach itself on a failure. I don’t like that it seems to launch a gui gdb needing root to attach on x sessions though.
8. 
   Christewart commented at 12:18 pm on September 14, 2016: contributor

   @JeremyRubin I think it would be worthwhile to building in a property based testing framework into our custom framework if we are hell bent on building a framework from scratch. I’m not sure if I support completely eliminating the Boost dependency, since the overhead of maintaining the new framework might be more trouble than its worth.

   Would it just be more wise to run our own fork of Boost and fix the performance issues you mentioned above? Could those features you mentioned above be layered on top of Boost?

9. 
   sipa commented at 5:06 pm on September 14, 2016: member

   Feature request: a command line option to make the application abort() on the first error, allowing the creation of a core dump and inspect afterwards in gdb.
10. 
    laanwj commented at 8:34 am on September 28, 2016: member

    The unit tests are pretty slow (especially under wine) which can cause timeouts in travis.

    Also on FreeBSD: test_bitcoin takes 237s, whereas on Linux only 40s on older hardware.

    It’s starting to look like boost::test is optimized specifically for Linux.

11. 
    laanwj commented at 9:57 am on September 28, 2016: member

    NM, I had accidentally disabled optimization for the entire build (issue #3921). New time on FreeBSD is the same (within rounding error) as the Linux time.
12. 
    NicolasDorier commented at 8:39 am on February 3, 2017: contributor

    A test I added during segwit https://github.com/bitcoin/bitcoin/blob/dd98f045382428f450dc917a3933c9e4e8e1ba99/src/test/transaction_tests.cpp#L415 is very slow. (I think like 30sec)

    I am not sure if it is a good idea to remove it though.

13. 
    JeremyRubin commented at 12:22 pm on February 14, 2017: contributor

    @NicolasDorier This is (probably) because you’re creating a ton of threads, I think!

    for (int i=0; i<20; i++)

    There’s no reason to do that, try i < 2.

14. 
    NicolasDorier commented at 9:25 am on February 15, 2017: contributor

    @JeremyRubin the test was testing race conditions.
15. 
    NicolasDorier commented at 9:26 am on February 15, 2017: contributor

    The problem is that signing so much input takes time. Not so much the number of spawned thread.
16. 
    JeremyRubin commented at 10:19 am on February 15, 2017: contributor

    Ah ok – maybe use pregenerated test vectors as is done in a couple other places? Or use the threads to generate the signatures (do they need to be signed in serial order)?
17. 
    NicolasDorier commented at 12:15 pm on February 15, 2017: contributor

    I remember having tried to parallelize signature but that it was not worth it, pregenerated test vectors are better indeed.

    Anyway, I think this is a separate problem from this issue. I would like better test framework for sure.

18. bitcoin deleted a comment on Dec 27, 2018
19. 
    BrannonKing commented at 9:50 am on January 8, 2019: none

    I don’t think that there is any reason to change to a different unit test framework as long as we have a single Boost dependency remaining. This would be the very last dependency to go. And if it’s time for Boost Test to go away, I think you should replace it with an already-existing framework like Catch2 rather than write your own. I don’t want to have to learn a new framework to write tests for bitcoin. It feels like a custom test framework is too far outside the direction and responsibility of the bitcoin codebase.
20. 
    maflcko commented at 7:39 pm on October 7, 2019: member

    At the very least, we should look into replacing the BOOST_CHECK (et al.) macros with our own. None of the BOOST_ macros are thread-safe. However, Bitcoin Core runs multiple threads (indexers, validation interface, main thread, …), so we can use those macros in at most one thread. See for example this place https://github.com/bitcoin/bitcoin/pull/13023/files#diff-83ded29db9bfc1c9342adc826ca2b5abR168
21. JeremyRubin closed this on Dec 16, 2022

    ---

22. bitcoin locked this on Dec 16, 2023

Contributors
JeremyRubin Christewart maflcko sipa laanwj NicolasDorier BrannonKing

Labels
Tests