Run functional tests from make check #18816

1. 
   maflcko commented at 1:53 pm on April 29, 2020: member

   We have a bunch of tests:

   • unit tests
   • util tests written in python
   • bench runner
   • subtree tests

   All of them are run when you type make check.

   However, for running the functional tests, one has to type ./test/functional/test_runner.py

   I think some people asked for the functional tests to be run as part of make check, or maybe a different target like make check-functional?

2. maflcko added the label Feature on Apr 29, 2020
3. maflcko added the label Brainstorming on Apr 29, 2020
4. maflcko added the label Tests on Apr 29, 2020
5. 
   rodentrabies commented at 3:18 pm on April 29, 2020: contributor

   Perhaps both add a new check-functional target that does ./test/functional/test_runner.py and make it the final step of check target?
6. 
   practicalswift commented at 3:51 pm on April 29, 2020: contributor

   Personally I like that make check is relatively quick thanks to not running the heavy functional tests.

   What about adding another target (say make check-all?) which runs make check + test/functional/test_runner.py? (Edit: Oh, perhaps that was the suggestion with make check-functional. Sorry!)

7. 
   maflcko commented at 4:06 pm on April 29, 2020: member

   The unit tests run in 1 minute and 9 seconds on my system. Mostly because the libsecp tests are not run in parallel. The functional tests run in under 3 minutes.

   An alternative would be to have ./configure --with-functional-tests --without-subtree-tests && make check run Bitcoin Core tests only.

8. 
   brakmic commented at 4:23 pm on April 29, 2020: contributor

   Hi,

   I have played a bit with Makefiles and came with this cheap “solution”. Not sure if idiomatic or proper in any way, but it works.

   • Add new Makefile file to top test directory with this content:

   0check-functional:
   1  $(PYTHON) ../test/functional/test_runner.py
   

   • Change Makefile.am

    0diff --git a/Makefile.am b/Makefile.am
    1index c35f5080a..bcf3b064c 100644
    2--- a/Makefile.am
    3+++ b/Makefile.am
    4@@ -342,3 +342,6 @@ clean-local: clean-docs
    5        rm -rf coverage_percent.txt test_bitcoin.coverage/ total.coverage/ fuzz.coverage/ test/tmp/ cache/ $(OSX_APP)
    6        rm -rf test/functional/__pycache__ test/functional/test_framework/__pycache__ test/cache share/rpcauth/__pycache__
    7        rm -rf osx_volname dist/ dpi36.background.tiff dpi72.background.tiff
    8+
    9+check-functional:
   10+       $(MAKE) -C $(top_builddir)/test
   

   • test run with make check-functional

   

   –EDIT: I should have maybe added option for passing arguments? Like make check-functional p2p_segwit.py

   –EDIT2: my repo containing this solution.

9. 
   hebasto commented at 3:55 am on April 30, 2020: member

   I agree to have a make target to run all of the tests. But I’d preserve a target which runs current set of tests without functional ones.
10. 
    jonatack commented at 11:53 am on April 30, 2020: member

    I use a bash alias to run the unit tests followed by the functional tests. Agree with keeping the current make check default and maybe adding an additional make check -all or similar.
11. adamjonas added the label good first issue on Aug 3, 2022
12. 
    AmitNudel commented at 1:03 pm on September 29, 2022: none

    I’m looking to contribute for the first time to open source. Is it still an open issue? I’d like to try and help.
13. 
    ismaelsadeeq commented at 1:01 pm on March 9, 2023: member

    Hello! I noticed this issue and would like to contribute by enabling running the tests in ./test/functional/test_runner.py with make check-functional. My plan is to also add a make check-all target to run both make check and make check-functional tests. Any suggestions on how to make the solution even better, I’m all ears! Thank you so much for your time and help.
14. 
    maflcko commented at 1:42 pm on March 10, 2023: member

    Given that we will switch from cmake to make, it could make sense to wait for that and then only implement this in cmake?
15. maflcko removed the label good first issue on Mar 10, 2023
16. 
    maflcko commented at 9:25 am on November 18, 2024: member

    I am using a bash alias myself, and given the upvotes, I guess everyone else is doing that, too.

    So I guess this isn’t needed, and not trivially doable anyway: https://github.com/bitcoin/bitcoin/pull/31312/files#r1846159390

17. maflcko closed this on Nov 18, 2024

    ---

Contributors
maflcko rodentrabies practicalswift brakmic hebasto jonatack AmitNudel ismaelsadeeq

Labels
Feature Brainstorming Tests