Run all CI unit tests with DEBUG_LOG_OUT #28466

1. 
   vasild commented at 7:49 am on September 13, 2023: contributor

   Motivation

   Some unit tests use a deterministic random number generator where all generated numbers are deterministically derived from a given starting number (seed) which is generated “randomly”. Those tests can detect bugs and fail accordingly based on the random numbers produced by the generator. It is very useful if it is possible to reproduce a test failure.

   Thus it would be very useful to print the seed, so that if a test fails, that failure can later be reproduced by providing the same seed via the RANDOM_CTX_SEED environment variable.

   It appears that in the CI multiprocess, i686, DEBUG the seed is printed (it uses DEBUG_LOG_OUT), but is not printed for previous releases, qt5 dev package and depends packages, DEBUG (it does not use DEBUG_LOG_OUT).

   Possible solution

   Use DEBUG_LOG_OUT also in previous releases, qt5 dev package and depends packages, DEBUG.

   See #28196 (review)

   Useful Skills

   • Compiling Bitcoin Core from source
   • Running the C++ unit tests

   Guidance for new contributors

   Want to work on this issue?

   For guidance on contributing, please read CONTRIBUTING.md before opening your pull request.

2. vasild added the label good first issue on Sep 13, 2023
3. 
   pgalyen1987 commented at 10:08 pm on September 23, 2023: none

   Hello, I’d like to work on this if it is available.
4. 
   www222fff commented at 9:13 am on October 18, 2023: none

   If not done, let me check this.
5. 
   vasild commented at 9:44 am on October 18, 2023: contributor

   It is not. I guess somebody will post here if there is a PR for it. @pgalyen1987?
6. 
   www222fff commented at 7:12 am on October 19, 2023: none

   @vasild Could you share more detail of the request issue? If possible one case example would be better.
7. 
   vasild commented at 1:00 pm on October 19, 2023: contributor

   The CI job multiprocess, i686, DEBUG is executing test_bitcoin by appending -- DEBUG_LOG_OUT at the end of the command line. Because this would grow the log file too much, the output is redirected to a file which is displayed only if there is a failure:

   https://github.com/bitcoin/bitcoin/blob/9e616baec0f2d94c5b9970cba657a3c79787ef84/src/Makefile.test.include#L427

   The CI job previous releases, qt5 dev package and depends packages, DEBUG executes test_bitcoin in another way:

   https://github.com/bitcoin/bitcoin/blob/9e616baec0f2d94c5b9970cba657a3c79787ef84/ci/test/06_script_b.sh#L170

   without using DEBUG_LOG_OUT.

8. 
   www222fff commented at 7:40 am on October 23, 2023: none

   Understand, more questions what is “previous releases” indicating here? and how to simulate to trigger CI job for previous releases to verify UT result, locally i only have setup for development?
9. 
   www222fff commented at 5:56 am on October 24, 2023: none

   DEBUG_LOG_OUT has difference with using –log_sink? log_sink Specifies log sink: stdout (default), stderr or file name. –log_sink=<stderr|stdout|file name> -k <stderr|stdout|file name>
10. vasild referenced this in commit 895e6463f2 on Oct 26, 2023
11. vasild referenced this in commit fd4ba89575 on Oct 26, 2023
12. 
    vasild commented at 12:49 pm on October 26, 2023: contributor

    This should resolve it: #28736, would be nice if you can review that.

    It turned out that there is a small complication that if we just run the entire test_bitcoin with DEBUG_LOG_OUT, then the log is about 80MB which is too much for CI logs and it is necessary to only selectively print the relevant parts of it to avoid flooding the CI.

13. vasild referenced this in commit 67f3a76a0a on Oct 27, 2023
14. vasild referenced this in commit 1bc0914143 on Oct 27, 2023
15. vasild referenced this in commit aa66ab87f6 on Oct 30, 2023
16. vasild referenced this in commit 58f6074841 on Oct 30, 2023
17. vasild referenced this in commit 62b799aff3 on Oct 31, 2023
18. vasild referenced this in commit e3ade1fbc1 on Oct 31, 2023
19. vasild referenced this in commit 2b7888ec3e on Dec 1, 2023
20. vasild referenced this in commit 57e8bc6b31 on Jan 16, 2024
21. 
    achow101 commented at 3:23 pm on April 9, 2024: member

    The feature request didn’t seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.

    Closing due to lack of interest. Pull requests with improvements are always welcome.

22. achow101 closed this on Apr 9, 2024

    ---

Contributors
vasild pgalyen1987 www222fff achow101

Labels
good first issue