Run the unit tests with DEBUG_LOG_OUT
in the case of RUN_UNIT_TESTS_SEQUENTIAL
.
Because the output would be too big (about 80MB), redirect it to a file and only show relevant bits from it in case of errors.
Run the unit tests with DEBUG_LOG_OUT
in the case of RUN_UNIT_TESTS_SEQUENTIAL
.
Because the output would be too big (about 80MB), redirect it to a file and only show relevant bits from it in case of errors.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
For detailed information about the code coverage, see the test coverage report.
See the guideline for information on the review process.
Type | Reviewers |
---|---|
Concept ACK | jonatack |
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.
No conflicts as of last run.
188+ q
189+EDCOMMANDS
190+ done
191+ exit 1
192+ fi
193+ "
Not sure about 20 lines of bash code that are impossible to read, likely no one will review, and which are hard to maintain or change in the future.
On a second though, I wonder how often this feature will be needed, or if there is a simpler hack that achieves something similar.
Some of this was already there before. The new code is:
0 # find all lines that match 'error: in test_suite/test_case' and extract the test suite and case
1 for t in \$(sed -E -n 's/.*error: in \"(.+\\/.+)\":.*/\\1/p' < ${LOG}) ; do
2 test_suite=\${t%/*} # e.g. net_tests
3 test_case=\${t#*/} # e.g. v2transport_test
4 ed -s ${LOG} <<EDCOMMANDS
5 /Entering test suite \"\${test_suite}\"/
6 /Entering test case \"\${test_case}\"/,/Leaving test case \"\${test_case}\"/p
7 q
8EDCOMMANDS
I do not think that is too different in terms of complexity from what we already have:
0%.cpp.test: %.cpp [@echo](/bitcoin-bitcoin/contributor/echo/) Running tests: $$(\
1 cat $< | \
2 grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | \
3 cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1\
4 ) from $<
5 $(AM_V_at)export TEST_LOGFILE=$(abs_builddir)/$$(\
6 echo $< | grep -E -o "(wallet/test/.*\.cpp|test/.*\.cpp)" | $(SED) -e s/\.cpp/.log/ \
7 ) && \
8 $(TEST_BINARY) --catch_system_errors=no -l test_suite -t "$$(\
9 cat $< | \
10 grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | \
11 cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1\
12 )" -- DEBUG_LOG_OUT > "$$TEST_LOGFILE" 2>&1 || (cat "$$TEST_LOGFILE" && false)
If you find ed
hard to read then I could change it to another language (python?) but I suspect it is going to be more lines of code. I any case, I don’t think that’s prohibitively complex task: “find ‘Entering test suite’, after that print the lines between ‘Entering test case’ and ‘Leaving test case’”.
Some extra escaping is needed because this is inside bash -c "here;"
. What is the point of doing that, given that this is already a bash script? I guess doing just here;
should have the same effect.
I wonder how often this feature will be needed
This I do not know, but I know that when a test fails on CI that depends on a particular seed, having that seed is a game changer for fixing the bug.
Another option would be to print the seed unconditionally, even if DEBUG_LOG_OUT
is not defined. But having the debug log in case of failure is useful beyond the rng seed.
Another option would be to print the seed unconditionally
Sure, why not?
To extend my previous feedback, on why this bash code isn’t ideal:
set -ex
in the script? Is the error code properly propagated in all cases? We’ve had tests in the past that silently passed regardless of the result, so it would be good to not accidentally re-introduce that. Doing output parsing in bash makes it easy to discard the error code, for example when adding a pipe but no pipefail.I do not think that is too different in terms of complexity from what we already have:
Pretty sure this will go away with cmake, no?
Another option would be to print the seed unconditionally
Sure, why not?
Yeah, this will solve the original issue with the seed. But I think this PR has a wider benefit of seeing the debug log for any type of failure, like in the RUN_UNIT_TESTS
case.
To extend my previous feedback, on why this bash code isn’t ideal:
Thanks, that makes it easier to understand and address.
- It assumes a specific unit test output format, so if boost is removed or if the format or wording changes, the code will silently stop to work
True, but that will not remain unnoticed (as long as somebody cares to look at the log after a failure) and should be easy to adjust. This can use test_bitcoin --log_format=XML or JUNIT
. Even with that the point is still valid - if the format is changed then this would have to be adjusted. I think the benefit of seeing the debug log outweights the need to possibly have to adjust this in the future.
- Putting the bash code into a string and then executing the string makes it impossible to check with shellcheck. And also manual review is hard, because everything is escaped.
I agree, removed the bash -c "..."
surrounding. Why is that used all over the place in 06_script_b.sh
?
- How does it interact with
set -ex
in the script? Is the error code properly propagated in all cases? We’ve had tests in the past that silently passed regardless of the result, so it would be good to not accidentally re-introduce that. Doing output parsing in bash makes it easy to discard the error code, for example when adding a pipe but no pipefail.
It works correctly - if ! command that fails ; then foo ; exit 1; fi
the failing command will not trigger the exit due to set -e
and will execute all commands inside the if
body. I put exit 1
at the end to terminate the script because being inside the if
means the command failed.
I do not think that is too different in terms of complexity from what we already have:
Pretty sure this will go away with cmake, no?
Good point. And actually my argument was flawed - already having some “bad” code is not enough justification, alone, to add more of the same.
I agree, removed the
bash -c "..."
surrounding. Why is that used all over the place in06_script_b.sh
?
It is a leftover from the travis integration that started to use Docker initially, I think.
Edit: Yes, see 59e9688eda4c4b01ee1713625632cd766c1a7ca9
I was thinking about getting rid of it. Maybe even completely rewrite the whole script in a sane language, instead of “rewriting” half of it from bash to bash.
fd4ba89575...67f3a76a0a
: improve readability
67f3a76a0a...1bc0914143
: add quotes around ${variables}
to pet the linter
CI:
0+ '[' -n '' ']'
1+ '[' false = true ']'
2+ '[' true = true ']'
3+ LOG=/ci_container_base/ci/scratch/build/test_bitcoin.log
4+ DIR_UNIT_TEST_DATA=/ci_container_base/ci/scratch/qa-assets/unit_test_data/ LD_LIBRARY_PATH=/ci_container_base/depends/x86_64-pc-linux-gnu/lib /ci_container_base/ci/scratch/out/bin/test_bitcoin --catch_system_errors=no -l test_suite -- DEBUG_LOG_OUT
5++ sed -E -n 's|.*error: in "(.+)/(.+)":.*|\1 \2|p'
6+ FAILED_TESTS=
7+ printf 'Error: the following tests failed from test_bitcoin:\n%s\n' ''
8Error: the following tests failed from test_bitcoin:
9
10+ read test_suite test_case
11+ ed -s /ci_container_base/ci/scratch/build/test_bitcoin.log
12/ci_container_base/ci/test/06_script_b.sh: line 185: ed: command not found
13
14Exit status: 127����������������
previous releases, qt5 dev package and depends packages, DEBUG
does not run anymore in my personal github repository (due to “No public persistent worker pools found!”). I will push a few times here, sorry for the noise. Converted to Draft.
I deliberately broke some tests to see how this will report the failures. Looks good:
0+ LOG=/ci_container_base/ci/scratch/build/test_bitcoin.log
1+ DIR_UNIT_TEST_DATA=/ci_container_base/ci/scratch/qa-assets/unit_test_data/
2+ LD_LIBRARY_PATH=/ci_container_base/depends/x86_64-pc-linux-gnu/lib
3+ /ci_container_base/ci/scratch/out/bin/test_bitcoin --catch_system_errors=no --color_output=false --log_level=test_suite -- DEBUG_LOG_OUT
4++ sed -E -n 's|.*error: in "(.+)/(.+)":.*|\1 \2|p'
5++ sort -u
6+ FAILED_TESTS='arith_uint256_tests basics
7crypto_tests hmac_sha256_testvectors
8crypto_tests ripemd160_testvectors
9uint256_tests basics'
10+ printf 'Error: the following tests failed from test_bitcoin:\n%s\n' 'arith_uint256_tests basics
11crypto_tests hmac_sha256_testvectors
12crypto_tests ripemd160_testvectors
13uint256_tests basics'
14Error: the following tests failed from test_bitcoin:
15arith_uint256_tests basics
16crypto_tests hmac_sha256_testvectors
17crypto_tests ripemd160_testvectors
18uint256_tests basics
19+ read test_suite test_case
20+ ed -s /ci_container_base/ci/scratch/build/test_bitcoin.log
21test/arith_uint256_tests.cpp(18): Entering test suite "arith_uint256_tests"
22test/arith_uint256_tests.cpp(68): Entering test case "basics"
23test/arith_uint256_tests.cpp(70): error: in "arith_uint256_tests/basics": check false has failed
24test/arith_uint256_tests.cpp(68): Leaving test case "basics"; testing time: 1726us
25+ read test_suite test_case
26+ ed -s /ci_container_base/ci/scratch/build/test_bitcoin.log
27test/crypto_tests.cpp(28): Entering test suite "crypto_tests"
28test/crypto_tests.cpp(459): Entering test case "hmac_sha256_testvectors"
292023-10-31T09:08:26.365840Z [test] [test/util/random.cpp:31] [Seed] Seed: Setting random seed for current tests to RANDOM_CTX_SEED=c5cfe029b57c3d3e34ab62c0a41c092b5bdbf67486eab8f3ec47f86ef0ca536e
302023-10-31T09:08:26.365877Z [test] [init/common.cpp:153] [LogPackageVersion] Bitcoin Core version v26.99.0-86ddf78fc463-dirty (debug build)
312023-10-31T09:08:26.366041Z [test] [kernel/context.cpp:24] [Context] Using the 'x86_shani(1way,2way)' SHA256 implementation
322023-10-31T09:08:26.366052Z [test] [random.cpp:98] [ReportHardwareRand] Using RdSeed as an additional entropy source
332023-10-31T09:08:26.366058Z [test] [random.cpp:101] [ReportHardwareRand] Using RdRand as an additional entropy source
342023-10-31T09:08:26.366695Z [test] [script/sigcache.cpp:103] [InitSignatureCache] Using 16 MiB out of 16 MiB requested for signature cache, able to store 524288 elements
352023-10-31T09:08:26.367058Z [test] [validation.cpp:1831] [InitScriptExecutionCache] Using 16 MiB out of 16 MiB requested for script execution cache, able to store 524288 elements
36test/crypto_tests.cpp(38): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
37test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
38test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
39test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
40test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
41test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
42test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
43test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
44test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
45test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
46test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
47test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
48test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
49test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
50test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
51test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
52test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
53test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
54test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
55test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
56test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
57test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
58test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
59test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
60test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
61test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
62test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
63test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
64test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
65test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
66test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
67test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
68test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
69test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
70test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
71test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
72test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
73test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
74test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
75test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
76test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
77test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
78test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
79test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
80test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
81test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
82test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
83test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
84test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
85test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
86test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
87test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
88test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
89test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
90test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
91test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
92test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
93test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
94test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
95test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
96test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
97test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
98test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
99test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
100test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
101test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
102test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
103test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
104test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
105test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
106test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
107test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
108test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
109test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
110test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
111test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
112test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
113test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
114test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
115test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
116test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
117test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
118test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
119test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
120test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
121test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
122test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
123test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
124test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
125test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
126test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
127test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
128test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
129test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
130test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
131test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
132test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
133test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
134test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
135test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
136test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
137test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
138test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
139test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
140test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
141test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
142test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
143test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
144test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
145test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
146test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
147test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
148test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
149test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
150test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
151test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
152test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
153test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
154test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
155test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
156test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
157test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
158test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
159test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
160test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
161test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
162test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
163test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
164test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
165test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
166test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
167test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
168test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
169test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
170test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
171test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
172test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
173test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
174test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
175test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
176test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
177test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
178test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
179test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
180test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
181test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
182test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
183test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
184test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
185test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
186test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
187test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
188test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
189test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
190test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
191test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
192test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
193test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
194test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
195test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
196test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
197test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
198test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
199test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
200test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
201test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
202test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
203test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
204test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
205test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
206test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
207test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
208test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
209test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
210test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
211test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
212test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
213test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
214test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
215test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
216test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
217test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
218test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
219test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
220test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
221test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
222test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
223test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
224test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
225test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
226test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
227test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
228test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
229test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
230test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
231test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
232test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
233test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
234test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
235test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
236test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
237test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
238test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
239test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
240test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
241test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
242test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
243test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
244test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
245test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
246test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
247test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
248test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
249test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
250test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
251test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
252test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
253test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
254test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
255test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
256test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
257test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
258test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
259test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
260test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
261test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
262test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
263test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
264test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
265test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
266test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
267test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
268test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
269test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
270test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
271test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
272test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
273test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
274test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
275test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
276test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
277test/crypto_tests.cpp(51): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
278test/crypto_tests.cpp(55): error: in "crypto_tests/hmac_sha256_testvectors": check hash == out has failed
279test/crypto_tests.cpp(459): Leaving test case "hmac_sha256_testvectors"; testing time: 8277us
280+ read test_suite test_case
281+ ed -s /ci_container_base/ci/scratch/build/test_bitcoin.log
282test/crypto_tests.cpp(28): Entering test suite "crypto_tests"
283test/crypto_tests.cpp(368): Entering test case "ripemd160_testvectors"
2842023-10-31T09:08:26.191363Z [test] [test/util/random.cpp:31] [Seed] Seed: Setting random seed for current tests to RANDOM_CTX_SEED=c5cfe029b57c3d3e34ab62c0a41c092b5bdbf67486eab8f3ec47f86ef0ca536e
2852023-10-31T09:08:26.191426Z [test] [init/common.cpp:153] [LogPackageVersion] Bitcoin Core version v26.99.0-86ddf78fc463-dirty (debug build)
2862023-10-31T09:08:26.191610Z [test] [kernel/context.cpp:24] [Context] Using the 'x86_shani(1way,2way)' SHA256 implementation
2872023-10-31T09:08:26.191626Z [test] [random.cpp:98] [ReportHardwareRand] Using RdSeed as an additional entropy source
2882023-10-31T09:08:26.191632Z [test] [random.cpp:101] [ReportHardwareRand] Using RdRand as an additional entropy source
2892023-10-31T09:08:26.192320Z [test] [script/sigcache.cpp:103] [InitSignatureCache] Using 16 MiB out of 16 MiB requested for signature cache, able to store 524288 elements
2902023-10-31T09:08:26.192683Z [test] [validation.cpp:1831] [InitScriptExecutionCache] Using 16 MiB out of 16 MiB requested for script execution cache, able to store 524288 elements
291test/crypto_tests.cpp(38): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
292test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
293test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
294test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
295test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
296test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
297test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
298test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
299test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
300test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
301test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
302test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
303test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
304test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
305test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
306test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
307test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
308test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
309test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
310test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
311test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
312test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
313test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
314test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
315test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
316test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
317test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
318test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
319test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
320test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
321test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
322test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
323test/crypto_tests.cpp(55): error: in "crypto_tests/ripemd160_testvectors": check hash == out has failed
324test/crypto_tests.cpp(368): Leaving test case "ripemd160_testvectors"; testing time: 164979us
325+ read test_suite test_case
326+ ed -s /ci_container_base/ci/scratch/build/test_bitcoin.log
327test/uint256_tests.cpp(18): Entering test suite "uint256_tests"
328test/uint256_tests.cpp(75): Entering test case "basics"
329test/uint256_tests.cpp(125): error: in "uint256_tests/basics": check 0 has failed
330test/uint256_tests.cpp(75): Leaving test case "basics"; testing time: 207us
331+ read test_suite test_case
332+ exit 1
Concept ACK.
(Maybe unrelated, but the Win64 CI task is one I’ve often wished for more debug info from when a unit test fails.)
172+ # it to a file and only if errors occur, then extract the relevant snippets
173+ # from it - between "Entering test case" and "Leaving test case".
174+ LOG="${BASE_BUILD_DIR}/test_bitcoin.log"
175+ if ! DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" \
176+ LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" \
177+ ${TEST_RUNNER_ENV} \
export TEST_RUNNER_ENV="ENV_VAR_FOO_BAR=foo_bar"
and observe a failure?
Easiest is to drop it after #28954 which removes TEST_RUNNER_ENV
altogether.
A type-safe language might be a better fit for this, but it is already written in bash and I chose the path of least resistance. IMO the changes in their current form (after #28954 is merged) are an improvement over the current situation. The fact that there might be even better improvement does not make them less valuable.
I wouldn’t go at rewriting this in another language. If you do, I can review.
8@@ -9,7 +9,7 @@ export LC_ALL=C.UTF-8
9 export CONTAINER_NAME=ci_native_qt5
10 export CI_IMAGE_NAME_TAG="docker.io/debian:bullseye"
11 # Use minimum supported python3.9 and gcc-10, see doc/dependencies.md
12-export PACKAGES="gcc-10 g++-10 python3-zmq qtbase5-dev qttools5-dev-tools libdbus-1-dev libharfbuzz-dev"
13+export PACKAGES="gcc-10 g++-10 python3-zmq qtbase5-dev qttools5-dev-tools libdbus-1-dev libharfbuzz-dev ed"
RUN_UNIT_TESTS_SEQUENTIAL=true
, it will fail as well?
the global packages env var
Do you mean this one?
CI_BASE_PACKAGES
in latest push.
e3ade1fbc1...2b7888ec3e
: rebase due to conflicts and add the ed
package to CI_BASE_PACKAGES
(see #28736 (review)).
Resolves https://github.com/bitcoin/bitcoin/issues/28466
2b7888ec3e...57e8bc6b31
: rebase due to conflicts
The PR didn’t seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open PRs.
Closing due to lack of interest.
To add some context. I think it would be nice to print the seed. However, the changes here had to be pushed several times, because the bash code was wrong, and the current version passes CI, but it is unclear if the bash code is correct.
For example, it seems better to fail CI on a test failure and not print the seed, than to print the seed and not fail the CI on a test failure.