Intermittent issue in test/ipc_tests.cpp Fatal glibc error: pthread_mutex_lock.c:450 (__pthread_mutex_lock_full): assertion failed: e != ESRCH || !robust

maflcko commented at 9:29 am on April 16, 2024: member

Happened on current master d29fc3a245c070494155dad4cf68b9c95d99c13e in ci_i686_multiprocess

 0Running tests: wallet_util_tests from wallet/test/rpc_util_tests.cpp
 1Running tests: scriptpubkeyman_tests from wallet/test/scriptpubkeyman_tests.cpp
 2Running tests: walletload_tests from wallet/test/walletload_tests.cpp
 3Running tests: group_outputs_tests from wallet/test/group_outputs_tests.cpp
 4Running tests: db_tests from wallet/test/db_tests.cpp
 5Running tests: ipc_tests from test/ipc_tests.cpp
 6/bin/bash: line 2: 28702 Aborted                 (core dumped) test/test_bitcoin --catch_system_errors=no -l test_suite -t "$(cat test/ipc_tests.cpp | grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1)" -- DEBUG_LOG_OUT > "$TEST_LOGFILE" 2>&1
 7Running 1 test case...
 8Test cases order is shuffled using seed: 227983234
 9Entering test module "Bitcoin Core Test Suite"
10wallet/test/feebumper_tests.cpp(18): Test suite "feebumper_tests" is skipped because disabled
11test/validation_flush_tests.cpp(13): Test suite "validation_flush_tests" is skipped because disabled
12test/raii_event_tests.cpp(16): Test suite "raii_event_tests" is skipped because disabled
13test/denialofservice_tests.cpp(36): Test suite "denialofservice_tests" is skipped because disabled
14test/i2p_tests.cpp(45): Test suite "i2p_tests" is skipped because disabled
15test/interfaces_tests.cpp(16): Test suite "interfaces_tests" is skipped because disabled
16test/bloom_tests.cpp(25): Test suite "bloom_tests" is skipped because disabled
17test/txvalidation_tests.cpp(20): Test suite "txvalidation_tests" is skipped because disabled
18test/orphanage_tests.cpp(19): Test suite "orphanage_tests" is skipped because disabled
19test/blockencodings_tests.cpp(19): Test suite "blockencodings_tests" is skipped because disabled
20test/pmt_tests.cpp(28): Test suite "pmt_tests" is skipped because disabled
21test/streams_tests.cpp(15): Test suite "streams_tests" is skipped because disabled
22test/skiplist_tests.cpp(15): Test suite "skiplist_tests" is skipped because disabled
23test/coinstatsindex_tests.cpp(16): Test suite "coinstatsindex_tests" is skipped because disabled
24test/minisketch_tests.cpp(17): Test suite "minisketch_tests" is skipped because disabled
25test/allocator_tests.cpp(16): Test suite "allocator_tests" is skipped because disabled
26test/mempool_tests.cpp(16): Test suite "mempool_tests" is skipped because disabled
27test/coins_tests.cpp(100): Test suite "coins_tests" is skipped because disabled
28test/hash_tests.cpp(14): Test suite "hash_tests" is skipped because disabled
29test/uint256_tests.cpp(17): Test suite "uint256_tests" is skipped because disabled
30wallet/test/coinselector_tests.cpp(24): Test suite "coinselector_tests" is skipped because disabled
31test/blockfilter_index_tests.cpp(25): Test suite "blockfilter_index_tests" is skipped because disabled
32test/validation_chainstate_tests.cpp(21): Test suite "validation_chainstate_tests" is skipped because disabled
33test/disconnected_transactions.cpp(10): Test suite "disconnected_transactions" is skipped because disabled
34test/sync_tests.cpp(80): Test suite "sync_tests" is skipped because disabled
35test/getarg_tests.cpp(19): Test suite "getarg_tests" is skipped because disabled
36test/logging_tests.cpp(20): Test suite "logging_tests" is skipped because disabled
37test/blockmanager_tests.cpp(25): Test suite "blockmanager_tests" is skipped because disabled
38test/ipc_tests.cpp(8): Entering test suite "ipc_tests"
39test/ipc_tests.cpp(9): Entering test case "ipc_tests"
40Fatal glibc error: pthread_mutex_lock.c:450 (__pthread_mutex_lock_full): assertion failed: e != ESRCH || !robust
41make[3]: *** [Makefile:22535: test/ipc_tests.cpp.test] Error 1
42make[3]: *** Waiting for unfinished jobs....
43make[3]: Leaving directory '/ci_container_base/ci/scratch/build/bitcoin-i686-pc-linux-gnu/src'
44make[2]: *** [Makefile:20499: check-am] Error 2
45make[2]: Leaving directory '/ci_container_base/ci/scratch/build/bitcoin-i686-pc-linux-gnu/src'
46make[1]: *** [Makefile:20164: check-recursive] Error 1
47make[1]: Leaving directory '/ci_container_base/ci/scratch/build/bitcoin-i686-pc-linux-gnu/src'
48make: *** [Makefile:755: check-recursive] Error 1

maflcko added the label Tests on Apr 16, 2024

maflcko added the label CI failed on Apr 16, 2024

maflcko commented at 9:31 am on April 16, 2024: member

0# podman exec ci_i686_multiprocess  uname -a
1Linux a943c4649ecd 5.14.21-150400.24.100-default [#1](/bitcoin-bitcoin/1/) SMP PREEMPT_DYNAMIC Mon Dec 4 19:12:13 UTC 2023 (3f5cd84) x86_64 x86_64 x86_64 GNU/Linux

ryanofsky commented at 4:42 pm on April 16, 2024: contributor

I wonder if there’s any way to get a stack trace. Seeing log output could also be helpful as the test should be logging everything with LogPrintf.

maflcko commented at 4:46 pm on April 16, 2024: member

I threw away the pod, so the logs are gone, but I wonder why they were not printed to the screen.

Otherwise, if this issue happens on Cirrus or GHA, the logs won’t be available either.

maflcko commented at 6:59 pm on April 16, 2024: member

Actually, I think the logs are complete for the log level test_suite. If more details are needed test_suite could be changed to all.

maflcko commented at 12:45 pm on April 17, 2024: member

stack trace

#29900

maflcko commented at 3:43 pm on April 17, 2024: member

I guess it could make more sense if someone tried to reproduce this in rr.

Intermittent issue in test/ipc_tests.cpp Fatal glibc error: pthread_mutex_lock.c:450 (__pthread_mutex_lock_full): assertion failed: e != ESRCH || !robust #29889