bitcoind v23.0.0: traps: b-scheduler[3227] trap invalid opcode

github12101 commented at 2:00 am on December 14, 2022: none

Issue: bitcoind process ended abruptly

Expected behaviour: bitcoind should continue to run

log output: nothing unusual terminal output: 1879 Illegal instruction (core dumped) dmesg output: traps: b-scheduler[3227] trap invalid opcode ip:555ad42d5cb2 sp:7efc897f9590 error:0 in bitcoind[555ad3c91000+9b4000]

Cannot reproduce easily, this happened first time in last 4 years running bitcoind non-stop.

$ bitcoin-cli –version Bitcoin Core RPC client version v23.0.0

Official build from the website, running normally from the user folder. No systemd, no service, but in screen session, in bash loop. Bash loop allows it to automatically restart, so I experienced no downtime, bitcoind has started again automatically. Full command line bitcoind was running with is: /home/user/bitcoin/bitcoin-23.0/bin/bitcoind > /home/user/.bitcoin/logs/2022-11-14_02h04m24s.log 2>&1

Server-style machine with ECC RAM: OS: Debian GNU/Linux 11 (bullseye) x86_64 CPU: AMD Ryzen 7 1700 (16) @ 3.000GHz [28.0°C] RAM: 64GB RAM 2666 MT/s Multi-bit ECC HDD: Btrfs raid10 mode, 4x 4TB Hard Drives. No issues with Btrfs or drives, all perfectly clean

No GUI, bitcoind running in screen session, logging to a file

I could upload entire session debug.log (2.5MiB) on request. bitcoind process was running from 11 November to 13 December then crashed. Log file ends cleanly with no error messages, last few lines are:

02022-12-13T17:50:00Z Socks5() connect to xxx.onion:8333 failed: host unreachable
12022-12-13T17:50:44Z connect() to 127.0.0.1:7656 failed after wait: Connection refused (111)
22022-12-13T17:53:14Z connect() to 127.0.0.1:7656 failed after wait: Connection refused (111)
32022-12-13T17:57:22Z UpdateTip: new best=000000000000000000031509b87e82e97a74d09c703ed21ee0a91fd90b5c6656 height=767249 version=0x20a00000 log2_work=93.893910 tx=787817132 date='2022-12-13T17:57:18Z' progress=1.000000 cache=5.6MiB(23274txo)
42022-12-13T17:59:16Z connect() to 127.0.0.1:7656 failed after wait: Connection refused (111)
52022-12-13T17:59:34Z UpdateTip: new best=000000000000000000044c776b362f68b0e645a1d7366e3b50a10a7809af50fc height=767250 version=0x23778000 log2_work=93.893921 tx=787819399 date='2022-12-13T17:59:22Z' progress=1.000000 cache=6.6MiB(31759txo)

github12101 commented at 2:05 am on December 14, 2022: none

bitcoin.conf:

 0testnet=0
 1server=1
 2rpcuser=x
 3rpcpassword=x
 4txindex=1
 5addresstype=p2sh-segwit
 6deprecatedrpc=signrawtransaction
 7blockfilterindex=1
 8peerblockfilters=1
 9peerbloomfilters=1
10logips=1
11dbcache=16384
12rpcworkqueue=64
13maxorphantx=1000
14maxmempool=1024
15par=16
16rpcthreads=16
17disablewallet=0
18proxy=127.0.0.1:9050
19listen=1
20bind=127.0.0.1
21torpassword=x
22onlynet=onion
23onlynet=i2p
24i2psam=127.0.0.1:7656
25i2pacceptincoming=1

maflcko added the label Bug on Dec 14, 2022

willcl-ark commented at 2:25 pm on December 15, 2022: contributor

Thanks for the report @github12101

If you didn’t say you had ECC RAM I would be inclined to think this was a cosmic ray bitflip or similar type of incident. Unfortunately I don’t think it’ll be possible for me to reproduce this, even with all the (detailed!) info you’ve provided, and I would be surprised if anyone can learn anything from that dmesg line.

On that basis I don’t know whether it’s worth keeping this issue open at all, as I don’t see how it can be addressed. If it is closed then will still be searchable via the repo in the case that we get a second report of it.

Sorry not to be more help.

maflcko added the label Data corruption on Dec 15, 2022

github12101 commented at 12:33 pm on December 16, 2022: none

Hi Marco, Totally understand. I will continue to run bitcoind as always, if I ever encounter this bug again, I will not hesitate to report. In the meantime, you can close the bug if necessary. You can also tell me if there are any steps I can take to debug bitcoind more thoroughly than the default way, I can install debugger or whatever is necessary, as long as it let run bitcoind normally so I can use it as usual.

Kind regards

maflcko commented at 2:14 pm on January 6, 2023: member

Not sure what to do here. Closing for now, but feel free to continue the discussion. This can be re-opened or filed again if new details are available.

maflcko closed this on Jan 6, 2023

bitcoin locked this on Jan 6, 2024

bitcoind v23.0.0: traps: b-scheduler[3227] trap invalid opcode #26696