`./bitcoind': munmap_chunk(): invalid pointer after few minutes of running bitcoind #9047

Describe the issue

Can you reliably reproduce the issue?

Cannot reliably reproduce the Issue. Happened randomly twice. On running bitcoind the third time, it's been running up and fine for more than 1 hour without the issue recurring.

I ran Bitcoind using the following parameters:

pi@xxxx~/bin/bitcoin/bin $ ./bitcoind -datadir=/home/pi/.bitcoin/Bitcoin -onion=127.0.0.1:9150 *** Error in ./bitcoind': munmap_chunk(): invalid pointer: 0x6664a220 *** Aborted pi@xxxx:~/bin/bitcoin/bin $ ./bitcoind -datadir=/home/pi/.bitcoin/Bitcoin -onion=127.0.0.1:9150 -maxconnections=11 -dbcache=50 *** Error in ./bitcoind': munmap_chunk(): invalid pointer: 0x6acb17d8 *** Aborted

On running with this parameter the third time it seems to be running fine: ./bitcoind -datadir=/home/pi/.bitcoin/Bitcoin -onion=127.0.0.1:9150 -maxconnections=11

Expected behaviour

Bitcoind starts up perfectly, has 10 connections (8 Outbound and 2 Inbound). The node runs over Tor onion Socks 5 proxy. Should continue to sync and verify blocks as usual.

Actual behaviour

Bitcoind starts up perfectly, has 10 connections (8 Outbound and 2 Inbound). The node runs over Tor onion Socks 5 proxy. After 15 minutes of running shows the following error:

*** Error in `./bitcoind': munmap_chunk(): invalid pointer: 0x6664a220 *** Aborted

and Bitcoind crashes.

What version of bitcoin-core are you using?

Official Binary - Bitcoin 0.13.1 ARM linux 32 bit.

Machine specs:

• OS: Rasbpian OS Jessie
• CPU: 900 MHz quad-core ARM Cortex A7 (ARMv7)
• RAM: 1GB Ram, no SWAP
• Disk size: 400 GB external Hard disk (Has the 100% synchronized blockchain in it)
• Disk Type (HD/SDD): HD

Any extra information that might be useful in the debugging process.

You reported on reddit that a self-compiled v0.13.0 doesn't have this problem - but can you try the official v0.13.0 ARM binary just to see if it does?

Hi Luke, I downloaded the 0.13.0 ARM 32 bit version and ran ./bitcoind on it. Took quite long to verify the blocks but after that it's been running okay for about 45 minutes. Do you want me to continue for longer?

Might as well. There weren't any major bugs fixed in 0.13.1 IIRC.

It would be nice if you could run whatever version is crashing from within GDB with debug symbols (not sure if you have to build yourself to get this or if the shipping binaries have sufficient debug symbols) so that we could get a backtrace.

I cannot replicate the problem on my Raspberry Pis v2B or the Pi3. I have tested 0.13.1 in command line and QT. Both run well. Sync times are slow after block 200,000 as you would expect on the ARM but it seems very stable.

ETA I compiled mine using this bash script https://github.com/MrChrisJ/fullnode/blob/master/Scripts/Upgrading_to_Bitcoin_0.13.1.sh

After looking through the debug.log that @DaReaper provided, I notice a pattern that the node shuts down successfully when it advertises with "AdvertiseLocal: advertising address 202.62.91.54:8333".

The node is not gracefully shutdown 5 out of 6 times when it advertises with: "AdvertiseLocal: advertising address e3amozdbgvuh424k.onion:8333". One of the 5 failures stands out as being maybe unrelated because it fails differently and the cache settings are way different, so I'm going to ignore that. The other 4 fail with the similar log entries in the last second noted below:

1) 2016-10-31 09:05:15 AdvertiseLocal: advertising address e3amozdbgvuh424k.onion:8333
2) 2016-10-31 10:12:06 AdvertiseLocal: advertising address e3amozdbgvuh424k.onion:8333
3) 2016-10-31 13:15:57 AdvertiseLocal: advertising address e3amozdbgvuh424k.onion:8333
   2016-10-31 13:15:57 socket send error Bad file descriptor (9)
4) 2016-10-31 13:42:04 AdvertiseLocal: advertising address e3amozdbgvuh424k.onion:8333

So, maybe some code that advertises local onion addresses has something to do with this?

Simply search for "Bitcoin version v0.13.1" and then look back a few lines to see if the node is shutdown gracefully or not.

If this is the pre-build executable: can you get a backtrace with gdb? Remember to do set disable-randomization to disable ASLR. Then I can (probably) look up the addresses to function/lines.

Closing this, as it is impossible to troubleshoot this using the available information, and others seem not to be able to reproduce it.