hard crash when validateaddress RPC endpoint is requested #6963

I'm running 4ee149a compiled on an ARM architecture and using bitcoind to solo mine on testnet with cgminer. However, now that I updated to 4ee149a, bitcoind crashes hard during the first request for a block template from cgminer. I set debug=1 and this is all the output I get before the crash:

 2015-11-06 20:56:31 Received a POST request for / from 192.168.1.14:59243
 2015-11-06 20:56:31 CreateNewBlock(): total size 1000
 2015-11-06 20:56:31       - Connect 1 transactions: 0.11ms (0.111ms/tx, 0.000ms/txin) [0.12s]
 2015-11-06 20:56:31     - Verify 0 txins: 0.71ms (0.000ms/txin) [1.60s]
 2015-11-06 20:56:31 Received a POST request for / from 192.168.1.14:59243
 2015-11-06 20:56:31 Received a POST request for / from 192.168.1.14:59243
 2015-11-06 20:56:31 ThreadRPCServer method=validateaddress

Upon closer review, it's not the block template request that causes the crash, it's the validateaddress call. I have reproduced it via the RPC interface:

 curl --user miner --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "validateaddress", "params": ["mq4oDPjmNWnBxbzx7qouzhpCSTMePUtYDF"] }' -H 'content-type: text/plain;' http://192.168.1.3:18332/

Can you check dmesg to see if you simply ran out of memory?

Yeah, that was the first thing I checked since it's the most common cause. But it's not a memory issue this time; I've reproduced the problem multiple times with my node after it has only been running a couple minutes - memory usage is quite low since it's testnet.

Can you run it in the debugger and reproduce?

Is it possible you've started this bitcoind up with -disablewallet? I think that can trigger this crash on validateaddress.

@sdaftuar You're exactly right: I configured --without-wallet which would have done it. I'll recompile with it enabled and report back if it's still crashing. Either way, it seems like a hard crash is not the behavior we'd want for any RPC call...

Obviously needs fixing!

I recompiled with the wallet enabled and it works fine. Validating an address certainly shouldn't require wallet functionality, and the fact that this RPC call is not disabled when the wallet is disabled leads me to believe that the intention is for it to work in either case.

https://github.com/bitcoin/bitcoin/blob/master/src/rpcserver.cpp#L318

It should work with the wallet disabled. On Nov 6, 2015 4:21 PM, "Jameson Lopp" notifications@github.com wrote:

I recompiled with the wallet enabled and it works fine. Validating an address certainly shouldn't require wallet functionality, and the fact that this RPC call is not disabled when the wallet is disabled leads me to believe that the intention is for it to work in either case.

https://github.com/bitcoin/bitcoin/blob/master/src/rpcserver.cpp#L318

— Reply to this email directly or view it on GitHub #6963 (comment).

Upon further testing, I cannot reproduce this bug while running on mainnet. I'm thinking that this bug is testnet-specific while running with the wallet disabled.

Can't reproduce in the current master compiled with --disable-wallet on the testnet. The answer is (after json_pp):

{
   "error" : null,
   "id" : null,
   "result" : {
      "isvalid" : true,
      "scriptPubKey" : "76a91468bedce8982d25c3b6b03f6238cbad00378b8ead88ac",
      "address" : "mq4oDPjmNWnBxbzx7qouzhpCSTMePUtYDF"
   }
}

Hm; maybe this is also ARM-specific? I'm reaching out to the Bitseed team to see if they can reproduce. I don't have any experience running bitcoind in a debugger; any recommendations / documentation for next steps I should take?

Instead of running ./bitcoind -testnet ..., run

gdb bitcoind
> run -testnet ...

Try to crash it now. Then use the command where in gdb.

Unfortunately it seems to be crashing immediately in the debugger...

linaro@btc:~/bitcoin$ gdb src/bitcoind
GNU gdb (Ubuntu 7.7-0ubuntu3) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from src/bitcoind...done.
(gdb) run -testnet -disablewallet
Starting program: /home/linaro/bitcoin/src/bitcoind -testnet -disablewallet
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Cannot access memory at address 0x0

Program received signal SIGILL, Illegal instruction.
0xb6d0a248 in ?? () from /lib/arm-linux-gnueabihf/libcrypto.so.1.0.0

Why sudo?

I tried it both with and without sudo; same error each time.

How is bitcoind compiled? Can you recompile with --enable-debug?

Hi, this is Jay from Bitseed. The most recent version I have built is v0.11.1. validateaddress is not something I would specifically test. The main issues I have seen building on armhf are missing dependencies or incorrect versions of dependencies. I can run a build here. Are you building from Master?

Just tried validateaddress on a mainnet node with current master with --disable-wallet. Works for me. The question is: why would you wan't to use validateaddress in wallet disable mode? It's a utility function only then. Why not using something like https://github.com/petertodd/python-bitcoinlib or https://github.com/libbtc/libbtc? IMO there is no need for a RPC roundtrip for a such utility only function.

@jonasschnelli there have been quite a few incorrect address validator code snippits out there. There is nothing wrong with using Bitcoin Core for this, especially if you're already using it for other things.

We could just as easily build it into bitcoin-cli and avoid the server round-trip.

@jgarzik Only parts of it, e.g.:

  "ismine": true,

can't be made -cli only thing. The same applies to iswatchonly.

Please; the cost of forking bitcoin-cli and linking in all its libraries and such is significant enough that special casing this would be a waste of time and review bandwidth.

bitcoin-cli would be the wrong binary. bitcoin-tx is the current utility application, but right now, its a bit limited to transaction manipulation/creation. But sure, there is nothing wrong with using the RPC server for this. I would just reconsider my API design if I would run a intensive business and validate serval addresses per second over RPC.

Nod - for addresses, the plan has always been a separate "bitcoin-key" utility, which lets you play around with addresses and keys

Agreed, this would be better removed from the RPC for such a trivial utility function. However, its still a crash opportunity in its current state.

@paveljanik I'm still not having any luck with the debugger...

./autogen.sh
./configure --enable-debug
make
gdb src/bitcoind
GNU gdb (Ubuntu 7.7-0ubuntu3) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from src/bitcoind...done.
(gdb) run
Starting program: /home/linaro/bitcoin/src/bitcoind 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Cannot access memory at address 0x0

Program received signal SIGILL, Illegal instruction.
0xb6d0a248 in ?? () from /lib/arm-linux-gnueabihf/libcrypto.so.1.0.0

SIGILL in libcrypto. Interesting. If I have to guess, your OpenSSL is compiled with instruction set enabled that are not supported on your board. If it's not that then it's jumping to an invalid address somewhere... Does 'bt' still work in gdb to get a backtrace? It is also good to see what instruction is crashing it. Try x/10i $pc.

• Tried on a node compiled with --disable-wallet: validateaddress works fine
• Tried when compiled with wallet, but -disablewallet at run time: bitcoind crashes:

Is it possible you've started this bitcoind up with -disablewallet? I think that can trigger this crash on validateaddress.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffed7fa700 (LWP 23455)]
0x00005555556f90e1 in DescribeAddressVisitor::operator() (this=this@entry=0x7fffed7f9a00, scriptID=...) at rpcmisc.cpp:131
131             if (pwalletMain->GetCScript(scriptID, subscript)) {

This is different from your SIGILL crash.

@laanwj Your patch fixes the validateaddress crash I reported :+1:

As for the crash I experience in the debugger, here's the output I get from those commands:

(gdb) run -testnet -disablewallet
Starting program: /home/linaro/bitcoin/src/bitcoind -testnet -disablewallet
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Cannot access memory at address 0x0

Program received signal SIGILL, Illegal instruction.
0xb6d0a248 in ?? () from /lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
(gdb) bt
[#0](/bitcoin-bitcoin/0/)  0xb6d0a248 in ?? () from /lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
[#1](/bitcoin-bitcoin/1/)  0xb6d07ef0 in OPENSSL_cpuid_setup () from /lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
[#2](/bitcoin-bitcoin/2/)  0xb6fea20e in call_init (l=<optimized out>, argc=3, argv=0xbefff6e4, env=0xbefff6f4) at dl-init.c:78
[#3](/bitcoin-bitcoin/3/)  0xb6fea2a0 in _dl_init (main_map=0xb6fff958, argc=3, argv=0xbefff6e4, env=0xbefff6f4) at dl-init.c:126
[#4](/bitcoin-bitcoin/4/)  0xb6fdfbf2 in _dl_start_user () from /lib/ld-linux-armhf.so.3
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) x/10i $pc 
=> 0xb6d0a248:  mrc 15, 0, r0, cr9, cr13, {0}
   0xb6d0a24c:  bx  lr
   0xb6d0a250:  ldrex   r2, [r0]
   0xb6d0a254:  add r3, r2, r1
   0xb6d0a258:  strex   r2, r3, [r0]
   0xb6d0a25c:  cmp r2, [#0](/bitcoin-bitcoin/0/)
   0xb6d0a260:  bne 0xb6d0a250
   0xb6d0a264:  mov r0, r3
   0xb6d0a268:  bx  lr
   0xb6d0a26c <OPENSSL_cleanse>:    eor r12, r12, r12

It's trying to read the PMCCNTR register: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0363g/Bgbcefbc.html Looks like it is probing for support of that CPU feature, which is absent in your case. The debugger gets the first dibs on any signal so gets it before OpenSSL's own signal handler, which would have proceeded as normal. OK, thanks, seems this issue is resolved.