Bitcoind crashes without error due to rpc/zmq calls? #12773

Hello, my bitcoin node (v0.16.99.0-e057589) crashes without throwing an error while syncing indexd. It happens after a few hours (it seems to be happening faster every time) and is really frustrating. I'm using the latest release in this server: https://github.com/CounterpartyXCP/indexd-server It happens on both ubuntu and centos 7 on different systems so it doesn't seem hardware related.

bitcoin.conf

listen=1
rpcuser=xxxxxxxxxxxxx
rpcpassword=xxxxxxxxxxxx
rpcport=8332
rpcconnect=127.0.0.1
rpcallowip=222.22.22.22
rpcallowip=127.0.0.1
rpcallowip=22.222.222.22
zmqpubhashtx=tcp://127.0.0.1:38832
zmqpubhashblock=tcp://127.0.0.1:38832
server=1
txindex=1
addrindex=1
walletbroadcast=0
daemon=1
rpcthreads=16
rpcworkqueue=32

I've tried setting the rpcworkqueue and rpcthreads lower but it didn't help a bit. I've also played with the RPCCONCURRENT setting in the indexd server, setting it to 32 or 16 wont didn't help.

The debug.log, don't think there is anything interesting there. https://pastebin.com/T830Qg8k

Any alternative rest api's that I can use to get utxo's from an address are welcome, Its frustrating that I don't have many options here, I prefer to use a core node.

@dcousens

Have you tried other versions of bitcoind?

Yes I've tried it on 0.15.(1) too.

It would be helpful to run it in gdb and get a backtrace after the crash.

Probably I can give a hint here because I discovered something maybe related to that in 0.16 (release).

With debug enabled I got log entries from a certain point and later crashing:

2018-03-26 18:24:01 WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting

Well, turned out that my RPC client was sending incorrect requests like still asking for the getinfo or wrong tx info. The client was a small python asking pretty frequently but more or less ignoring the answer waiting forever for the expected one. Every minute it opened a new connection while the old was still active never timing out on client side.

Looks like we are keeping the crap in the queue on server side as well or at least for an hour as this was the average time from the first log entry to crash. Probably there's some room for improvement.

Increasing rpcworkqueue and workerthreads doesn't help

Has this issue persisted past v0.16.99? I'm unable to find related reports for later versions.

Closing for lack of activity/response.