getrawtransaction delay too long on non-existent txid #11506

issue doersf opened this issue on October 16, 2017

doersf commented at 4:03 AM on October 16, 2017: none

As of 0.15.0.0, and 0.15.0.1, the getrawtransaction call lags when requesting non-existent transaction IDs.

Tested on Ubuntu 14.04 and 16.04.

Reproduce with:

$ bitcoin-cli getrawtransaction `openssl rand -hex 32` 1
fanquake added the label RPC/REST/ZMQ on Oct 16, 2017
achow101 commented at 4:12 AM on October 16, 2017: member

Lags by how much? It's almost an instant response for me.

Also, what did you expect? The transaction index (if you have that enabled for getrawtransaction to be useful) is massive. It's not exactly going to be a fast operation since it has to search through the entire index to find your nonexistent transaction. Although I have txindex enabled on my node and it returned a response almost immediately.
doersf commented at 4:15 AM on October 16, 2017: none

If it's as intended, that's fine. But large services doing lookups for user-entered inputs will suffer.

You can close this if it's not going to be fixed.

The issue was introduced in the 0.15.x branch.
achow101 commented at 4:22 AM on October 16, 2017: member

I am unable to reproduce your issue, so it does not even seem like there is an issue here. We can't fix a bug for which we cannot reproduce and have no debugging information for. Please provide more information if you think there is a bug.

Information that would be useful: Do you have txindex enabled? How long is the lag?
doersf commented at 4:36 AM on October 16, 2017: none

I missed that :)

Lag for a single non-existent transaction is between 0.321s and 0.350s. This is not fine. Lag for a single valid transaction is between 0.004s and 0.005s. Tested with size: 225b. This is fine.

txindex is enabled. Above results hold with and without re-indexing post upgrade to 0.15.

Highest machine configuration with the above results:

Ubuntu 16.04 8 CPUs 24GB RAM, 10GB free
achow101 commented at 5:03 AM on October 16, 2017: member

I see. This is a side effect of the new UTXO database structure. The slowdown is on searching the UTXO set for the txid, not on txindex. I think it can be sped up, but that may take a bit of work to do. Because the UTXO database is now keyed on the outpoint (txid hash and output index), we have to search through all possible outpoints for a given txid which is slower than previously which was keyed on only the txid. I'll see if it can be sped up.
sipa commented at 5:05 AM on October 16, 2017: member

This seems expected when you don't have txindex one, but with it it seems easy to avoid.
achow101 commented at 5:19 AM on October 16, 2017: member

This seems expected when you don't have txindex one, but with it it seems easy to avoid.

Is it not possible to just search for the prefix of the key (i.e. the txid) instead of the entire key?
achow101 commented at 5:32 AM on October 16, 2017: member

@doersf Does #11507 fix your problem?
sipa commented at 5:34 AM on October 16, 2017: member

@achow101 The UTXO set cache in memory is a hashtable - it does not support searching by prefix.
doersf commented at 7:37 AM on October 16, 2017: none

@achow101 this is better. The results are now:

single valid txid: 0.004s to 0.005s single non-existent txid: 0.004s to 0.005s

This should make 0.15.x production ready for us.
fanquake commented at 3:06 AM on October 21, 2017: member

@doersf Is this fixed now that #11529 is merged? Can you close if that is the case.
doersf commented at 3:14 AM on October 21, 2017: none

Things look fine. Thanks!
doersf closed this on Oct 21, 2017
MarcoFalke locked this on Sep 8, 2021

Contributors

Labels

Linked (view graph)

#11507 [RPC] Don't do slow transaction lookup when txindex is enabled