missing transaction raw data #2077

Using today's HEAD (85887020dc6080e5707632a4ea68232f66d51d69) I downloaded all blocks, and doing TX introspection block #210990 has tx a4e35aba7b7e12756ddb1084abf0cd124e14d5214253aeb2ea33783dbebfdfca but calling getrawtransaction yields error: {"code":-5,"message":"No information available about transaction"}.

This is a known side effect of the new database layout in v0.8. There is no global transaction index anyone, only one for unspent transactions, since that is all that is needed for normal operation. The intention is to add a full index, but make it optional for people who need getrawtransaction for any transaction.

Oh, I did not know that, but it obviously makes perfect sense when using the client as a wallet. I am using this particular instance as a transaction and block sniffer and as such need to have all the transaction data.

I assume that, since I don't have anything on this instance's wallet, as wallets get broadcasted all spent transactions are trimmed away from the db, which should be a simple thing to prevent? I'll look at the code as soon as I have a moment, but any pointers are obviously welcome.

It has nothing to do with wallets - the block validation system and the wallet system are quite separate. Wallets keep whatever information they need themselves.

It's just how the new validation engine is designed: there is never need for more information than unspent transactions outputs, so that is the only piece of data that is kept. Changing this would break it.

However, we still have all full blocks around on disk. All that is needed is adding an extra database with a transaction index like we used to have, which is only used for getrawtransaction.

Ok, so when parsing and validating a block the client does what it needs to do and then stores the unspent transactions for validating future blocks only, and when these are marked as spent they get removed from the database. Is this correct?

Not removing spent transactions would break the validation engine because it is assuming all transactions in the db have unspent outputs? My initial thought was that if the spent transactions weren't removed from the db, it would be the simplest solution to my needs but that would imply them being properly marked as spent or not for the validation engine, of course.

There is currently NO transaction index at all. The fact that getrawtransaction works, is because we can use the UTXO database to find (for unspent transaction) the confirmations of the transaction, calculate which height that corresponds to, search the block index for the disk location of the block at that height, read that entire block from disk, and look for the transaction you wanted in it. This is not fast, and doesn't work well at all, but it works a bit. However, this database was never designed to do this well. The only thing it is good at is verifying transactions very quickly.

If you need the ability to look up transactions in the block chain, the only real solution is an index. I really prefer not to mess with the UTXO database to be able to deal with such queries in a bad way.

Fair enough, thanks for all the explaining. I might get my hands dirty on the transaction index, but I'm well served with 0.7.1 for now, just wanted to get some testing on the development version.

How about another approach for getting the raw transactions through the API when the block is known to the client:

Add third optional "blockhash" parameter to "getrawtransaction". The parameter could be used to specify the block which is used to look up the raw transaction.

Or alternatively: Introduce optional "verbose" bool flag to "getblock", which would output all the included transactions in full instead of just their IDs.

What do you think? This would work without the additional index. Which approach would be better?

Hi! Blockhash parameter on getrawtransaction is good help for any transaction inspector. Regards!

Hi! I generated a commit usefull for me. I sent a pull request #2421 Regards!