mempool contains stale txs #4791

I'm noticing that the mempool of bitcoind is containing mostly stale txs that don't seem to be entering the blockchain:-

2014-08-29 18:32:40 AcceptToMemoryPool: peer=3 /Satoshi:0.9.2.1/ : accepted d1e57f6e66efec64cc923fea0a3344c07b44a6f513a936c614393951ad3508f5 (poolsz 754)

I then suspended my computer, resumed some hours later, catch up with the blockchain (but without accepting new txs until it's caughtup):-

2014-08-30 02:15:16 UpdateTip: new best=00000000000000001234c9309d7880eb14a9c5b11465a6cf1bea993888c70657 height=318173 log2_work=80.426739 tx=45613440 date=2014-08-30 01:27:51 progress=0.999879

The very next tx it accepts, reports:- 2014-08-30 02:15:16 AcceptToMemoryPool: peer=154 /Satoshi:0.9.2.1/ : accepted 694c6a535f349cc2e003fc6ef1de2e67e4dce8bc0a29b3c8ec3c59f699a25ae2 (poolsz 630)

So, eight hours later, 629 out of 630 txs in the mempool are almost eight hours old and still haven't been accepted into the blockchain.

What should the node do in this situation? Flush them from the mempool or should it be rebroadcasting them to the network? It doesn't currently look like it's doing either.

Version running is the latest master (Bitcoin version v0.9.99.0-unk (Jul 16 2014, 19:05:51))

See #3753

@laanwj But why are miners not including these transactions? I'm seeing frequent blocks that are significantly smaller than the maximum block size, so it's looking like there's space in the block chain for these transactions, isn't there?

I'm currently working on a patch that will give users the option to not relay blocks from stingy miners. I think this is the best way as a community we can recover the ability to reduce tx fees.

Miners are not obligated to mine every transaction -- and we are thankful they do not. Fees are not the only criteria for transaction selection. Issues like creating lots of dust outputs imply there will always be some garbage that simply does not get confirmed.

Thus, mempool janitor :) #3753

@jgarzik I agree, but wouldn't there be some wisdom is recitfying the current situation which favorably biases the smaller blocks (since they propagate faster)? With some code changes, it should be possible to delay the propagation of the smaller blocks so that they don't have this advantage. Also, in order to keep tx fees lower, giving priority to blocks which include more txs would help to incentivise miners to include more transactions. Given the amount of tx traffic currently, it surely makes more sense to reduce the number of empty blocks still being accepted into the blockchain.

There is work in progress for allowing faster relay of large blocks, by making use of data that was already transferred. See https://bitcoinfoundation.org/2014/08/a-bitcoin-backbone/ This is superior to delaying small blocks.

@rebroad You need to catch up on bitcoin-development mailing list traffic. As @laanwj indicated, current discussions focus on how to avoid re-sending every TX twice (once relay, once inside a block). Solving that problem is a much better path than delay blocks etc.