Failed transactions on importing mempool #32125

1. 
   jsarenik commented at 5:47 am on March 23, 2025: none

   Is there an existing issue for this?

   • I have searched the existing issues

   Current behaviour

   Hello! This is rather a question. I am wondering: are the 10 failed related to package relaying?

   29.0rc2

    02025-03-23T05:24:22Z SetNetworkActive: false
    12025-03-23T05:24:27Z Loading 20483 mempool transactions from file...
    22025-03-23T05:24:27Z Progress loading mempool transactions from file: 10% (tried 2049, 18434 remaining)
    32025-03-23T05:24:27Z Progress loading mempool transactions from file: 20% (tried 4097, 16386 remaining)
    42025-03-23T05:24:27Z Progress loading mempool transactions from file: 30% (tried 6145, 14338 remaining)
    52025-03-23T05:24:27Z Progress loading mempool transactions from file: 40% (tried 8194, 12289 remaining)
    62025-03-23T05:24:28Z Progress loading mempool transactions from file: 50% (tried 10242, 10241 remaining)
    72025-03-23T05:24:28Z Progress loading mempool transactions from file: 60% (tried 12290, 8193 remaining)
    82025-03-23T05:24:28Z Progress loading mempool transactions from file: 70% (tried 14339, 6144 remaining)
    92025-03-23T05:24:28Z Progress loading mempool transactions from file: 80% (tried 16387, 4096 remaining)
   102025-03-23T05:24:28Z Progress loading mempool transactions from file: 90% (tried 18435, 2048 remaining)
   112025-03-23T05:24:28Z Imported mempool transactions from file: 0 succeeded, 10 failed, 0 expired, 20473 already there, 0 waiting for initial broadcast
   122025-03-23T05:24:28Z SetNetworkActive: true
   

   To get fresh mempool ready to import, see:

   • mymempool: txt, dat, log, getmempoolinfo (JSON)

   Expected behaviour

   The nodes I test are running on my local network so I guess all the transactions are importable, but the 10 failed could be 5 packages of 2.

   Steps to reproduce

   Import from file using RPC

   Relevant log output

   No response

   How did you obtain Bitcoin Core

   Compiled from source

   What version of Bitcoin Core are you using?

   v29.0rc2

   Operating system and version

   Alpine Linux 3.21.3

   Machine specifications

   Linux x86_64

2. 
   maflcko commented at 8:59 am on March 24, 2025: member

   The nodes I test are running on my local network so I guess all the transactions are importable, but the 10 failed could be 5 packages of 2.

   Can you add exact steps to reproduce, please?

   The mempool acceptance depends on the chain state (the best block hash of your node at the time of the mempool import), as well as the mempool file contents.

   Without any exact info, my guess would be that the 10 failures are due to transactions already confirmed (either themselves or a replacement of themselves).

   Also, package acceptance is already a todo comment in the code, see

   https://github.com/bitcoin/bitcoin/blob/770d39a37652d40885533fecce37e9f71cc0d051/test/functional/mempool_ephemeral_dust.py#L129-L135

3. maflcko added the label Questions and Help on Mar 24, 2025
4. 
   jsarenik commented at 4:22 pm on March 24, 2025: none

   Thank you for feedback, @maflcko ! In the end of this post I include shell script. The node is started always with persistmempool=0 and I have also mempool.dat file symlinked to /dev/null just to remind me on a filesystem level.

   Yes, my chainstate always contains the last-recent block when starting import.

   Now I tried to artificially change priority of each of the transactions before importing them (see in the script at the end) and it worked:

   02025-03-24T16:06:18Z Imported mempool transactions from file: 40560 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
   

   Just that I imagine it could be much faster than 9 minutes I got with this shell hack if there was an option to importmempool RPC that would make sure every single (unless spent already) gets into mempool.

   Here is the script I call mymempoolhere.sh:

    0#!/bin/sh
    1
    2BN=89999115000 # arbitrary big number usable here
    3hp=anyone.eu.org
    4bitcoin-cli echo hello | grep -q . || exit 1
    5ping -qc 1 -W5 -w5 $hp >/dev/null || { hp=192.168.3.118:12321; scheme=http; }
    6url=${scheme:-"https"}://$hp
    7out=/dev/shm/mempool.copy
    8touch $out 2>/dev/null || out=$HOME/mempool.copy
    9rm -f $out
   10uptime=$(bitcoin-cli uptime) || exit 1
   11test $uptime -gt 613 || bitcoin-cli setnetworkactive false
   12wget -O $out $url/mempool.copy
   13wget -qO - $url/mymempool.txt \
   14  | xargs -I TXID -n 1 -P 200 bitcoin-cli prioritisetransaction TXID "0.0" $BN \
   15    >/dev/null 2>&1
   16bitcoin-cli importmempool $out '{
   17"apply_fee_delta_priority":true,
   18"apply_unbroadcast_set":true}'
   19rm -f $out
   20bitcoin-cli setnetworkactive true
   21date -u
   22
   23# optiontal clean-up would mess up logs so maybe exit before here
   24bitcoin-cli getprioritisedtransactions \
   25  | grep -e '^  "' -e '"fee_delta":' \
   26  | tr -d '  {,' | paste -d" " - - \
   27  | sed 's/"fee_delta":/"0.0" +/' \
   28  | sed 's/+-//' | tr '+' '-' | tr -d : \
   29  | xargs -n 3 -P 200 bitcoin-cli prioritisetransaction \
   30    >/dev/null 2>&1
   

   UPDATE: it takes ~2 minutes on an already-loaded mempool so for me it is already OK

5. 
   jsarenik commented at 8:37 pm on March 24, 2025: none

   Another example of what I was thinking of in the beginning and what is already possible using the script that prioritizes transactions before importing them:

   0newlines added
   12025-03-24T20:31:45Z
   2  Imported mempool transactions from file:
   3    1267 succeeded, 0 failed, 0 expired, 29536 already there,
   4    0 waiting for initial broadcast
   

   One thing I can think of right now would be an RPC which clears all prioritization. No idea how to do it in Bitcoin Core yet, but the idea is here. Such RPC would shrink the clean-up part of my script to one line.

6. maflcko removed the label Questions and Help on Mar 25, 2025
7. 
   maflcko commented at 8:37 am on March 25, 2025: member

   You can modify the source code in src/node/mempool_persist.cpp to print the transaction ids (and error reasons) of the failed imports. Then call getmempoolentry to get the mempool topology and fees after your successful import to debug this further.
8. 
   willcl-ark commented at 11:05 am on March 28, 2025: member

   Going to close this as I don’t think there’s anything left to do here. Let me know if I’ve misunderstood if you want it re-opened.
9. willcl-ark closed this on Mar 28, 2025

   ---

Contributors
jsarenik maflcko willcl-ark