too many removeprunedfunds and listtransactions , bitcoind will be crash #9729

1. 
   JokerCatz commented at 10:25 am on February 9, 2017: none

   Hi , my wallet.dat file size is 300MB and over 6k wallet & many many transactions

   and I try to use removeprunedfunds to remove transactions , so I use listtransactions (no account specify , just listtransactions) to list and removeprunedfunds to remove

   my process log like

   [listtransactions]
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [listtransactions]
   [...removeprunedfunds TXID...] : 2017-02-09 16:36:06
   [...removeprunedfunds TXID...] : 2017-02-09 16:46:11
   [...removeprunedfunds TXID...] : 2017-02-09 16:46:34
   [...removeprunedfunds TXID...] : 2017-02-09 16:46:34
   [...removeprunedfunds TXID...] : 2017-02-09 16:50:58
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [...removeprunedfunds TXID...] : 2017-02-09 16:51:15
   [listtransactions]
   ## bitcoind crash (killed)
   

   and debug.log like (enabled decode=1)

   2017-02-09 10:09:31 received: tx (334 bytes) peer=3
   2017-02-09 10:09:31 Blockpolicy mempool tx ffd880afd9 adding to FeeRate
   2017-02-09 10:09:31 AcceptToMemoryPool: peer=3: accepted ffd880afd965ed98d7d0c2f13f242433dd09e6ab44fc808407721f2ac2041e57 (poolsz 4 txn, 4 kB)
   2017-02-09 10:09:31 received: tx (189 bytes) peer=4
   2017-02-09 10:09:31 Blockpolicy mempool tx eaec1247e8 adding to FeeRate
   2017-02-09 10:09:31 AcceptToMemoryPool: peer=4: accepted eaec1247e8f5b59a9228725ef89799aa6197776a447811cf235324253dfa3ae8 (poolsz 5 txn, 6 kB)
   2017-02-09 10:09:31 sending: inv Received a POST request for / from 127.0.0.1:35116
   2017-02-09 10:09:31 (109 bytes) peer=4
   2017-02-09 10:09:31 ThreadRPCServer method=listtransactions
   2017-02-09 10:09:31 received: headers (325 bytes) peer=5
   [EOF]
   

   my config has

   upnp=0
   server=1
   keypool=300
   mintxfee=0.00000001
   txindex=1
   debug=1
   

   my pc is Ubuntu 16.10 / SSD / 7 CPU / 19G RAM / bitcoin core ver : 130200

   hmm , anyway , I think this is bug , and if I remove all transaction , the wallet.dat file size will be smaller?

2. laanwj added the label Resource usage on Feb 9, 2017
3. laanwj added the label Wallet on Feb 9, 2017
4. 
   laanwj commented at 12:25 pm on February 9, 2017: member

   Not much to go on for troubleshooting this. Though (Killed) sounds like Linux’ OOM killer got to it. How much memory is it using? If it is it growing faster if you use listtransactions/removeprunedfunds may indicate a memory leak.
5. 
   JokerCatz commented at 2:39 am on February 10, 2017: none

   hi , thanx reply , I monitor the IO(iotop) < 10% , CPU(top) 99%+ , memory(top) < 7.2% (start at 3.4% and when I start process , grow to 7.2%，so 662MB => 1400MB !?)

   I mean “killed” is the bitcoind process disappear with no log

   the crash always after many removeprunedfunds and immediately listtransactions(crash here)

   but I tried removeprunedfunds + sleep(120sec) and listtransactions , it crash too

   my walletversion is 60000

   I use ruby + rest-client gem to use JSON-RPC , and code like this

   loop do
     puts "call : BitcoinRPC.listtransactions"
     transactions = BitcoinRPC.listtransactions
     break if transactions.empty?
     transactions.each do |tran|
       puts "#{tran['txid']} : #{Time.at(tran['time'].to_i).strftime('%Y-%m-%d %H:%M:%S')}"
       begin
         BitcoinRPC.removeprunedfunds(tran['txid'])
       rescue Exception => e
         raise e unless e.message.match(/Transaction does not exist in wallet/)
       end
     end
   end
   

   just a loop to get transactions and remove it when bitcoind crash , I got exception from rest-client gem : end of file reached (EOFError)

6. 
   laanwj commented at 10:41 am on February 10, 2017: member

   Would be useful to run bitcoind in gdb and get a stack traceback (bt) when it happens.
7. 
   MarcoFalke commented at 11:41 pm on March 23, 2018: member

   @JokerCatz Any success digging up the bt?
8. 
   laanwj commented at 3:35 pm on June 18, 2018: member

   Should be fixed by #13437
9. laanwj closed this on Jun 18, 2018

   ---

10. 
    MarcoFalke commented at 3:38 pm on June 18, 2018: member

    I am not 100% that this issue is fixed in master so if @JokerCatz could try again with current master and report back, that’d be great.
11. DrahtBot locked this on Sep 8, 2021

Contributors
JokerCatz laanwj MarcoFalke

Labels
Wallet Resource usage