Bitcoin-core multiwallets limited to about 300 loads #15291

issue fscemama openend this issue on January 30, 2019
  1. fscemama commented at 10:32 am on January 30, 2019: none

    Hi, Using Bitcoin Core RPC client version v0.17.0.1-g1476554d3b1db5352c0de1de7524864d4bce3aac

    I’m using bitcoin-cli loadwallet and unloadwallet commands. They both seem to work fine. Unfortunately, memory is getting higher and higher, and no wallet can be loaded after 300 have been. The problem may be related to BerkeleyDB rather than to Bitcoin Core itself. Fact is that lsof shows handles not cleared off after the unloadwallet command is issued. Files still handled are .walletlock, db.log and database/log.0000000001 . When 300 different wallets have been loaded/unloaded, used RAM is over 8 GB, and bitcoind will finally become unstable and fail.

    Any help would be appreciated. Regards,

  2. promag commented at 12:01 pm on January 30, 2019: member

    Are you doing some stress test or are these load/unload done over a long period of time?

    Is the same wallet loaded/unloaded multiple times?

  3. fscemama commented at 12:05 pm on January 30, 2019: none

    Hi Joao,

    These load/unloads are done on a short period of time. We check hundreds of wallets for new transactions, so to feed a database.

    The same wallet may be loaded/unloaded multiple times, but one is enough
    the file handles won’t be released.

    Regards,

    Fabrice SCEMAMA EFFITEK SSII - http://www.effitek.fr Tour Onyx, 10 rue Vandrezanne, 75013 Paris fscemama@effitek.fr mailto:fscemama@effitek.fr - Tel. : 06 65 50 29 92

    Le 30/01/2019 à 13:02, João Barbosa a écrit :

    Are you doing some stress test or are these load/unload done over a long period of time?

    Is the same wallet loaded/unloaded multiple times?

    — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/15291#issuecomment-458919362, or mute the thread https://github.com/notifications/unsubscribe-auth/As_ZItqQCSIKHksHP8sn34z0Oc0VpieFks5vIYnIgaJpZM4aZ6AT.

  4. promag commented at 12:25 pm on January 30, 2019: member
    unloadwallet behavior was changed in #14941, you could try that (it was also backport to 0.17 branch in #15002).
  5. fscemama commented at 12:37 pm on January 30, 2019: none
    Thank you for the reference. I’m going to give it a test. Agreed, an asynchrous unloadwallet can be useful (faster), but can also lead to disasters. I’ve always been surprised that BerkeleyDB 4 should be used, but that’s another story.
  6. promag commented at 3:57 pm on January 30, 2019: member
    @fscemama thanks, please let us know how it went.
  7. fscemama commented at 10:16 pm on January 30, 2019: none
    Hi, just stress-tested with today’s github master. Result is alas same. Basic testing: Firstly, load 30 wallets: j=0; for i in /wallets/*; do j=echo $j+1|bc; if [ $j -lt 30 ]; then bitcoin-cli loadwallet "$i"; fi; done Then, unload them: for i in /wallets/*;do bitcoin-cli unloadwallet "$i"; done Look for handles: lsof -p pidof bitcoind |grep wallet 89 lines, .walletlock, db.log, etc. As for memory, 1.5 GB are used by bitcoind. I must mention unloadwallet commands have taken no time to execute. Thanks !
  8. promag commented at 2:42 pm on January 31, 2019: member
    But do you see memory leak?
  9. fscemama commented at 2:57 pm on January 31, 2019: none
    I’m not sure about what is a memory leak. Surely, the memory footprint remains very high (1.5 GB for 30 loaded wallets, and no memory is released after wallets are unloaded). In fact, lsof return proves wallets db files still being open. Unloadwallet does not close DB files. BTW, loading one more wallet should not take 400+ MB of memory, there’s certainly something to optimize there, but that’s a drawback we can live with. I’m sorry, I have no C++ knowledge to give any useful help in code.
  10. promag commented at 3:06 pm on January 31, 2019: member
    If each wallet takes 400+ MB then with 30 wallets you would need much more memory than 1.5GB. Can you check memory usage after each loadwallet and then after each unloadwallet. You could test #15297 branch which closes those files?
  11. fscemama commented at 3:09 pm on January 31, 2019: none
    40-50 MB. Sorry, end of week!!
  12. fscemama commented at 3:11 pm on January 31, 2019: none
    I agree to test, please let me know when I can download the master. Then I’ll compile and let you know. The hack proposed in #15297 looks fine.
  13. promag commented at 3:18 pm on January 31, 2019: member

    It would help if you test that branch instead of master:

    0git fetch git@github.com:bitcoin/bitcoin.git refs/pull/15297/head
1git checkout -b pr15297 FETCH_HEAD
2make -j
3...
  14. fscemama commented at 3:19 pm on January 31, 2019: none
    THANKS.
  15. fscemama commented at 6:14 pm on January 31, 2019: none

    Still unrelased handles. Sample output of the lsof command:

    0bitcoind 1568 root   37uW     REG      8,1         0   670564 /wallets/xxxx/.walletlock
1bitcoind 1568 root   38w      REG      8,1         0   670567 /wallets/xxxx/db.log
2bitcoind 1568 root   41u      REG      8,1   1048576   654805 /wallets/xxxx/database/log.0000000001

    I’ve considered the possibility of clearing off these file handles the kernel way, like this: echo -e "p close(654805)\nquit\ny\n" | gdp -p 1568 And this is not a good idea : BerkeleyDB will complain and this could lead to database corruption. Maybe I’ve missed something, the #15297 thread makes me feel like trying to google the BDB issue and try to patch it.

  16. fscemama commented at 6:25 pm on January 31, 2019: none
    I’m a newbie as for C++ and I may be wrong, but I’m not sure the closing of the WalletDatabase is ever called.
  17. promag commented at 6:32 pm on January 31, 2019: member
    Mind sharing the steps to reproduce here?
  18. fscemama commented at 6:38 pm on January 31, 2019: none

    No more than that:

    Basic testing: Firstly, load 30 wallets: j=0; for i in /wallets/*; do j=echo $j+1|bc; if [ $j -lt 30 ]; then bitcoin-cli loadwallet "$i"; fi; done Then, unload them: for i in /wallets/*;do bitcoin-cli unloadwallet "$i"; done Look for handles: lsof -p pidof bitcoind |grep wallet 89 lines, .walletlock, db.log, etc. As for memory, 1.5 GB are used by bitcoind.

    Regards,

  19. promag commented at 7:00 pm on January 31, 2019: member

    @fscemama are you sure you built #15297 branch? I can’t reproduce your case.

    Anyway, I’ll push a commit later testing that the files should not exists after unloadwallet calls.

  20. fscemama commented at 10:00 pm on January 31, 2019: none

    Really got it. Bitcoin Core RPC client version v0.17.0.1-g1476554d3b1db5352c0de1de7524864d4bce3aac

    My part of db.cpp:

     0void BerkeleyEnvironment::Close()
 1{
 2    if (!fDbEnvInit)
 3        return;
 4
 5    fDbEnvInit = false;
 6
 7    for (auto& db : m_databases) {
 8        auto count = mapFileUseCount.find(db.first);
 9        assert(count == mapFileUseCount.end() || count->second == 0);
10        BerkeleyDatabase& database = db.second.get();
11        if (database.m_db) {
12            database.m_db->close(0);
13            database.m_db.reset();
14        }
15    }
16
17    FILE* error_file = nullptr;
18    dbenv->get_errfile(&error_file);
19
20    int ret = dbenv->close(0);
21    if (ret != 0)
22        LogPrintf("BerkeleyEnvironment::Close: Error %d closing database environment: %s\n", ret, DbEnv::strerror(ret));
23    if (!fMockDb)
24        DbEnv((u_int32_t)0).remove(strPath.c_str(), 0);
25
26    if (error_file) fclose(error_file);
27
28    UnlockDirectory(strPath, ".walletlock");
29}
  21. promag commented at 10:07 pm on January 31, 2019: member
    Sorry, what gives ./src/bitcoind --version?
  22. fscemama commented at 10:31 pm on January 31, 2019: none
    For some reason, make install in master-bitcoin dit not overwrite /usr/local/bin/bitcoind. I’ve done it manually and retested. I confirm the file handles are cleared off. Memory footprint is not reduced, though.
  23. fscemama commented at 10:37 pm on January 31, 2019: none 
     0# /usr/local/bin/bitcoind --version
 1Bitcoin Core Daemon version v0.17.99.0-bdbb3b5a8
 2Copyright (C) 2009-2019 The Bitcoin Core developers
 3
 4Please contribute if you find Bitcoin Core useful. Visit
 5<https://bitcoincore.org> for further information about the software.
 6The source code is available from <https://github.com/bitcoin/bitcoin>.
 7
 8This is experimental software.
 9Distributed under the MIT software license, see the accompanying file COPYING
10or <https://opensource.org/licenses/MIT>
11
12This product includes software developed by the OpenSSL Project for use in the
13OpenSSL Toolkit <https://www.openssl.org> and cryptographic software written by
14Eric Young and UPnP software written by Thomas Bernard.
  24. sipa commented at 10:38 pm on January 31, 2019: member

    Releasing memory back to the system is often nontrivial due to fragmentation.

    The bigger question is whether if you first load 300 wallets, then unload them, and then load 300 (other) wallets, does the memory usage stay constant the second time, or does it go up further?

  25. promag commented at 10:46 pm on January 31, 2019: member

    Thanks for testing @fscemama and confirming the issue is fixed.

    Regarding memory, @sipa is right, a good test should use lots of non empty wallets (with lots of transactions and keys).

  26. fscemama commented at 11:22 pm on January 31, 2019: none

    Thanks @promag and @sipa for making Bitcoin Core better. Here are a few tests, which may help understand why memory is not released.

      0bitcoin-cli stop
  1...restart...
  2ls -l /root/wallets|wc
  3#     160    1433   12484
  4j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 0 ] && [ $j -lt 10 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
  5# memory: 988168
  6for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
  7# memory: 988696
  8j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 10 ] && [ $j -lt 20 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
  9# memory: 990.3m
 10for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
 11# memory: 990.3m
 12j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 20 ] && [ $j -lt 30 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
 13# memory: 0.991g
 14for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
 15# memory: 0.991g
 16j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 0 ] && [ $j -lt 30 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
 17# memory: 1.415g
 18for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
 19# memory: 1.415g
 20j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 0 ] && [ $j -lt 30 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
 21# memory: 1.439g
 22for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
 23# memory: 1.439g
 24j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 30 ] && [ $j -lt 60 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
 25# memory: 1.462g
 26for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
 27# memory: 1.461g
 28j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 60 ] && [ $j -lt 90 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
 29# memory: 1.485g
 30for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
 31# memory: 1.485g
 32j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 0 ] && [ $j -lt 90 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
 33# memory: 2.582g
 34for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
 35# memory: 2.582g
 36j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 0 ] && [ $j -lt 90 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
 37# memory: 2.606g
 38for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
 39# memory: 2.607g
 40
 41lsof -p `pidof bitcoind`
 42
 43COMMAND   PID USER   FD      TYPE             DEVICE  SIZE/OFF      NODE NAME
 44bitcoind 2862 root  cwd       DIR                8,3      4096  99876868 /home/bitcoin/testnet3
 45bitcoind 2862 root  rtd       DIR                8,2      4096         2 /
 46bitcoind 2862 root  txt       REG                8,2 122945248    156543 /usr/local/bin/bitcoind
 47bitcoind 2862 root  mem       REG                8,3   2180920  99877103 /home/bitcoin/testnet3/chainstate/018395.ldb
 48bitcoind 2862 root  mem       REG                8,3   2179799  99891374 /home/bitcoin/testnet3/chainstate/009873.ldb
 49bitcoind 2862 root  mem       REG                8,3   2174556  99877377 /home/bitcoin/testnet3/chainstate/019086.ldb
 50bitcoind 2862 root  mem       REG                8,3   2173092  99877375 /home/bitcoin/testnet3/chainstate/019084.ldb
 51bitcoind 2862 root  mem       REG                8,3   2176542  99877374 /home/bitcoin/testnet3/chainstate/019083.ldb
 52bitcoind 2862 root  mem       REG                8,2     84848    917110 /lib/x86_64-linux-gnu/libresolv-2.24.so
 53bitcoind 2862 root  mem       REG                8,2     47632    917022 /lib/x86_64-linux-gnu/libnss_files-2.24.so
 54bitcoind 2862 root  mem       REG                8,3   2180040  99891355 /home/bitcoin/testnet3/chainstate/010262.ldb
 55bitcoind 2862 root  mem       REG                8,3   2182434  99876958 /home/bitcoin/testnet3/chainstate/018475.ldb
 56bitcoind 2862 root  mem       REG                8,3   2175576  99877027 /home/bitcoin/testnet3/chainstate/018391.ldb
 57bitcoind 2862 root  mem       REG                8,3   2179082  99877025 /home/bitcoin/testnet3/chainstate/018380.ldb
 58bitcoind 2862 root  mem       REG                8,3   2174953  99876890 /home/bitcoin/testnet3/chainstate/018342.ldb
 59bitcoind 2862 root  mem       REG                8,3   2175645  99876892 /home/bitcoin/testnet3/chainstate/018480.ldb
 60bitcoind 2862 root  mem       REG                8,3   2183696  99877147 /home/bitcoin/testnet3/chainstate/018412.ldb
 61bitcoind 2862 root  mem       REG                8,3   2177506  99876959 /home/bitcoin/testnet3/chainstate/018345.ldb
 62bitcoind 2862 root  mem       REG                8,3   2176305  99877277 /home/bitcoin/testnet3/chainstate/018618.ldb
 63bitcoind 2862 root  mem       REG                8,3   2177619  99876984 /home/bitcoin/testnet3/chainstate/018419.ldb
 64bitcoind 2862 root  mem       REG                8,3   2182692  99891431 /home/bitcoin/testnet3/chainstate/003178.ldb
 65bitcoind 2862 root  mem       REG                8,3   2179768  99877121 /home/bitcoin/testnet3/chainstate/018359.ldb
 66bitcoind 2862 root  mem       REG                8,3   2181562  99877243 /home/bitcoin/testnet3/chainstate/018612.ldb
 67bitcoind 2862 root  mem       REG                8,3   2178768  99891433 /home/bitcoin/testnet3/chainstate/003179.ldb
 68bitcoind 2862 root  mem       REG                8,3   2182919  99891799 /home/bitcoin/testnet3/chainstate/003133.ldb
 69bitcoind 2862 root  mem       REG                8,3   2176229  99876887 /home/bitcoin/testnet3/chainstate/018340.ldb
 70bitcoind 2862 root  mem       REG                8,3   2180656  99891669 /home/bitcoin/testnet3/chainstate/005762.ldb
 71bitcoind 2862 root  mem       REG                8,3   2177080  99891451 /home/bitcoin/testnet3/chainstate/009564.ldb
 72bitcoind 2862 root  mem       REG                8,3   2181179  99890591 /home/bitcoin/testnet3/chainstate/003108.ldb
 73bitcoind 2862 root  mem       REG                8,3   2181264  99877029 /home/bitcoin/testnet3/chainstate/018382.ldb
 74bitcoind 2862 root  mem       REG                8,3   2177242  99890345 /home/bitcoin/testnet3/chainstate/008057.ldb
 75bitcoind 2862 root  mem       REG                8,3   2181907  99890437 /home/bitcoin/testnet3/chainstate/009699.ldb
 76bitcoind 2862 root  mem       REG                8,3   2178513  99890553 /home/bitcoin/testnet3/chainstate/009866.ldb
 77bitcoind 2862 root  mem       REG                8,3   2177449  99891315 /home/bitcoin/testnet3/chainstate/006630.ldb
 78bitcoind 2862 root  mem       REG                8,3   2177974  99892097 /home/bitcoin/testnet3/chainstate/010241.ldb
 79bitcoind 2862 root  mem       REG                8,3   2177291  99891201 /home/bitcoin/testnet3/chainstate/009639.ldb
 80bitcoind 2862 root  mem       REG                8,3   2177460  99877233 /home/bitcoin/testnet3/chainstate/018495.ldb
 81bitcoind 2862 root  mem       REG                8,3   2181081  99891351 /home/bitcoin/testnet3/chainstate/009733.ldb
 82bitcoind 2862 root  mem       REG                8,3   2177924  99877118 /home/bitcoin/testnet3/chainstate/018398.ldb
 83bitcoind 2862 root  mem       REG                8,3   2176261  99877033 /home/bitcoin/testnet3/chainstate/018406.ldb
 84bitcoind 2862 root  mem       REG                8,3   2177667  99876962 /home/bitcoin/testnet3/chainstate/018476.ldb
 85bitcoind 2862 root  mem       REG                8,3   2178498  99891398 /home/bitcoin/testnet3/chainstate/003300.ldb
 86bitcoind 2862 root  mem       REG                8,3   2175006  99877010 /home/bitcoin/testnet3/chainstate/018347.ldb
 87bitcoind 2862 root  mem       REG                8,3   2182108  99891788 /home/bitcoin/testnet3/chainstate/003216.ldb
 88bitcoind 2862 root  mem       REG                8,3   2176581  99877128 /home/bitcoin/testnet3/chainstate/018364.ldb
 89bitcoind 2862 root  mem       REG                8,3   2182743  99889425 /home/bitcoin/testnet3/chainstate/003161.ldb
 90bitcoind 2862 root  mem       REG                8,3   2179578  99891379 /home/bitcoin/testnet3/chainstate/010397.ldb
 91bitcoind 2862 root  mem       REG                8,3   2182454  99891715 /home/bitcoin/testnet3/chainstate/003211.ldb
 92bitcoind 2862 root  mem       REG                8,3   2174949  99877122 /home/bitcoin/testnet3/chainstate/018360.ldb
 93bitcoind 2862 root  mem       REG                8,3   2182177  99891031 /home/bitcoin/testnet3/chainstate/009558.ldb
 94bitcoind 2862 root  mem       REG                8,3   2177919  99877235 /home/bitcoin/testnet3/chainstate/018497.ldb
 95bitcoind 2862 root  mem       REG                8,3   2183410  99891519 /home/bitcoin/testnet3/chainstate/003312.ldb
 96bitcoind 2862 root  mem       REG                8,3   2177699  99891771 /home/bitcoin/testnet3/chainstate/008062.ldb
 97bitcoind 2862 root  mem       REG                8,3   2182885  99891367 /home/bitcoin/testnet3/chainstate/009871.ldb
 98bitcoind 2862 root  mem       REG                8,3   2178331  99890491 /home/bitcoin/testnet3/chainstate/003173.ldb
 99bitcoind 2862 root  mem       REG                8,3   2177929  99891055 /home/bitcoin/testnet3/chainstate/010257.ldb
100bitcoind 2862 root  mem       REG                8,3   2182026  99876997 /home/bitcoin/testnet3/chainstate/018427.ldb
101bitcoind 2862 root  mem       REG                8,3   2177335  99877213 /home/bitcoin/testnet3/chainstate/018465.ldb
102bitcoind 2862 root  mem       REG                8,3   2176444  99892072 /home/bitcoin/testnet3/chainstate/009794.ldb
103bitcoind 2862 root  mem       REG                8,3   2176904  99891310 /home/bitcoin/testnet3/chainstate/009641.ldb
104bitcoind 2862 root  mem       REG                8,3   2177353  99877223 /home/bitcoin/testnet3/chainstate/018483.ldb
105bitcoind 2862 root  mem       REG                8,3   2183308  99876900 /home/bitcoin/testnet3/chainstate/018469.ldb
106bitcoind 2862 root  mem       REG                8,3   2178994  99891811 /home/bitcoin/testnet3/chainstate/003219.ldb
107bitcoind 2862 root  mem       REG                8,3   2178455  99877144 /home/bitcoin/testnet3/chainstate/018366.ldb
108bitcoind 2862 root  mem       REG                8,3   2178159  99891313 /home/bitcoin/testnet3/chainstate/003174.ldb
109bitcoind 2862 root  mem       REG                8,3   2174213  99877116 /home/bitcoin/testnet3/chainstate/018357.ldb
110bitcoind 2862 root  mem       REG                8,3   2182530  99891407 /home/bitcoin/testnet3/chainstate/009672.ldb
111bitcoind 2862 root  mem       REG                8,3   2176661  99876921 /home/bitcoin/testnet3/chainstate/011680.ldb
112bitcoind 2862 root  mem       REG                8,3   2180478  99890336 /home/bitcoin/testnet3/chainstate/009886.ldb
113bitcoind 2862 root  mem       REG                8,3   2182688  99891741 /home/bitcoin/testnet3/chainstate/009704.ldb
114bitcoind 2862 root  mem       REG                8,3   2181569  99880806 /home/bitcoin/testnet3/chainstate/009787.ldb
115bitcoind 2862 root  mem       REG                8,3   2179517  99877205 /home/bitcoin/testnet3/chainstate/018461.ldb
116bitcoind 2862 root  mem       REG                8,3   2175639  99891738 /home/bitcoin/testnet3/chainstate/010413.ldb
117bitcoind 2862 root  mem       REG                8,3   2184423  99890271 /home/bitcoin/testnet3/chainstate/003164.ldb
118bitcoind 2862 root  mem       REG                8,3   2181247  99891587 /home/bitcoin/testnet3/chainstate/003112.ldb
119bitcoind 2862 root  mem       REG                8,3   2177058  99877078 /home/bitcoin/testnet3/chainstate/018385.ldb
120bitcoind 2862 root  mem       REG                8,3   2181950  99890418 /home/bitcoin/testnet3/chainstate/010245.ldb
121bitcoind 2862 root  mem       REG                8,3   2180119  99889728 /home/bitcoin/testnet3/chainstate/009732.ldb
122bitcoind 2862 root  mem       REG                8,3   2181931  99891810 /home/bitcoin/testnet3/chainstate/003136.ldb
123bitcoind 2862 root  mem       REG                8,3   2175033  99876888 /home/bitcoin/testnet3/chainstate/018341.ldb
124bitcoind 2862 root  mem       REG                8,3   2183125  99890558 /home/bitcoin/testnet3/chainstate/009780.ldb
125bitcoind 2862 root  mem       REG                8,3   2177796  99876875 /home/bitcoin/testnet3/chainstate/011679.ldb
126bitcoind 2862 root  mem       REG                8,3   2176473  99891244 /home/bitcoin/testnet3/chainstate/009640.ldb
127bitcoind 2862 root  mem       REG                8,3   2181659  99891476 /home/bitcoin/testnet3/chainstate/009692.ldb
128bitcoind 2862 root  mem       REG                8,3   2175739  99891854 /home/bitcoin/testnet3/chainstate/008066.ldb
129bitcoind 2862 root  mem       REG                8,3   2175583  99877276 /home/bitcoin/testnet3/chainstate/018617.ldb
130bitcoind 2862 root  mem       REG                8,3   2183806  99891823 /home/bitcoin/testnet3/chainstate/009770.ldb
131bitcoind 2862 root  mem       REG                8,3   2181095  99891518 /home/bitcoin/testnet3/chainstate/009604.ldb
132bitcoind 2862 root  mem       REG                8,3   2180066  99891221 /home/bitcoin/testnet3/chainstate/009689.ldb
133bitcoind 2862 root  mem       REG                8,3   2180697  99890670 /home/bitcoin/testnet3/chainstate/003111.ldb
134bitcoind 2862 root  mem       REG                8,3   2178143  99877034 /home/bitcoin/testnet3/chainstate/018384.ldb
135bitcoind 2862 root  mem       REG                8,3   2184015  99891294 /home/bitcoin/testnet3/chainstate/003231.ldb
136bitcoind 2862 root  mem       REG                8,3   2179576  99891525 /home/bitcoin/testnet3/chainstate/010407.ldb
137bitcoind 2862 root  mem       REG                8,3   2180078  99891428 /home/bitcoin/testnet3/chainstate/010238.ldb
138bitcoind 2862 root  mem       REG                8,3   2176392  99891794 /home/bitcoin/testnet3/chainstate/003132.ldb
139bitcoind 2862 root  mem       REG                8,3   2174950  99876884 /home/bitcoin/testnet3/chainstate/018339.ldb
140bitcoind 2862 root  mem       REG                8,3   2181444  99891806 /home/bitcoin/testnet3/chainstate/003218.ldb
141bitcoind 2862 root  mem       REG                8,3   2178471  99877134 /home/bitcoin/testnet3/chainstate/018365.ldb
142bitcoind 2862 root  mem       REG                8,3   2181278  99877191 /home/bitcoin/testnet3/chainstate/018453.ldb
143bitcoind 2862 root  mem       REG                8,3   2177401  99877237 /home/bitcoin/testnet3/chainstate/018499.ldb
144bitcoind 2862 root  mem       REG                8,3   2178140  99891785 /home/bitcoin/testnet3/chainstate/003129.ldb
145bitcoind 2862 root  mem       REG                8,3   2173424  99877107 /home/bitcoin/testnet3/chainstate/018387.ldb
146bitcoind 2862 root  mem       REG                8,3   2176727  99891495 /home/bitcoin/testnet3/chainstate/010406.ldb
147bitcoind 2862 root  mem       REG                8,3   2180570  99890709 /home/bitcoin/testnet3/chainstate/009669.ldb
148bitcoind 2862 root  mem       REG                8,3   2182058  99891385 /home/bitcoin/testnet3/chainstate/010398.ldb
149bitcoind 2862 root  mem       REG                8,3   2181779  99890529 /home/bitcoin/testnet3/chainstate/003106.ldb
150bitcoind 2862 root  mem       REG                8,3   2177996  99877026 /home/bitcoin/testnet3/chainstate/018381.ldb
151bitcoind 2862 root  mem       REG                8,3   2179807  99877085 /home/bitcoin/testnet3/chainstate/018394.ldb
152bitcoind 2862 root  mem       REG                8,3   2180720  99891824 /home/bitcoin/testnet3/chainstate/009789.ldb
153bitcoind 2862 root  mem       REG                8,3   2179210  99891363 /home/bitcoin/testnet3/chainstate/009529.ldb
154bitcoind 2862 root  mem       REG                8,3   2177479  99891136 /home/bitcoin/testnet3/chainstate/009637.ldb
155bitcoind 2862 root  mem       REG                8,3   2177184  99877182 /home/bitcoin/testnet3/chainstate/018450.ldb
156bitcoind 2862 root  mem       REG                8,3   2177910  99891323 /home/bitcoin/testnet3/chainstate/009642.ldb
157bitcoind 2862 root  mem       REG                8,3   2181534  99892068 /home/bitcoin/testnet3/chainstate/009791.ldb
158bitcoind 2862 root  mem       REG                8,3   2176418  99891442 /home/bitcoin/testnet3/chainstate/009535.ldb
159bitcoind 2862 root  mem       REG                8,3   2182250  99891317 /home/bitcoin/testnet3/chainstate/009766.ldb
160bitcoind 2862 root  mem       REG                8,3   2177254  99891071 /home/bitcoin/testnet3/chainstate/009576.ldb
161bitcoind 2862 root  mem       REG                8,3   2178975  99891289 /home/bitcoin/testnet3/chainstate/009527.ldb
162bitcoind 2862 root  mem       REG                8,3   2180133  99891395 /home/bitcoin/testnet3/chainstate/009532.ldb
163bitcoind 2862 root  mem       REG                8,3   2178033  99891393 /home/bitcoin/testnet3/chainstate/003297.ldb
164bitcoind 2862 root  mem       REG                8,3   2173533  99877009 /home/bitcoin/testnet3/chainstate/018346.ldb
165bitcoind 2862 root  mem       REG                8,3   2180343  99890995 /home/bitcoin/testnet3/chainstate/009781.ldb
166bitcoind 2862 root  mem       REG                8,3   2180057  99891444 /home/bitcoin/testnet3/chainstate/009617.ldb
167bitcoind 2862 root  mem       REG                8,3   2180022  99890470 /home/bitcoin/testnet3/chainstate/009634.ldb
168bitcoind 2862 root  mem       REG                8,3   2179722  99891520 /home/bitcoin/testnet3/chainstate/003313.ldb
169bitcoind 2862 root  mem       REG                8,3   1578408  99877344 /home/bitcoin/testnet3/blocks/index/021504.ldb
170bitcoind 2862 root  mem       REG                8,3   2176121  99889726 /home/bitcoin/testnet3/chainstate/010404.ldb
171bitcoind 2862 root  mem       REG                8,3   1120135  99877335 /home/bitcoin/testnet3/blocks/index/021503.ldb
172bitcoind 2862 root  mem       REG                8,3   2135889  99877324 /home/bitcoin/testnet3/blocks/index/021502.ldb
173bitcoind 2862 root  mem       REG                8,3   2135851  99877322 /home/bitcoin/testnet3/blocks/index/021501.ldb
174bitcoind 2862 root  mem       REG                8,3   2136050  99877321 /home/bitcoin/testnet3/blocks/index/021500.ldb
175bitcoind 2862 root  mem       REG                8,3   2136114  99877316 /home/bitcoin/testnet3/blocks/index/021499.ldb
176bitcoind 2862 root  mem       REG                8,3   2136041  99877313 /home/bitcoin/testnet3/blocks/index/021498.ldb
177bitcoind 2862 root  mem       REG                8,3   2135537  99876931 /home/bitcoin/testnet3/blocks/index/021497.ldb
178bitcoind 2862 root  mem       REG                8,3   2137271  99890379 /home/bitcoin/testnet3/blocks/index/006674.ldb
179bitcoind 2862 root  mem       REG                8,3   2139535  99890378 /home/bitcoin/testnet3/blocks/index/006673.ldb
180bitcoind 2862 root  mem       REG                8,3   2137524  99890377 /home/bitcoin/testnet3/blocks/index/006672.ldb
181bitcoind 2862 root  mem       REG                8,3   2135784  99890376 /home/bitcoin/testnet3/blocks/index/006671.ldb
182bitcoind 2862 root  mem       REG                8,3   2136821  99890375 /home/bitcoin/testnet3/blocks/index/006670.ldb
183bitcoind 2862 root  mem       REG                8,3   2136765  99890374 /home/bitcoin/testnet3/blocks/index/006669.ldb
184bitcoind 2862 root  mem       REG                8,3   2138046  99890373 /home/bitcoin/testnet3/blocks/index/006668.ldb
185bitcoind 2862 root  mem       REG                8,3    498092  99877353 /home/bitcoin/testnet3/chainstate/019082.ldb
186bitcoind 2862 root  mem       REG                8,3    605726  99877351 /home/bitcoin/testnet3/chainstate/019080.ldb
187bitcoind 2862 root  mem       REG                8,3    629112  99877350 /home/bitcoin/testnet3/chainstate/019079.ldb
188bitcoind 2862 root  mem       REG                8,3   2135945  99890372 /home/bitcoin/testnet3/blocks/index/006667.ldb
189bitcoind 2862 root  mem       REG                8,3   2136313  99890371 /home/bitcoin/testnet3/blocks/index/006666.ldb
190bitcoind 2862 root  mem       REG                8,3    559747  99877349 /home/bitcoin/testnet3/chainstate/019078.ldb
191bitcoind 2862 root  mem       REG                8,3    443670  99877348 /home/bitcoin/testnet3/chainstate/019077.ldb
192bitcoind 2862 root  mem       REG                8,3    558150  99877326 /home/bitcoin/testnet3/chainstate/019074.ldb
193bitcoind 2862 root  mem       REG                8,3    494770  99877289 /home/bitcoin/testnet3/chainstate/019068.ldb
194bitcoind 2862 root  mem       REG                8,3   2135877  99890365 /home/bitcoin/testnet3/blocks/index/006665.ldb
195bitcoind 2862 root  mem       REG                8,3    561643  99877284 /home/bitcoin/testnet3/chainstate/019066.ldb
196bitcoind 2862 root  mem       REG                8,3    601259  99877275 /home/bitcoin/testnet3/chainstate/019062.ldb
197bitcoind 2862 root  mem       REG                8,3    667092  99877274 /home/bitcoin/testnet3/chainstate/019061.ldb
198bitcoind 2862 root  mem       REG                8,3   2137878  99890313 /home/bitcoin/testnet3/blocks/index/006664.ldb
199bitcoind 2862 root  mem       REG                8,3   2137619  99890256 /home/bitcoin/testnet3/blocks/index/006663.ldb
200bitcoind 2862 root  mem       REG                8,3   2135200  99890964 /home/bitcoin/testnet3/blocks/index/006709.ldb
201bitcoind 2862 root  mem       REG                8,3   2136640  99890249 /home/bitcoin/testnet3/blocks/index/006662.ldb
202bitcoind 2862 root  mem       REG                8,3   2138864  99890963 /home/bitcoin/testnet3/blocks/index/006708.ldb
203bitcoind 2862 root  mem       REG                8,3   2137613  99890248 /home/bitcoin/testnet3/blocks/index/006661.ldb
204bitcoind 2862 root  mem       REG                8,3   2137101  99890247 /home/bitcoin/testnet3/blocks/index/006660.ldb
205bitcoind 2862 root  mem       REG                8,3   2136169  99889599 /home/bitcoin/testnet3/blocks/index/006580.ldb
206bitcoind 2862 root  mem       REG                8,3   2137039  99890245 /home/bitcoin/testnet3/blocks/index/006659.ldb
207bitcoind 2862 root  mem       REG                8,3   2135933  99890243 /home/bitcoin/testnet3/blocks/index/006658.ldb
208bitcoind 2862 root  mem       REG                8,3   2135700  99891216 /home/bitcoin/testnet3/blocks/index/000217.ldb
209bitcoind 2862 root  mem       REG                8,3   2136490  99890238 /home/bitcoin/testnet3/blocks/index/006657.ldb
210bitcoind 2862 root  mem       REG                8,3   2135904  99891380 /home/bitcoin/testnet3/blocks/index/000216.ldb
211bitcoind 2862 root  mem       REG                8,3   2137118  99890228 /home/bitcoin/testnet3/blocks/index/006656.ldb
212bitcoind 2862 root  mem       REG                8,3   2137139  99890221 /home/bitcoin/testnet3/blocks/index/006655.ldb
213bitcoind 2862 root  mem       REG                8,3   2136018  99891284 /home/bitcoin/testnet3/blocks/index/000215.ldb
214bitcoind 2862 root  mem       REG                8,3   2136959  99890215 /home/bitcoin/testnet3/blocks/index/006654.ldb
215bitcoind 2862 root  mem       REG                8,3   2138516  99890214 /home/bitcoin/testnet3/blocks/index/006653.ldb
216bitcoind 2862 root  mem       REG                8,3   2136307  99891164 /home/bitcoin/testnet3/blocks/index/000214.ldb
217bitcoind 2862 root  mem       REG                8,3   2136791  99890201 /home/bitcoin/testnet3/blocks/index/006652.ldb
218bitcoind 2862 root  mem       REG                8,3   2135915  99891467 /home/bitcoin/testnet3/blocks/index/000213.ldb
219bitcoind 2862 root  mem       REG                8,3   2136885  99890200 /home/bitcoin/testnet3/blocks/index/006651.ldb
220bitcoind 2862 root  mem       REG                8,3   2136738  99890194 /home/bitcoin/testnet3/blocks/index/006650.ldb
221bitcoind 2862 root  mem       REG                8,3   2135963  99891252 /home/bitcoin/testnet3/blocks/index/000212.ldb
222bitcoind 2862 root  mem       REG                8,3   2136683  99890187 /home/bitcoin/testnet3/blocks/index/006649.ldb
223bitcoind 2862 root  mem       REG                8,3   2136826  99890185 /home/bitcoin/testnet3/blocks/index/006648.ldb
224bitcoind 2862 root  mem       REG                8,3   2136199  99891447 /home/bitcoin/testnet3/blocks/index/000211.ldb
225bitcoind 2862 root  mem       REG                8,3   2137974  99890144 /home/bitcoin/testnet3/blocks/index/006647.ldb
226bitcoind 2862 root  mem       REG                8,3   2136211  99891420 /home/bitcoin/testnet3/blocks/index/000210.ldb
227bitcoind 2862 root  mem       REG                8,3   2138113  99890125 /home/bitcoin/testnet3/blocks/index/006646.ldb
228bitcoind 2862 root  mem       REG                8,3   2136442  99889997 /home/bitcoin/testnet3/blocks/index/006645.ldb
229bitcoind 2862 root  mem       REG                8,3   2135951  99891227 /home/bitcoin/testnet3/blocks/index/000209.ldb
230bitcoind 2862 root  mem       REG                8,3   2136821  99889976 /home/bitcoin/testnet3/blocks/index/006644.ldb
231bitcoind 2862 root  mem       REG                8,3   2136714  99889975 /home/bitcoin/testnet3/blocks/index/006643.ldb
232bitcoind 2862 root  mem       REG                8,3   2136053  99877309 /home/bitcoin/testnet3/blocks/index/021496.ldb
233bitcoind 2862 root  mem       REG                8,3   2136096  99891275 /home/bitcoin/testnet3/blocks/index/000208.ldb
234bitcoind 2862 root  mem       REG                8,3   2136037  99877308 /home/bitcoin/testnet3/blocks/index/021495.ldb
235bitcoind 2862 root  mem       REG                8,3   2136186  99877305 /home/bitcoin/testnet3/blocks/index/021494.ldb
236bitcoind 2862 root  mem       REG                8,3   2137831  99889973 /home/bitcoin/testnet3/blocks/index/006642.ldb
237bitcoind 2862 root  mem       REG                8,3   2135819  99877301 /home/bitcoin/testnet3/blocks/index/021492.ldb
238bitcoind 2862 root  mem       REG                8,3   2136108  99877300 /home/bitcoin/testnet3/blocks/index/021491.ldb
239bitcoind 2862 root  mem       REG                8,3   2135969  99891281 /home/bitcoin/testnet3/blocks/index/000106.ldb
240bitcoind 2862 root  mem       REG                8,3   2136108  99889972 /home/bitcoin/testnet3/blocks/index/006641.ldb
241bitcoind 2862 root  mem       REG                8,3   2136250  99877299 /home/bitcoin/testnet3/blocks/index/021490.ldb
242bitcoind 2862 root  mem       REG                8,3   2135964  99877286 /home/bitcoin/testnet3/blocks/index/021489.ldb
243bitcoind 2862 root  mem       REG                8,3   2136340  99877285 /home/bitcoin/testnet3/blocks/index/021488.ldb
244bitcoind 2862 root  mem       REG                8,3   2136310  99889971 /home/bitcoin/testnet3/blocks/index/006640.ldb
245bitcoind 2862 root  mem       REG                8,3   2135922  99877273 /home/bitcoin/testnet3/blocks/index/021487.ldb
246bitcoind 2862 root  mem       REG                8,3    215722  99890481 /home/bitcoin/testnet3/blocks/index/006675.ldb
247bitcoind 2862 root  mem       REG                8,3   1446810  99877303 /home/bitcoin/testnet3/blocks/index/021493.ldb
248bitcoind 2862 root  mem       REG                8,3   1071297  99877269 /home/bitcoin/testnet3/blocks/index/021486.ldb
249bitcoind 2862 root  mem       REG                8,3   2136043  99891174 /home/bitcoin/testnet3/blocks/index/000105.ldb
250bitcoind 2862 root  mem       REG                8,3   2136904  99889970 /home/bitcoin/testnet3/blocks/index/006639.ldb
251bitcoind 2862 root  mem       REG                8,3   2142388  99877258 /home/bitcoin/testnet3/blocks/index/021483.ldb
252bitcoind 2862 root  mem       REG                8,3   1545966  99877257 /home/bitcoin/testnet3/blocks/index/021482.ldb
253bitcoind 2862 root  mem       REG                8,2     22928    917083 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
254bitcoind 2862 root  mem       REG                8,3   2137078  99889969 /home/bitcoin/testnet3/blocks/index/006638.ldb
255bitcoind 2862 root  mem       REG                8,3   1747373  99891228 /home/bitcoin/testnet3/blocks/index/000104.ldb
256bitcoind 2862 root  mem       REG                8,3    649470  99877266 /home/bitcoin/testnet3/chainstate/019059.ldb
257bitcoind 2862 root  mem       REG                8,3   2140557  99877255 /home/bitcoin/testnet3/blocks/index/021481.ldb
258bitcoind 2862 root  mem       REG                8,3   1728515  99876973 /home/bitcoin/testnet3/blocks/index/021480.ldb
259bitcoind 2862 root  mem       REG                8,3   2142815  99876932 /home/bitcoin/testnet3/blocks/index/021479.ldb
260bitcoind 2862 root  mem       REG                8,3   2135893  99889968 /home/bitcoin/testnet3/blocks/index/006637.ldb
261bitcoind 2862 root  mem       REG                8,3    579566  99877267 /home/bitcoin/testnet3/chainstate/019060.ldb
262bitcoind 2862 root  mem       REG                8,3    548580  99877259 /home/bitcoin/testnet3/chainstate/019057.ldb
263bitcoind 2862 root  mem       REG                8,3   2135394  99891347 /home/bitcoin/testnet3/blocks/index/000103.ldb
264bitcoind 2862 root  mem       REG                8,3   2136837  99889957 /home/bitcoin/testnet3/blocks/index/006636.ldb
265bitcoind 2862 root  mem       REG                8,3   2137290  99889956 /home/bitcoin/testnet3/blocks/index/006635.ldb
266bitcoind 2862 root  mem       REG                8,3   2135457  99891282 /home/bitcoin/testnet3/blocks/index/000102.ldb
267bitcoind 2862 root  mem       REG                8,3   2136564  99889949 /home/bitcoin/testnet3/blocks/index/006634.ldb
268bitcoind 2862 root  mem       REG                8,3   2138394  99889947 /home/bitcoin/testnet3/blocks/index/006633.ldb
269bitcoind 2862 root  mem       REG                8,3   2136030  99891441 /home/bitcoin/testnet3/blocks/index/000100.ldb
270bitcoind 2862 root  mem       REG                8,3   2136735  99889945 /home/bitcoin/testnet3/blocks/index/006632.ldb
271bitcoind 2862 root  mem       REG                8,3   2137542  99889943 /home/bitcoin/testnet3/blocks/index/006631.ldb
272bitcoind 2862 root  mem       REG                8,3   2135721  99891449 /home/bitcoin/testnet3/blocks/index/000099.ldb
273bitcoind 2862 root  mem       REG                8,3   2136354  99889932 /home/bitcoin/testnet3/blocks/index/006630.ldb
274bitcoind 2862 root  mem       REG                8,3   2135690  99891093 /home/bitcoin/testnet3/blocks/index/000098.ldb
275bitcoind 2862 root  mem       REG                8,3    586886  99877254 /home/bitcoin/testnet3/chainstate/019056.ldb
276bitcoind 2862 root  mem       REG                8,3    615990  99877253 /home/bitcoin/testnet3/chainstate/019055.ldb
277bitcoind 2862 root  mem       REG                8,3    571455  99877250 /home/bitcoin/testnet3/chainstate/019053.ldb
278bitcoind 2862 root  mem       REG                8,3   2135703  99889931 /home/bitcoin/testnet3/blocks/index/006629.ldb
279bitcoind 2862 root  mem       REG                8,3   2137312  99889927 /home/bitcoin/testnet3/blocks/index/006628.ldb
280bitcoind 2862 root  mem       REG                8,3   2135304  99891163 /home/bitcoin/testnet3/blocks/index/000096.ldb
281bitcoind 2862 root  mem       REG                8,3   2137016  99889924 /home/bitcoin/testnet3/blocks/index/006627.ldb
282bitcoind 2862 root  mem       REG                8,3   2136814  99889923 /home/bitcoin/testnet3/blocks/index/006626.ldb
283bitcoind 2862 root  mem       REG                8,3   2135902  99891171 /home/bitcoin/testnet3/blocks/index/000095.ldb
284bitcoind 2862 root  mem       REG                8,3   2136330  99889921 /home/bitcoin/testnet3/blocks/index/006625.ldb
285bitcoind 2862 root  mem       REG                8,3   2137377  99889920 /home/bitcoin/testnet3/blocks/index/006624.ldb
286bitcoind 2862 root  mem       REG                8,3   2135655  99891167 /home/bitcoin/testnet3/blocks/index/000094.ldb
287bitcoind 2862 root  mem       REG                8,3   2135930  99889918 /home/bitcoin/testnet3/blocks/index/006623.ldb
288bitcoind 2862 root  mem       REG                8,3   2135798  99891269 /home/bitcoin/testnet3/blocks/index/000092.ldb
289bitcoind 2862 root  mem       REG                8,3   2136814  99889916 /home/bitcoin/testnet3/blocks/index/006622.ldb
290bitcoind 2862 root  mem       REG                8,3    504105  99877265 /home/bitcoin/testnet3/chainstate/019058.ldb
291bitcoind 2862 root  mem       REG                8,3    588520  99877249 /home/bitcoin/testnet3/chainstate/019052.ldb
292bitcoind 2862 root  mem       REG                8,3    606325  99877246 /home/bitcoin/testnet3/chainstate/019051.ldb
293bitcoind 2862 root  mem       REG                8,3   2137399  99889912 /home/bitcoin/testnet3/blocks/index/006621.ldb
294bitcoind 2862 root  mem       REG                8,3   2135504  99891469 /home/bitcoin/testnet3/blocks/index/000091.ldb
295bitcoind 2862 root  mem       REG                8,3   2136501  99889909 /home/bitcoin/testnet3/blocks/index/006620.ldb
296bitcoind 2862 root  mem       REG                8,3   2137438  99889906 /home/bitcoin/testnet3/blocks/index/006619.ldb
297bitcoind 2862 root  mem       REG                8,3   2135912  99891280 /home/bitcoin/testnet3/blocks/index/000090.ldb
298bitcoind 2862 root  mem       REG                8,3   2137631  99889900 /home/bitcoin/testnet3/blocks/index/006618.ldb
299bitcoind 2862 root  mem       REG                8,3   2136659  99889892 /home/bitcoin/testnet3/blocks/index/006617.ldb
300bitcoind 2862 root  mem       REG                8,3   2135525  99891070 /home/bitcoin/testnet3/blocks/index/000088.ldb
301bitcoind 2862 root  mem       REG                8,3   2136814  99889865 /home/bitcoin/testnet3/blocks/index/006616.ldb
302bitcoind 2862 root  mem       REG                8,3   2136363  99889861 /home/bitcoin/testnet3/blocks/index/006615.ldb
303bitcoind 2862 root  mem       REG                8,3   2135776  99891339 /home/bitcoin/testnet3/blocks/index/000087.ldb
304bitcoind 2862 root  mem       REG                8,3   2136792  99889852 /home/bitcoin/testnet3/blocks/index/006614.ldb
305bitcoind 2862 root  mem       REG                8,3   2135986  99891445 /home/bitcoin/testnet3/blocks/index/000086.ldb
306bitcoind 2862 root  mem       REG                8,3   2135870  99889833 /home/bitcoin/testnet3/blocks/index/006613.ldb
307bitcoind 2862 root  mem       REG                8,3   2136296  99889808 /home/bitcoin/testnet3/blocks/index/006612.ldb
308bitcoind 2862 root  mem       REG                8,3   1742483  99891041 /home/bitcoin/testnet3/blocks/index/000085.ldb
309bitcoind 2862 root  mem       REG                8,3   2136917  99889800 /home/bitcoin/testnet3/blocks/index/006611.ldb
310bitcoind 2862 root  mem       REG                8,3   1458075  99892142 /home/bitcoin/testnet3/blocks/index/007259.ldb
311bitcoind 2862 root  mem       REG                8,3   2137012  99889795 /home/bitcoin/testnet3/blocks/index/006610.ldb
312bitcoind 2862 root  mem       REG                8,3   2136739  99889793 /home/bitcoin/testnet3/blocks/index/006609.ldb
313bitcoind 2862 root  mem       REG                8,3    785364  99877208 /home/bitcoin/testnet3/chainstate/019050.ldb
314bitcoind 2862 root  mem       REG                8,3    626559  99877204 /home/bitcoin/testnet3/chainstate/019049.ldb
315bitcoind 2862 root  mem       REG                8,3    619293  99877202 /home/bitcoin/testnet3/chainstate/019048.ldb
316bitcoind 2862 root  mem       REG                8,3   2136704  99889790 /home/bitcoin/testnet3/blocks/index/006608.ldb
317bitcoind 2862 root  mem       REG                8,3    457077  99877280 /home/bitcoin/testnet3/chainstate/019063.ldb
318bitcoind 2862 root  mem       REG                8,3    484919  99877201 /home/bitcoin/testnet3/chainstate/019047.ldb
319bitcoind 2862 root  mem       REG                8,3    666410  99877198 /home/bitcoin/testnet3/chainstate/019046.ldb
320bitcoind 2862 root  mem       REG                8,3    478433  99877137 /home/bitcoin/testnet3/chainstate/019045.ldb
321bitcoind 2862 root  mem       REG                8,3   2136658  99889787 /home/bitcoin/testnet3/blocks/index/006607.ldb
322bitcoind 2862 root  mem       REG                8,3   2136759  99889786 /home/bitcoin/testnet3/blocks/index/006606.ldb
323bitcoind 2862 root  mem       REG                8,3    334035  99877352 /home/bitcoin/testnet3/chainstate/019081.ldb
324bitcoind 2862 root  mem       REG                8,3   2135957  99889781 /home/bitcoin/testnet3/blocks/index/006605.ldb
325bitcoind 2862 root  mem       REG                8,3   2137974  99889772 /home/bitcoin/testnet3/blocks/index/006604.ldb
326bitcoind 2862 root  mem       REG                8,3     49828  99877347 /home/bitcoin/testnet3/chainstate/019076.ldb
327bitcoind 2862 root  mem       REG                8,3     37754  99877346 /home/bitcoin/testnet3/chainstate/019075.ldb
328bitcoind 2862 root  mem       REG                8,3     40612  99877325 /home/bitcoin/testnet3/chainstate/019073.ldb
329bitcoind 2862 root  mem       REG                8,3     42384  99877314 /home/bitcoin/testnet3/chainstate/019072.ldb
330bitcoind 2862 root  mem       REG                8,3     41863  99877294 /home/bitcoin/testnet3/chainstate/019071.ldb
331bitcoind 2862 root  mem       REG                8,3     39658  99877293 /home/bitcoin/testnet3/chainstate/019070.ldb
332bitcoind 2862 root  mem       REG                8,3   1258343  99877136 /home/bitcoin/testnet3/chainstate/018401.ldb
333bitcoind 2862 root  mem       REG                8,3   2136515  99889771 /home/bitcoin/testnet3/blocks/index/006603.ldb
334bitcoind 2862 root  mem       REG                8,3   2136656  99889770 /home/bitcoin/testnet3/blocks/index/006602.ldb
335bitcoind 2862 root  mem       REG                8,3     30143  99877376 /home/bitcoin/testnet3/chainstate/019085.ldb
336bitcoind 2862 root  mem       REG                8,3     43866  99877291 /home/bitcoin/testnet3/chainstate/019069.ldb
337bitcoind 2862 root  mem       REG                8,3     38250  99877288 /home/bitcoin/testnet3/chainstate/019067.ldb
338bitcoind 2862 root  mem       REG                8,3   2137564  99889765 /home/bitcoin/testnet3/blocks/index/006601.ldb
339bitcoind 2862 root  mem       REG                8,3     43770  99877283 /home/bitcoin/testnet3/chainstate/019065.ldb
340bitcoind 2862 root  mem       REG                8,3    202673  99891517 /home/bitcoin/testnet3/chainstate/010403.ldb
341bitcoind 2862 root  mem       REG                8,3   2139610  99889764 /home/bitcoin/testnet3/blocks/index/006600.ldb
342bitcoind 2862 root  mem       REG                8,3     42660  99877251 /home/bitcoin/testnet3/chainstate/019054.ldb
343bitcoind 2862 root  mem       REG                8,3    761005  99877139 /home/bitcoin/testnet3/chainstate/018402.ldb
344bitcoind 2862 root  mem       REG                8,3   2137226  99889749 /home/bitcoin/testnet3/blocks/index/006599.ldb
345bitcoind 2862 root  mem       REG                8,3   2137010  99889746 /home/bitcoin/testnet3/blocks/index/006598.ldb
346bitcoind 2862 root  mem       REG                8,3     20720  99877005 /home/bitcoin/testnet3/indexes/txindex/036707.ldb
347bitcoind 2862 root  mem       REG                8,3     88154  99876999 /home/bitcoin/testnet3/chainstate/019044.ldb
348bitcoind 2862 root  mem       REG                8,3    293730  99891850 /home/bitcoin/testnet3/chainstate/009544.ldb
349bitcoind 2862 root  mem       REG                8,3   2136380  99889738 /home/bitcoin/testnet3/blocks/index/006597.ldb
350bitcoind 2862 root  mem       REG                8,3   2137195  99889734 /home/bitcoin/testnet3/blocks/index/006596.ldb
351bitcoind 2862 root  mem       REG                8,3    105558  99876950 /home/bitcoin/testnet3/chainstate/019043.ldb
352bitcoind 2862 root  mem       REG                8,3    125317  99876908 /home/bitcoin/testnet3/chainstate/019042.ldb
353bitcoind 2862 root  mem       REG                8,3    109595  99876905 /home/bitcoin/testnet3/chainstate/019041.ldb
354bitcoind 2862 root  mem       REG                8,3   1818268  99892162 /home/bitcoin/testnet3/blocks/index/007225.ldb
355bitcoind 2862 root  mem       REG                8,3   2136826  99892161 /home/bitcoin/testnet3/blocks/index/007224.ldb
356bitcoind 2862 root  mem       REG                8,3   2136691  99892160 /home/bitcoin/testnet3/blocks/index/007223.ldb
357bitcoind 2862 root  mem       REG                8,3   2136293  99892159 /home/bitcoin/testnet3/blocks/index/007222.ldb
358bitcoind 2862 root  mem       REG                8,3   2136486  99892158 /home/bitcoin/testnet3/blocks/index/007221.ldb
359bitcoind 2862 root  mem       REG                8,3   2136574  99892157 /home/bitcoin/testnet3/blocks/index/007220.ldb
360bitcoind 2862 root  mem       REG                8,3   2136766  99892156 /home/bitcoin/testnet3/blocks/index/007219.ldb
361bitcoind 2862 root  mem       REG                8,3   2137041  99892155 /home/bitcoin/testnet3/blocks/index/007218.ldb
362bitcoind 2862 root  mem       REG                8,3   2136735  99892154 /home/bitcoin/testnet3/blocks/index/007217.ldb
363bitcoind 2862 root  mem       REG                8,3   2136359  99891025 /home/bitcoin/testnet3/blocks/index/007216.ldb
364bitcoind 2862 root  mem       REG                8,3   2175576  99876926 /home/bitcoin/testnet3/chainstate/018400.ldb
365bitcoind 2862 root  mem       REG                8,3   2179424  99891509 /home/bitcoin/testnet3/chainstate/009673.ldb
366bitcoind 2862 root  mem       REG                8,3     56077  99877264 /home/bitcoin/testnet3/blocks/index/021484.ldb
367bitcoind 2862 root  mem       REG                8,3    292996  99892153 /home/bitcoin/testnet3/blocks/index/007215.ldb
368bitcoind 2862 root  mem       REG                8,3     13579  99877282 /home/bitcoin/testnet3/chainstate/019064.ldb
369bitcoind 2862 root  mem       REG                8,2    178192    155975 /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.9
370bitcoind 2862 root  mem       REG                8,2     14640    917058 /lib/x86_64-linux-gnu/libdl-2.24.so
371bitcoind 2862 root  mem       REG                8,2     31744    917205 /lib/x86_64-linux-gnu/librt-2.24.so
372bitcoind 2862 root  mem       REG                8,2   1689360    917089 /lib/x86_64-linux-gnu/libc-2.24.so
373bitcoind 2862 root  mem       REG                8,2    135440    917180 /lib/x86_64-linux-gnu/libpthread-2.24.so
374bitcoind 2862 root  mem       REG                8,2     92584    917073 /lib/x86_64-linux-gnu/libgcc_s.so.1
375bitcoind 2862 root  mem       REG                8,2   1063328    917218 /lib/x86_64-linux-gnu/libm-2.24.so
376bitcoind 2862 root  mem       REG                8,2   1566168    133375 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
377bitcoind 2862 root  mem       REG                8,2    293264    155966 /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9
378bitcoind 2862 root  mem       REG                8,2     10008    155987 /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.9
379bitcoind 2862 root  mem       REG                8,2   2686672    133428 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
380bitcoind 2862 root  mem       REG                8,2   1828312    151957 /usr/lib/libdb_cxx-4.8.so
381bitcoind 2862 root  mem       REG                8,2     27048    154823 /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0
382bitcoind 2862 root  mem       REG                8,2    163240    154831 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
383bitcoind 2862 root  mem       REG                8,2    101624    133358 /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0
384bitcoind 2862 root  mem       REG                8,2     14512    133848 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0
385bitcoind 2862 root  mem       REG                8,2    153288    917042 /lib/x86_64-linux-gnu/ld-2.24.so
386bitcoind 2862 root  mem       REG                8,3       671  99877378 /home/bitcoin/testnet3/chainstate/019087.ldb
387bitcoind 2862 root  mem       REG                8,3     35476  99892186 /home/bitcoin/testnet3/blocks/index/007313.ldb
388bitcoind 2862 root  mem       REG                8,3      3477  99892093 /home/bitcoin/testnet3/blocks/index/007345.ldb
389bitcoind 2862 root  mem       REG                8,3      3503  99892193 /home/bitcoin/testnet3/blocks/index/007408.ldb
390bitcoind 2862 root  mem       REG                8,3       443  99892112 /home/bitcoin/testnet3/blocks/index/007440.ldb
391bitcoind 2862 root  mem       REG                8,3     11730  99892125 /home/bitcoin/testnet3/blocks/index/007595.ldb
392bitcoind 2862 root  mem       REG                8,3     21719  99876980 /home/bitcoin/testnet3/blocks/index/007629.ldb
393bitcoind 2862 root  mem       REG                8,3      6512  99877102 /home/bitcoin/testnet3/blocks/index/009054.ldb
394bitcoind 2862 root  mem       REG                8,3       582  99877022 /home/bitcoin/testnet3/blocks/index/009087.ldb
395bitcoind 2862 root  mem       REG                8,3      2547  99877004 /home/bitcoin/testnet3/blocks/index/009120.ldb
396bitcoind 2862 root  mem       REG                8,3      2381  99877092 /home/bitcoin/testnet3/blocks/index/010235.ldb
397bitcoind 2862 root  mem       REG                8,3       445  99877125 /home/bitcoin/testnet3/blocks/index/010268.ldb
398bitcoind 2862 root  mem       REG                8,3       306  99877110 /home/bitcoin/testnet3/blocks/index/010454.ldb
399bitcoind 2862 root  mem       REG                8,3       306  99877112 /home/bitcoin/testnet3/blocks/index/010583.ldb
400bitcoind 2862 root  mem       REG                8,3       713  99877132 /home/bitcoin/testnet3/blocks/index/010800.ldb
401bitcoind 2862 root  mem       REG                8,3       308  99877131 /home/bitcoin/testnet3/blocks/index/011055.ldb
402bitcoind 2862 root  mem       REG                8,3       444  99877142 /home/bitcoin/testnet3/blocks/index/011685.ldb
403bitcoind 2862 root  mem       REG                8,3       306  99877127 /home/bitcoin/testnet3/blocks/index/014328.ldb
404bitcoind 2862 root  mem       REG                8,3       987  99877165 /home/bitcoin/testnet3/blocks/index/015657.ldb
405bitcoind 2862 root  mem       REG                8,3       306  99877138 /home/bitcoin/testnet3/blocks/index/015810.ldb
406bitcoind 2862 root  mem       REG                8,3       305  99877166 /home/bitcoin/testnet3/blocks/index/016175.ldb
407bitcoind 2862 root  mem       REG                8,3       306  99877170 /home/bitcoin/testnet3/blocks/index/016478.ldb
408bitcoind 2862 root  mem       REG                8,3       305  99877104 /home/bitcoin/testnet3/blocks/index/016785.ldb
409bitcoind 2862 root  mem       REG                8,3       306  99877176 /home/bitcoin/testnet3/blocks/index/017034.ldb
410bitcoind 2862 root  mem       REG                8,3       307  99877189 /home/bitcoin/testnet3/blocks/index/017474.ldb
411bitcoind 2862 root  mem       REG                8,3       307  99877164 /home/bitcoin/testnet3/blocks/index/017758.ldb
412bitcoind 2862 root  mem       REG                8,3       305  99877168 /home/bitcoin/testnet3/blocks/index/018042.ldb
413bitcoind 2862 root  mem       REG                8,3       304  99877178 /home/bitcoin/testnet3/blocks/index/019548.ldb
414bitcoind 2862 root  mem       REG                8,2   1679776    132753 /usr/lib/locale/locale-archive
415bitcoind 2862 root  mem       REG                8,3      2248  99877292 /home/bitcoin/testnet3/blocks/index/020202.ldb
416bitcoind 2862 root  mem       REG                8,3      2541  99877256 /home/bitcoin/testnet3/blocks/index/020269.ldb
417bitcoind 2862 root  mem       REG                8,3       584  99877323 /home/bitcoin/testnet3/blocks/index/020364.ldb
418bitcoind 2862 root  mem       REG                8,3       306  99877287 /home/bitcoin/testnet3/blocks/index/021118.ldb
419bitcoind 2862 root  mem       REG                8,3       307  99877268 /home/bitcoin/testnet3/blocks/index/021308.ldb
420bitcoind 2862 root  mem       REG                8,3       306  99877281 /home/bitcoin/testnet3/blocks/index/021341.ldb
421bitcoind 2862 root  mem       REG                8,2     26258    133578 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
422bitcoind 2862 root    0u      CHR                1,3       0t0      1029 /dev/null
423bitcoind 2862 root    1u      CHR                1,3       0t0      1029 /dev/null
424bitcoind 2862 root    2u      CHR                1,3       0t0      1029 /dev/null
425bitcoind 2862 root    3uW     REG                8,3         0  99877725 /home/bitcoin/testnet3/.lock
426bitcoind 2862 root    4w      REG                8,3  10471265  99879150 /home/bitcoin/testnet3/debug.log
427bitcoind 2862 root    5u  a_inode               0,10         0      2054 [eventpoll]
428bitcoind 2862 root    6u     unix 0xffff8848bca2c400       0t0 401418105 type=STREAM
429bitcoind 2862 root    7u     unix 0xffff8848bca2c800       0t0 401418106 type=STREAM
430bitcoind 2862 root    8u  a_inode               0,10         0      2054 [eventfd]
431bitcoind 2862 root    9u     IPv6          401418109       0t0       TCP localhost:18332 (LISTEN)
432bitcoind 2862 root   10u     IPv4          401418111       0t0       TCP localhost:18332 (LISTEN)
433bitcoind 2862 root   11uW     REG                8,3         0  99879963 /home/bitcoin/testnet3/blocks/index/LOCK
434bitcoind 2862 root   12w      REG                8,3   4073303  99877200 /home/bitcoin/testnet3/blocks/index/021477.log
435bitcoind 2862 root   13w      REG                8,3     19429  99877203 /home/bitcoin/testnet3/blocks/index/MANIFEST-021475
436bitcoind 2862 root   14u     IPv4          401423878       0t0       TCP kimsufi1:59326->158.140.212.157:18333 (ESTABLISHED)
437bitcoind 2862 root   15uW     REG                8,3         0  99889659 /home/bitcoin/testnet3/chainstate/LOCK
438bitcoind 2862 root   16w      REG                8,3       130  99877341 /home/bitcoin/testnet3/chainstate/019039.log
439bitcoind 2862 root   17w      REG                8,3     61661  99877342 /home/bitcoin/testnet3/chainstate/MANIFEST-019037
440bitcoind 2862 root   18uW     REG                8,3         0  99890497 /home/bitcoin/testnet3/indexes/txindex/LOCK
441bitcoind 2862 root   19w      REG                8,3     11582  99877021 /home/bitcoin/testnet3/indexes/txindex/036708.log
442bitcoind 2862 root   20w      REG                8,3    160759  99877108 /home/bitcoin/testnet3/indexes/txindex/MANIFEST-036706
443bitcoind 2862 root   21uW     REG                8,3         0  99890745 /home/bitcoin/testnet3/.walletlock
444bitcoind 2862 root   22w      REG                8,3         0  99879654 /home/bitcoin/testnet3/db.log
445bitcoind 2862 root   23u      REG                8,3   1630208  99889541 /home/bitcoin/testnet3/wallet.dat
446bitcoind 2862 root   24u      REG                8,3   1048576  99876927 /home/bitcoin/testnet3/database/log.0000000001
447bitcoind 2862 root   25u     IPv4          401423978       0t0       TCP kimsufi1:45314->47.96.188.26:18333 (ESTABLISHED)
448bitcoind 2862 root   26u  a_inode               0,10         0      2054 [eventpoll]
449bitcoind 2862 root   27u     unix 0xffff88485733f400       0t0 401421983 type=STREAM
450bitcoind 2862 root   28u     unix 0xffff88485733fc00       0t0 401421984 type=STREAM
451bitcoind 2862 root   29u  a_inode               0,10         0      2054 [eventfd]
452bitcoind 2862 root   30u     IPv6          401421986       0t0       TCP *:18333 (LISTEN)
453bitcoind 2862 root   31u     IPv4          401421987       0t0       TCP *:18333 (LISTEN)
454bitcoind 2862 root   32u     IPv4          401422908       0t0       TCP kimsufi1:37568->77.105.216.41:18333 (ESTABLISHED)
455bitcoind 2862 root   33u     IPv6          401426464       0t0       TCP [2001:41d0:a:f115::1]:39958->nl.redfox.bz:18333 (ESTABLISHED)
456bitcoind 2862 root   34u     IPv4          401422909       0t0       TCP kimsufi1:46234->ec2-34-229-185-163.compute-1.amazonaws.com:18333 (ESTABLISHED)
  27. fscemama commented at 11:25 pm on January 31, 2019: none

    So, you improved things with this branch. The wallet itself is completely released, and this does deserve to be committed. Now, I suspect the opening of these wallets needed blocks to be opened for rescanning of transactions, and that these blocks have not been released. This is going to be a more difficult patch now to release blocks needed by an unloaded wallet, and not by other wallets that may be loaded.

    Thanks a lor for being patient.

  28. HashUnlimited commented at 9:44 pm on February 1, 2019: contributor

    this log reads too familiar for not attracting my attention….

    for the basics (and I suppose that’s what @sipa was after in his comment) unix based operating systems typically keep the contents of the memory until requested by another call. that’s what’s giving us the look-and-feel we are used to.

    so fighting the symptoms like it appears here might not be the best solution.

    so why familiar? we were looking at similar lines few months back and started tracing the issue. out of laziness we didn’t finish the trace as we couldn’t reproduce it on any other kernel/compiler combination than ours.

    findings on troubleshooting: while configuring our compiler, we’re asking it if it understands and can handle certain thingies like position independent code (PIC). typically, we will receive a “yes we can” answer. in the previous IF clause, the script decided to let the compiler apply the flags on demand by getting them passed through CXXFLAGS. unless we’re on windows. ironically, gcc decides against PIC, at least on the crypto libraries, we didn’t explicitly test against all (but bench would be a hot candidate)

  29. fscemama commented at 9:07 pm on February 2, 2019: none
    I think that after the #15297 hack, Bitcoin Core will be more stable when used in multiwallets context. I mean: the more memory you have, the more different wallets you should be able to use, given you load/unload without never reaching the ~300 simultaneous loaded wallets (unless you tweak BerlekeyDB settings and kernel semaphores, supposing this would work). Please, commit this to next official release, and I’ll give you a real-world, production-context feedback. @HashUnlimited : thank you for this analysis. I’ve always thought Bitcoin Core had been thought for a one-wallet use, and that BerkeleyDB had been chosen for easier cross-platform porting. I’m dreaming of a Bitcoin Core that would use an instance of Postgresql rather than BerkeleyDB, on a bitcoind setting. Why not after all.
  30. meshcollider closed this on Feb 5, 2019
  31. meshcollider referenced this in commit 30e799a5f7 on Feb 5, 2019
  32. linuxsh2 referenced this in commit 38b7c6cc7f on Jul 30, 2021
  33. linuxsh2 referenced this in commit 7195f7cdf7 on Aug 3, 2021
  34. DrahtBot locked this on Dec 16, 2021
  35. gades referenced this in commit 562d8894d3 on May 22, 2022


fscemama promag sipa HashUnlimited

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-04 22:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me