0.11.x Bitcoin-qt disk thrashing, nonresponsive 32 bit Win7 #7001

issue dthorpe openend this issue on November 13, 2015
  1. dthorpe commented at 2:18 am on November 13, 2015: none

    Bitcoin-qt v0.11.1 and now v0.11.2rc1 32 bit Windows, running on Win7 with 2GB ram.

    At app startup, the splash screen runs through verifying and rescanning status messages fairly quickly, then gets “stuck” showing “Activating best chain” message. The disk is thrashing. Process memory use is under 300MB, total system memory use is 1.5GB out of 2GB available. CPU load (2 cores) bounces between 10 and 60%, but not pegged at 100%. Disk use (as reported by Task Manage / Resource Manager) shows about 1MByte/sec read, 3MByte/sec write, with a disk queue length of around 2.5.

    After 1 to 2 HOURS, bitcoin-qt gets past the “activating best chain” message and opens the full UI window. The blockchain on this machine is 8 weeks out of date.

    While sync’ing to the network, the disk is still thrashing heavily and memory use has not increased above 300MB. The peer list shows it is attempting to connect to peers to ask for blocks, but most of the time the Ping Time shows as N/A and there is no user agent or other handshake info shown. The network use graph shows an occasional blip separated by 5 to 10 minutes of total network silence.

    Typing “help” in the Debug console window shows no response for nearly 30 seconds.

    The disk thrashing doesn’t appear to be caused by low memory page faults, because there’s 500MB of free system memory available.

    If I shut down the app (normal exit, not kill), wait for it to shut down, and then start it up again, it sits on the “activating best chain” message again for an hour or more.

    Is the v0.11.x build running a background thread to update the database and indices on disk? When this machine was last fired up on main net, it was running Bitcoinqt v0.10.x.

    It’s doing a heckovalot of disk I/O with absolutely nothing coming in from the peer network. Bitcoinqt is trying to connect to peers, but apparently can rarely can make it through the handshake before giving up and starting over. This isn’t a network or firewall/router/ISP saturation issue because this machine can’t even manage a connection to another 0.11.0 peer on the local network which has an up-to-date blockchain. I was thinking I could get the machine “caught up” in a few minutes by -connect(ing) it to the up to date machine over the 1GB local ethernet, but it’s barely able to connect at all.

    Note that this machine normally runs bitcoinqt on testnet, and has shown none of this behavior. However, it’s kept up to date, not 8 weeks behind there.

    Any idea what’s going on? Why is 0.11.x thrashing the disk so hard, even without incoming txs or blocks?

  2. NicolasDorier commented at 7:13 am on November 13, 2015: contributor
    can you try to run with “-checklevel 1” ?
  3. laanwj commented at 7:55 am on November 13, 2015: member
    Looks like a duplicate of #6784, #5664. Anything appearing in your debug.log when this happens? I expect it is “UpdateTip” messages while it is validating blocks and catching up. This indeed generates a lot of i/o while reading and updating the UTXO database. My hypothesis is that is results in contention on the cs_main lock, which can make the process otherwise unresponsive.
  4. sipa commented at 8:06 am on November 13, 2015: member

    The “Activating best chain” loop runs before normal startup, maybe proper shutdown handlers aren’t installed yet? Otherwise, it is pretty much permanently holding cs_main, only giving it up very briefly in between two blocks (in main.cpp:ActivateBestChain()).

    With default settings are and a system that has relatively slow I/O, this likely is slow. Increasing -dbcache would speed it up significantly.

  5. laanwj commented at 8:56 am on November 13, 2015: member
    And “Activating best chain” can take a while, sure. It’s processing the blocks that it downloaded before you exited the client last time, but haven’t been validated yet (right?).
  6. dthorpe commented at 10:23 pm on November 13, 2015: none

    Debug.log snippet:

    This shows the app is stuck at “Activating best chain” for almost exactly 1 hour.

      02015-11-13 01:31:49 Bitcoin version v0.11.2rc1 (2015-11-08 00:56:01 +0100)
      12015-11-13 01:31:49 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
      22015-11-13 01:31:49 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
      32015-11-13 01:31:49 Default data directory C:\Users\Owner\AppData\Roaming\Bitcoin
      42015-11-13 01:31:49 Using data directory C:\Users\Owner\AppData\Roaming\Bitcoin
      52015-11-13 01:31:49 Using config file C:\Users\Owner\AppData\Roaming\Bitcoin\bitcoin.conf
      62015-11-13 01:31:49 Using at most 25 connections (2048 file descriptors available)
      72015-11-13 01:31:49 Using 2 threads for script verification
      82015-11-13 01:31:49 scheduler thread start
      92015-11-13 01:31:49 Binding RPC on address :: port 8332 (IPv4+IPv6 bind any: 1)
     102015-11-13 01:31:49 Using wallet wallet.dat
     112015-11-13 01:31:49 init message: Verifying wallet...
     122015-11-13 01:31:49 CDBEnv::Open: LogDir=C:\Users\Owner\AppData\Roaming\Bitcoin\database ErrorFile=C:\Users\Owner\AppData\Roaming\Bitcoin\db.log
     132015-11-13 01:31:49 Bound to [::]:8333
     142015-11-13 01:31:49 Bound to 0.0.0.0:8333
     152015-11-13 01:31:49 Cache configuration:
     162015-11-13 01:31:49 * Using 2.0MiB for block index database
     172015-11-13 01:31:49 * Using 32.5MiB for chain state database
     182015-11-13 01:31:49 * Using 65.5MiB for in-memory UTXO set
     192015-11-13 01:31:49 init message: Loading block index...
     202015-11-13 01:31:49 Opening LevelDB in C:\Users\Owner\AppData\Roaming\Bitcoin\blocks\index
     212015-11-13 01:31:50 Opened LevelDB successfully
     222015-11-13 01:31:50 Opening LevelDB in C:\Users\Owner\AppData\Roaming\Bitcoin\chainstate
     232015-11-13 01:31:53 Opened LevelDB successfully
     242015-11-13 01:32:05 LoadBlockIndexDB: last block file = 338
     252015-11-13 01:32:05 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=68, size=49953223, heights=374057...374824, time=2015-09-11...2015-09-16)
     262015-11-13 01:32:05 Checking all blk files are present...
     272015-11-13 01:32:05 LoadBlockIndexDB: transaction index disabled
     282015-11-13 01:32:05 LoadBlockIndexDB: hashBestChain=00000000000000000665a2d4bf94a6169c33c9566fbd16d5b40a103a5c1b3806 height=374373 date=2015-09-13 19:56:30 progress=0.937629
     292015-11-13 01:32:05 init message: Verifying blocks...
     302015-11-13 01:32:05 Verifying last 288 blocks at level 3
     312015-11-13 01:32:31 No coin database inconsistencies in last 5 blocks (3717 transactions)
     322015-11-13 01:32:31  block index           41478ms
     332015-11-13 01:32:31 init message: Loading wallet...
     342015-11-13 01:32:31 nFileVersion = 110200
     352015-11-13 01:32:31 Keys: 102 plaintext, 0 encrypted, 1 w/ metadata, 102 total
     362015-11-13 01:32:31  wallet                  762ms
     372015-11-13 01:32:32 init message: Rescanning...
     382015-11-13 01:32:32 Rescanning last 77 blocks (from block 374296)...
     392015-11-13 01:32:34  rescan                 2745ms
     40==>2015-11-13 01:32:34 init message: Activating best chain...
     412015-11-13 01:32:42 UpdateTip: new best=00000000000000000b4c7d7194c7a26d70ac5eb8ec9b5c0ab673a4c6998b54ff  height=374374  log2_work=83.342045  tx=83565622  date=2015-09-13 19:58:09 progress=0.937630  cache=3.1MiB(1059tx)
     422015-11-13 01:33:17 UpdateTip: new best=00000000000000000d193d75ab78813db837c605288e9a0dc3beb9ae2ec4b280  height=374375  log2_work=83.342074  tx=83567755  date=2015-09-13 20:25:03 progress=0.937650  cache=17.7MiB(5882tx)
     432015-11-13 01:33:44 UpdateTip: new best=000000000000000008e6fe7a38957d49c3304bf16dc8f8fcc3198101747e98ea  height=374376  log2_work=83.342103  tx=83569561  date=2015-09-13 20:50:09 progress=0.937669  cache=32.7MiB(10355tx)
     442015-11-13 01:33:58 UpdateTip: new best=0000000000000000107fc52247ac7b096728d757ecb4327544889a0a9fe9cd7b  height=374377  log2_work=83.342131  tx=83570501  date=2015-09-13 20:58:51 progress=0.937676  cache=40.3MiB(12450tx)
     452015-11-13 01:34:08 Pre-allocating up to position 0x200000 in rev00337.dat
     462015-11-13 01:34:08 UpdateTip: new best=000000000000000006414073f8f1e17d0db29cd9a911d90a93fa2d1423a91673  height=374378  log2_work=83.34216  tx=83571300  date=2015-09-13 21:08:17 progress=0.937683  cache=45.0MiB(13896tx)
     472015-11-13 01:34:39 UpdateTip: new best=000000000000000008ca7ce3ed3963ea4a7411ead8eb95656766adc08a15a7a2  height=374379  log2_work=83.342189  tx=83572511  date=2015-09-13 21:21:04 progress=0.937692  cache=57.8MiB(19173tx)
     482015-11-13 01:35:07 UpdateTip: new best=000000000000000000dfba5808fbffe532e380e96e2e7e014e4e7ade1ac384fd  height=374380  log2_work=83.342218  tx=83573171  date=2015-09-13 21:30:04 progress=0.937699  cache=0.1MiB(0tx)
     492015-11-13 01:35:10 UpdateTip: new best=00000000000000000849fb1de3084da6fae2aa897112ca4afed45f3270077fd3  height=374381  log2_work=83.342247  tx=83573409  date=2015-09-13 21:33:15 progress=0.937701  cache=4.4MiB(733tx)
     502015-11-13 01:36:00 UpdateTip: new best=0000000000000000074d78827de76b1db7c6cd9b5f55b483c269c34f956a6df4  height=374382  log2_work=83.342275  tx=83574450  date=2015-09-13 21:41:44 progress=0.937707  cache=21.0MiB(6742tx)
     512015-11-13 01:36:24 UpdateTip: new best=00000000000000000182dff9ad21bf8f5a60a3ff7e767a7d70a313cec741e733  height=374383  log2_work=83.342304  tx=83574920  date=2015-09-13 21:48:40 progress=0.937712  cache=42.3MiB(11561tx)
     522015-11-13 01:36:24 UpdateTip: new best=0000000000000000002b5d0b1887de7faff5ee8f7e1c9cac1a0911bdc17fe09d  height=374384  log2_work=83.342333  tx=83574921  date=2015-09-13 21:48:24 progress=0.937712  cache=42.3MiB(11562tx)
     532015-11-13 01:36:25 UpdateTip: new best=000000000000000005ce32fc9997a4b9e7e497a94a5c6466b69fa92e9dbfe4d8  height=374385  log2_work=83.342362  tx=83575031  date=2015-09-13 21:49:47 progress=0.937713  cache=42.6MiB(11799tx)
     542015-11-13 01:37:28 UpdateTip: new best=000000000000000000ce2a9bdeb1fc7fae8d8d2a53214f8fc3560da1b7287b99  height=374386  log2_work=83.342391  tx=83575977  date=2015-09-13 21:51:01 progress=0.937714  cache=64.8MiB(18491tx)
     552015-11-13 01:37:31 UpdateTip: new best=000000000000000004830e05bcd892c5c1d05513a45489ada3a70baf4f03fb73  height=374387  log2_work=83.342419  tx=83576196  date=2015-09-13 21:53:15 progress=0.937716  cache=65.4MiB(19062tx)
     562015-11-13 01:37:50 UpdateTip: new best=00000000000000000af216b29f617a0a371e9e1bf228a8e012d5c9054291ae29  height=374388  log2_work=83.342448  tx=83578506  date=2015-09-13 22:35:53 progress=0.937747  cache=0.1MiB(0tx)
     572015-11-13 01:37:51 UpdateTip: new best=000000000000000005ea09f774c8314b43d288cfb34e8a71c0fb1238cc90ca72  height=374389  log2_work=83.342477  tx=83578507  date=2015-09-13 22:37:30 progress=0.937748  cache=0.1MiB(1tx)
     582015-11-13 01:38:09 UpdateTip: new best=000000000000000001bbda245864c017b207c86d72b269d8e4072635a6956bbe  height=374390  log2_work=83.342506  tx=83579173  date=2015-09-13 22:38:52 progress=0.937750  cache=20.9MiB(2489tx)
     592015-11-13 01:39:02 Pre-allocating up to position 0x700000 in rev00336.dat
     602015-11-13 01:39:02 UpdateTip: new best=000000000000000009a2d8309e57d34eab86193581d30b83f5039e7ca0ca78c8  height=374391  log2_work=83.342534  tx=83579960  date=2015-09-13 22:45:22 progress=0.937754  cache=37.5MiB(9046tx)
     612015-11-13 01:39:41 UpdateTip: new best=00000000000000000f8fe597f56146b71b79f2eae07034c7016752a0d44f3d1c  height=374392  log2_work=83.342563  tx=83581455  date=2015-09-13 22:55:46 progress=0.937762  cache=49.1MiB(14630tx)
     622015-11-13 01:40:24 UpdateTip: new best=00000000000000000c238436152f8734305badd906dd6d74c81d575161bca11f  height=374393  log2_work=83.342592  tx=83582486  date=2015-09-13 22:58:44 progress=0.937765  cache=0.1MiB(0tx)
     632015-11-13 01:40:31 UpdateTip: new best=00000000000000000a80c2b13755364426a9d253dba04bfc875e82907aa3a842  height=374394  log2_work=83.342621  tx=83583309  date=2015-09-13 23:05:08 progress=0.937770  cache=4.0MiB(1620tx)
     642015-11-13 01:41:06 UpdateTip: new best=0000000000000000045eb183a9862b934f2470fc84855ea70721e91ab21b09b5  height=374395  log2_work=83.34265  tx=83586551  date=2015-09-13 23:14:47 progress=0.937780  cache=31.3MiB(7073tx)
     652015-11-13 01:41:13 UpdateTip: new best=00000000000000000fb86b48c19397fca71dee2a8c8558900073a25788bd27a1  height=374396  log2_work=83.342678  tx=83587142  date=2015-09-13 23:24:16 progress=0.937787  cache=32.2MiB(8317tx)
     662015-11-13 01:42:07 UpdateTip: new best=000000000000000007cdef658d485b97b48881888e8b6b704d55514995b4f503  height=374397  log2_work=83.342707  tx=83587815  date=2015-09-13 23:32:18 progress=0.937792  cache=53.7MiB(15343tx)
     672015-11-13 01:42:09 UpdateTip: new best=00000000000000000ccb8c2b47537029c2b7ceeda30582d19e5b71e3873fc965  height=374398  log2_work=83.342736  tx=83588144  date=2015-09-13 23:32:10 progress=0.937792  cache=55.9MiB(15997tx)
     682015-11-13 01:42:20 UpdateTip: new best=000000000000000004b41bc5fb277047a5e78b72d5a055969a544050134f874c  height=374399  log2_work=83.342765  tx=83589002  date=2015-09-13 23:52:39 progress=0.937807  cache=63.9MiB(18219tx)
     692015-11-13 01:42:30 UpdateTip: new best=000000000000000009c3589882292186d6c41542a9ecc1b911e3c108b7ae1a1f  height=374400  log2_work=83.342793  tx=83589772  date=2015-09-14 00:00:11 progress=0.937813  cache=0.1MiB(0tx)
     702015-11-13 01:42:58 UpdateTip: new best=000000000000000000c1d1e79e0666f9018a50c4436a088d632c92d65da35ffe  height=374401  log2_work=83.342822  tx=83590325  date=2015-09-14 00:07:32 progress=0.937818  cache=17.1MiB(6145tx)
     712015-11-13 01:43:42 Pre-allocating up to position 0x800000 in rev00336.dat
     722015-11-13 01:43:42 UpdateTip: new best=00000000000000000245106b927329e763c4f8295e0aecb55c4fcc1bc0d504df  height=374402  log2_work=83.342851  tx=83590740  date=2015-09-14 00:10:49 progress=0.937820  cache=31.3MiB(11870tx)
     732015-11-13 01:43:51 UpdateTip: new best=0000000000000000020a5cf083dd770e26f14d96ffa8bd3e3c6e3c819e11ae82  height=374403  log2_work=83.34288  tx=83591387  date=2015-09-14 00:17:22 progress=0.937825  cache=38.0MiB(13779tx)
     742015-11-13 01:43:58 UpdateTip: new best=00000000000000000014073262137733616fd023b291ab6e3391ab4fc75ed2d9  height=374404  log2_work=83.342909  tx=83591782  date=2015-09-14 00:24:29 progress=0.937830  cache=41.8MiB(14828tx)
     752015-11-13 01:44:19 UpdateTip: new best=000000000000000005a5be7c79b586f3f9181f2fd7bff46d8a849ece0e3dc606  height=374405  log2_work=83.342937  tx=83592256  date=2015-09-14 00:28:35 progress=0.937833  cache=54.5MiB(19816tx)
     762015-11-13 01:44:22 UpdateTip: new best=00000000000000000bd76c9bf55e18130dec4c381079d9068feabc65bcb4ccea  height=374406  log2_work=83.342966  tx=83592468  date=2015-09-14 00:32:10 progress=0.937836  cache=54.6MiB(20278tx)
     772015-11-13 01:44:44 UpdateTip: new best=00000000000000000e3224b4442fd0743673d5a79f51b9b4f76788b6fef6bef8  height=374407  log2_work=83.342995  tx=83592924  date=2015-09-14 00:34:06 progress=0.937838  cache=63.2MiB(24163tx)
     782015-11-13 01:45:14 UpdateTip: new best=000000000000000003908dea6f9a10c9e3b8d089e5cb17eadc458309cc467928  height=374408  log2_work=83.343024  tx=83593415  date=2015-09-14 00:34:51 progress=0.937838  cache=0.2MiB(0tx)
     792015-11-13 01:45:14 UpdateTip: new best=0000000000000000092359a574045d6d325fc80208af70bb8f3d8b5afcbff588  height=374409  log2_work=83.343052  tx=83593417  date=2015-09-14 00:36:38 progress=0.937839  cache=0.2MiB(3tx)
     802015-11-13 01:45:17 UpdateTip: new best=000000000000000007c1335722b2b4245a41b8d9465e5dd6f2418d70b59139cc  height=374410  log2_work=83.343081  tx=83593625  date=2015-09-14 00:39:28 progress=0.937842  cache=1.6MiB(760tx)
     812015-11-13 01:45:40 UpdateTip: new best=000000000000000003afb3bc48a74c21bf2b355d10dd4900bd9d21c11d35d2d7  height=374411  log2_work=83.34311  tx=83594019  date=2015-09-14 00:37:47 progress=0.937841  cache=13.6MiB(6763tx)
     822015-11-13 01:46:26 UpdateTip: new best=00000000000000000c93b0a438035bec563578f636785cb9210e735353f3c0cd  height=374412  log2_work=83.343139  tx=83595112  date=2015-09-14 00:48:05 progress=0.937848  cache=31.4MiB(13323tx)
     832015-11-13 01:46:27 UpdateTip: new best=00000000000000000dce616d05b6bfe265ad894120810539ae932a9a902bb926  height=374413  log2_work=83.343167  tx=83595193  date=2015-09-14 00:49:53 progress=0.937849  cache=31.6MiB(13505tx)
     842015-11-13 01:46:48 Pre-allocating up to position 0x900000 in rev00336.dat
     852015-11-13 01:46:48 UpdateTip: new best=00000000000000000db87f1fed648b1cbfdcb3ac1ab5540c7df1d9b9332a2133  height=374414  log2_work=83.343196  tx=83599200  date=2015-09-14 00:59:00 progress=0.937860  cache=44.7MiB(18028tx)
     862015-11-13 01:47:13 UpdateTip: new best=0000000000000000028e14e765514953773d488c099c12a84ff68d0f8ddd0dec  height=374415  log2_work=83.343225  tx=83600968  date=2015-09-14 01:33:05 progress=0.937884  cache=56.7MiB(23206tx)
     872015-11-13 01:47:55 UpdateTip: new best=000000000000000007a7ed55a25b56c392d7deba4f62d5901e6e0a099d220ea2  height=374416  log2_work=83.343254  tx=83601616  date=2015-09-14 01:35:01 progress=0.937886  cache=0.2MiB(0tx)
     882015-11-13 01:48:16 UpdateTip: new best=000000000000000009186abc73c43a3ad15bf6b49e229805efdffd892e5147e5  height=374417  log2_work=83.343283  tx=83602987  date=2015-09-14 01:55:19 progress=0.937901  cache=22.0MiB(4587tx)
     892015-11-13 01:48:21 UpdateTip: new best=000000000000000004d93c410b76fce5b0219a83d89a6a83e32040e24e66fe5b  height=374418  log2_work=83.343311  tx=83603543  date=2015-09-14 02:03:15 progress=0.937907  cache=24.8MiB(5777tx)
     902015-11-13 01:48:41 UpdateTip: new best=00000000000000000ab00fb8126620ee30c27f630a06542a7e3bd9d82276a917  height=374419  log2_work=83.34334  tx=83604690  date=2015-09-14 02:11:16 progress=0.937913  cache=37.8MiB(12225tx)
     912015-11-13 01:48:50 UpdateTip: new best=0000000000000000021968f19f07b3510acf743942570bbd94a61cf9d68de69e  height=374420  log2_work=83.343369  tx=83605391  date=2015-09-14 02:14:47 progress=0.937916  cache=39.8MiB(13581tx)
     922015-11-13 01:49:14 UpdateTip: new best=0000000000000000104dd1e1c0ba4016c530bd3bb8c72bedd9215218296ac977  height=374421  log2_work=83.343398  tx=83607383  date=2015-09-14 02:23:03 progress=0.937923  cache=54.9MiB(18570tx)
     932015-11-13 01:49:33 UpdateTip: new best=000000000000000004440f88a578bb2d1512b41eb43669a81af1783ffa3ebbb0  height=374422  log2_work=83.343426  tx=83607907  date=2015-09-14 02:28:28 progress=0.937927  cache=65.1MiB(22529tx)
     942015-11-13 01:49:40 UpdateTip: new best=000000000000000005a342ff7af762b5efe443d64399640b16a78131d0776559  height=374423  log2_work=83.343455  tx=83608458  date=2015-09-14 02:35:16 progress=0.937933  cache=0.2MiB(0tx)
     952015-11-13 01:50:09 Pre-allocating up to position 0xa00000 in rev00336.dat
     962015-11-13 01:50:09 UpdateTip: new best=00000000000000000c2e3a6fcf6ba9afe215453c99ad4a50433dea9dbcb1cfd5  height=374424  log2_work=83.343484  tx=83610048  date=2015-09-14 02:49:16 progress=0.937943  cache=20.1MiB(5192tx)
     972015-11-13 01:50:15 UpdateTip: new best=00000000000000000b8a2a041eb3bf1a7a7830ac4391857503dfd0b50812ca38  height=374425  log2_work=83.343513  tx=83610534  date=2015-09-14 02:57:11 progress=0.937949  cache=24.2MiB(6417tx)
     982015-11-13 01:50:46 UpdateTip: new best=00000000000000001239b46be8bc83605725ea7d354cf2c1985500440f5bfb95  height=374426  log2_work=83.343541  tx=83610998  date=2015-09-14 03:00:35 progress=0.937952  cache=45.9MiB(12418tx)
     992015-11-13 01:50:56 UpdateTip: new best=000000000000000005d3e1099e7d8422ee190817d8590f7e0d930772644966d3  height=374427  log2_work=83.34357  tx=83611570  date=2015-09-14 03:07:58 progress=0.937957  cache=49.8MiB(14137tx)
    1002015-11-13 01:51:14 UpdateTip: new best=0000000000000000016dff3c599d0bb936984678c236d21db0991e7e11bb7e90  height=374428  log2_work=83.343599  tx=83612378  date=2015-09-14 03:18:19 progress=0.937965  cache=61.3MiB(18358tx)
    1012015-11-13 01:51:33 UpdateTip: new best=000000000000000006285042fda1aa350e03626263c7712b97d9f82bacb8d6f5  height=374429  log2_work=83.343628  tx=83612672  date=2015-09-14 03:18:04 progress=0.937965  cache=0.2MiB(0tx)
    1022015-11-13 01:51:46 UpdateTip: new best=000000000000000009d85ab03061e93a7148640f790446f35ba98a8d103c0f02  height=374430  log2_work=83.343656  tx=83613473  date=2015-09-14 03:29:45 progress=0.937973  cache=11.5MiB(2205tx)
    1032015-11-13 01:52:07 UpdateTip: new best=00000000000000000a92be86935fe9d901f023b0e04f0e58a75d76cf7fb83608  height=374431  log2_work=83.343685  tx=83615242  date=2015-09-14 03:53:18 progress=0.937991  cache=22.6MiB(5691tx)
    1042015-11-13 01:52:31 UpdateTip: new best=00000000000000001157af152645bd26e365f0fdcd30ba4144a1522e313baa10  height=374432  log2_work=83.343714  tx=83616090  date=2015-09-14 04:04:56 progress=0.937999  cache=40.4MiB(11135tx)
    1052015-11-13 01:52:32 UpdateTip: new best=000000000000000012cb2f945a19251f2e104ea0a90f2d7819d2754452e92884  height=374433  log2_work=83.343743  tx=83616295  date=2015-09-14 04:08:06 progress=0.938002  cache=40.5MiB(11505tx)
    1062015-11-13 01:52:47 UpdateTip: new best=0000000000000000112a38544ffe5ad4e8948b70ed2f0b55db0ab38493f6c689  height=374434  log2_work=83.343771  tx=83616801  date=2015-09-14 04:12:27 progress=0.938005  cache=54.9MiB(15327tx)
    1072015-11-13 01:52:54 UpdateTip: new best=00000000000000000ae90489c6e5fdb83be6e78e7228e19e5a392239f131c160  height=374435  log2_work=83.3438  tx=83617786  date=2015-09-14 04:28:50 progress=0.938017  cache=56.8MiB(17112tx)
    1082015-11-13 01:53:09 Pre-allocating up to position 0xb00000 in rev00336.dat
    1092015-11-13 01:53:10 UpdateTip: new best=000000000000000006b0db201fc80cdc04ecbf9f5cc14670564f17bfa0e5eb65  height=374436  log2_work=83.343829  tx=83620382  date=2015-09-14 04:39:45 progress=0.938027  cache=0.2MiB(0tx)
    1102015-11-13 01:53:39 UpdateTip: new best=000000000000000008f2ec67ecd6305668a1e66152c5fe6cbe87f5276306e3e1  height=374437  log2_work=83.343858  tx=83620916  date=2015-09-14 04:43:55 progress=0.938030  cache=12.8MiB(5039tx)
    1112015-11-13 01:54:06 UpdateTip: new best=000000000000000008914b01fc40cf4c7491f048a72aba292a5e8da8b9fa78b9  height=374438  log2_work=83.343886  tx=83623530  date=2015-09-14 04:46:09 progress=0.938034  cache=24.8MiB(10526tx)
    1122015-11-13 01:54:47 UpdateTip: new best=00000000000000001063455eb761528b6001f448c58884547d5c8590293e5d85  height=374439  log2_work=83.343915  tx=83624170  date=2015-09-14 04:48:59 progress=0.938036  cache=43.3MiB(16885tx)
    1132015-11-13 01:54:51 UpdateTip: new best=0000000000000000038ee0066680705455d500f287f6c56db7a979c2426a4c02  height=374440  log2_work=83.343944  tx=83624473  date=2015-09-14 04:53:13 progress=0.938039  cache=46.6MiB(17702tx)
    1142015-11-13 01:55:12 UpdateTip: new best=0000000000000000069b0c9677a2cf84b00e0b0046a56f60ab3e50a550e951d0  height=374441  log2_work=83.343973  tx=83625118  date=2015-09-14 04:58:19 progress=0.938043  cache=58.2MiB(23401tx)
    1152015-11-13 01:55:29 UpdateTip: new best=000000000000000001de250dcfa47f8313aec2f1f41a56f4fb0d099eb497c2b2  height=374442  log2_work=83.344001  tx=83625925  date=2015-09-14 05:03:30 progress=0.938047  cache=0.2MiB(0tx)
    1162015-11-13 01:55:46 UpdateTip: new best=00000000000000000733bae3b9101a66568074332e9e34773383754050c6ff4c  height=374443  log2_work=83.34403  tx=83626642  date=2015-09-14 05:05:30 progress=0.938049  cache=16.3MiB(6154tx)
    1172015-11-13 01:56:02 Pre-allocating up to position 0xc00000 in rev00336.dat
    1182015-11-13 01:56:02 UpdateTip: new best=00000000000000000aa98628177998cc2066559409c6d517f038fd5fa3b0067f  height=374444  log2_work=83.344059  tx=83628048  date=2015-09-14 05:28:49 progress=0.938066  cache=31.9MiB(10065tx)
    1192015-11-13 01:56:19 UpdateTip: new best=000000000000000004c241778cbbc269e912df5fe8d856efaea916daa82d2575  height=374445  log2_work=83.344088  tx=83629656  date=2015-09-14 05:51:00 progress=0.938083  cache=44.4MiB(13775tx)
    1202015-11-13 01:56:28 UpdateTip: new best=000000000000000005da8867286d40b1280583ce3f64f7aa7fad64bddfdc799a  height=374446  log2_work=83.344116  tx=83630486  date=2015-09-14 05:55:09 progress=0.938086  cache=55.0MiB(15455tx)
    1212015-11-13 01:57:04 UpdateTip: new best=00000000000000000dedf7f76ba8b9016c7708b9cc66c473fca0a6cb68492bfc  height=374447  log2_work=83.344145  tx=83631308  date=2015-09-14 06:06:50 progress=0.938095  cache=0.2MiB(0tx)
    1222015-11-13 01:57:08 UpdateTip: new best=00000000000000000e456dcb296688f05a0767e314f27e045a6c6323244f89fc  height=374448  log2_work=83.344174  tx=83631541  date=2015-09-14 06:08:57 progress=0.938096  cache=3.0MiB(717tx)
    1232015-11-13 01:57:23 UpdateTip: new best=00000000000000000cd4b43eff0f42c60b33c50ce6f7187a3c9d29fee9ebcc4d  height=374449  log2_work=83.344203  tx=83633146  date=2015-09-14 06:27:15 progress=0.938110  cache=14.5MiB(3975tx)
    1242015-11-13 01:57:47 UpdateTip: new best=00000000000000000ba48a54ad329f7d89b9756b73ca292966a45e63b646d247  height=374450  log2_work=83.344231  tx=83636252  date=2015-09-14 06:33:19 progress=0.938117  cache=24.0MiB(8193tx)
    1252015-11-13 01:58:06 UpdateTip: new best=00000000000000000226b4102a0926687e2cae2c624109c4b0d77a9fd6040994  height=374451  log2_work=83.34426  tx=83636935  date=2015-09-14 06:35:51 progress=0.938120  cache=34.0MiB(13819tx)
    1262015-11-13 01:58:06 UpdateTip: new best=00000000000000000789de381da218c28f7acab6bb324172136cb2d1f1d1918b  height=374452  log2_work=83.344289  tx=83637003  date=2015-09-14 06:36:51 progress=0.938120  cache=34.1MiB(13943tx)
    1272015-11-13 01:58:21 UpdateTip: new best=0000000000000000015c7f531571cbb3698adff6a5f0396ebe2d77f81aeb296a  height=374453  log2_work=83.344318  tx=83637727  date=2015-09-14 06:38:46 progress=0.938122  cache=42.2MiB(18323tx)
    1282015-11-13 01:58:41 UpdateTip: new best=00000000000000000dcc8e9019610f091e534caaaa5fafa8ce71c0c2dbd66871  height=374454  log2_work=83.344346  tx=83639004  date=2015-09-14 07:10:01 progress=0.938144  cache=51.4MiB(22020tx)
    1292015-11-13 01:59:01 UpdateTip: new best=00000000000000000861647533e30b51448056dea7bbf526b229601617dcb05f  height=374455  log2_work=83.344375  tx=83640760  date=2015-09-14 07:19:11 progress=0.938152  cache=0.2MiB(0tx)
    1302015-11-13 01:59:19 UpdateTip: new best=000000000000000006f45b2662d878a5c0f6ec1851ae962c4cae9f309b4a9313  height=374456  log2_work=83.344404  tx=83641390  date=2015-09-14 07:19:48 progress=0.938153  cache=13.3MiB(5987tx)
    1312015-11-13 01:59:41 UpdateTip: new best=0000000000000000045246e47765322bd83ae438bd19d2dc9e150787b25f907e  height=374457  log2_work=83.344433  tx=83645013  date=2015-09-14 07:34:02 progress=0.938166  cache=20.7MiB(9670tx)
    1322015-11-13 01:59:55 UpdateTip: new best=000000000000000005f9e6664082f2a7ae495c79581c97b92cefa14446711713  height=374458  log2_work=83.344461  tx=83645652  date=2015-09-14 07:22:47 progress=0.938159  cache=28.3MiB(13935tx)
    1332015-11-13 02:00:21 Pre-allocating up to position 0x300000 in rev00337.dat
    1342015-11-13 02:00:21 UpdateTip: new best=0000000000000000072e3d3af7eca4f4174b16f867e03f27325f097688ed3838  height=374459  log2_work=83.34449  tx=83646235  date=2015-09-14 07:28:40 progress=0.938164  cache=42.8MiB(17864tx)
    1352015-11-13 02:01:01 UpdateTip: new best=00000000000000000c2af4b9f72c75b55f4f6ab4b7011d3213d97cb0865c7751  height=374460  log2_work=83.344519  tx=83648168  date=2015-09-14 07:53:29 progress=0.938182  cache=54.3MiB(22424tx)
    1362015-11-13 02:02:08 UpdateTip: new best=000000000000000003a95cccc6a29391236d143d7fc9587fb080939716e07dbb  height=374461  log2_work=83.344548  tx=83649370  date=2015-09-14 08:05:52 progress=0.938191  cache=0.2MiB(0tx)
    1372015-11-13 02:02:08 UpdateTip: new best=000000000000000004ac6db2534580c62018a8e769d3319f751108322eaecdbd  height=374462  log2_work=83.344576  tx=83649371  date=2015-09-14 08:06:09 progress=0.938191  cache=0.2MiB(1tx)
    1382015-11-13 02:03:41 UpdateTip: new best=000000000000000009bfb15446a7275287689137ae7a70639c3b5b5761e0a3ca  height=374463  log2_work=83.344605  tx=83650290  date=2015-09-14 08:12:17 progress=0.938195  cache=16.5MiB(5798tx)
    1392015-11-13 02:04:24 UpdateTip: new best=00000000000000000e1c1f074b806d28b306a70e586f1bee5b8d815254bc4aa5  height=374464  log2_work=83.344634  tx=83651804  date=2015-09-14 08:32:50 progress=0.938210  cache=53.2MiB(10445tx)
    1402015-11-13 02:04:32 UpdateTip: new best=00000000000000000845aecec7ff4788414f478b0cdaf291b6dd443f1638b5ad  height=374465  log2_work=83.344663  tx=83652000  date=2015-09-14 08:34:11 progress=0.938211  cache=60.9MiB(11240tx)
    1412015-11-13 02:05:01 UpdateTip: new best=00000000000000000996d50d30d6b5604f59100218de1c61087dbca89df084ff  height=374466  log2_work=83.344691  tx=83652634  date=2015-09-14 08:40:50 progress=0.938216  cache=0.2MiB(0tx)
    1422015-11-13 02:05:42 UpdateTip: new best=000000000000000000a5b8b03740afea7abd33a194cf3e4de7e4beab8b7188e1  height=374467  log2_work=83.34472  tx=83653637  date=2015-09-14 08:41:32 progress=0.938217  cache=12.7MiB(5034tx)
    1432015-11-13 02:06:11 UpdateTip: new best=000000000000000000b1099e77f55e917fbdb445fb116456cf126ff44209b29b  height=374468  log2_work=83.344749  tx=83654679  date=2015-09-14 08:41:43 progress=0.938218  cache=26.0MiB(11284tx)
    1442015-11-13 02:06:29 Pre-allocating up to position 0x400000 in rev00337.dat
    1452015-11-13 02:06:29 UpdateTip: new best=00000000000000000222f31d4cbef619399242be2d45113062cda449f9fb164f  height=374469  log2_work=83.344778  tx=83655487  date=2015-09-14 08:50:50 progress=0.938225  cache=35.8MiB(13801tx)
    1462015-11-13 02:06:33 UpdateTip: new best=000000000000000010830614960b9b4e09c49ca8414478730e816161ff5dee90  height=374470  log2_work=83.344806  tx=83655740  date=2015-09-14 08:53:59 progress=0.938227  cache=41.8MiB(14433tx)
    1472015-11-13 02:07:07 UpdateTip: new best=000000000000000010ad9174e17274cd4004b5ef6eab815e5e37452fb80aca13  height=374471  log2_work=83.344835  tx=83656544  date=2015-09-14 08:59:56 progress=0.938231  cache=0.2MiB(0tx)
    1482015-11-13 02:07:41 UpdateTip: new best=000000000000000003663b9f371fa73dca936ca94d0ef5ae9ab1e9dd98e7d589  height=374472  log2_work=83.344864  tx=83657306  date=2015-09-14 09:08:14 progress=0.938237  cache=33.4MiB(5453tx)
    1492015-11-13 02:07:55 UpdateTip: new best=00000000000000000f5e8b422468e0782850594881f19fa27c136bf9e168ed0a  height=374473  log2_work=83.344892  tx=83658399  date=2015-09-14 09:22:52 progress=0.938248  cache=48.9MiB(8245tx)
    1502015-11-13 02:08:29 UpdateTip: new best=00000000000000000a20486eb488338252c9b236f69d975350f369392612bfdc  height=374474  log2_work=83.344921  tx=83659326  date=2015-09-14 09:26:14 progress=0.938251  cache=0.2MiB(0tx)
    1512015-11-13 02:09:07 UpdateTip: new best=0000000000000000033ac13d7f4d3defee41e0ce2c34d1b190a88a582c595f91  height=374475  log2_work=83.34495  tx=83660397  date=2015-09-14 09:33:54 progress=0.938257  cache=16.9MiB(6451tx)
    1522015-11-13 02:09:47 UpdateTip: new best=000000000000000004260f338d0c65ff887d50320864e148e172201cdbc3866e  height=374476  log2_work=83.344979  tx=83662582  date=2015-09-14 10:12:50 progress=0.938285  cache=43.0MiB(11626tx)
    1532015-11-13 02:10:19 UpdateTip: new best=000000000000000003783910316c65a836a613c62ed625dd639c7cc25c6d8c5e  height=374477  log2_work=83.345007  tx=83664123  date=2015-09-14 10:24:31 progress=0.938294  cache=61.0MiB(16172tx)
    1542015-11-13 02:10:23 UpdateTip: new best=00000000000000000ac9616939732fef37fcf40f25ab99fd35042973a5ea3255  height=374478  log2_work=83.345036  tx=83664292  date=2015-09-14 10:26:06 progress=0.938296  cache=63.3MiB(16621tx)
    1552015-11-13 02:10:56 UpdateTip: new best=000000000000000012001196e771b4aa15b5e2da7f5e85f029601d5d6c241721  height=374479  log2_work=83.345065  tx=83665877  date=2015-09-14 10:47:25 progress=0.938311  cache=0.2MiB(0tx)
    1562015-11-13 02:11:31 UpdateTip: new best=00000000000000000281c151ff37bb4e883be76457b4292c10e6d0e6b3369daa  height=374480  log2_work=83.345094  tx=83666545  date=2015-09-14 10:48:56 progress=0.938313  cache=20.1MiB(5726tx)
    1572015-11-13 02:11:55 Pre-allocating up to position 0xd00000 in rev00336.dat
    1582015-11-13 02:11:55 UpdateTip: new best=000000000000000000f87c0d3b84cad15dce31a324ab79ed55ed217e391d136c  height=374481  log2_work=83.345122  tx=83667738  date=2015-09-14 11:02:45 progress=0.938323  cache=33.8MiB(10899tx)
    1592015-11-13 02:12:14 UpdateTip: new best=000000000000000007053a77eead86209358060e4cca995f73cb2fc89d3e0ebd  height=374482  log2_work=83.345151  tx=83668869  date=2015-09-14 11:10:48 progress=0.938329  cache=42.1MiB(15649tx)
    1602015-11-13 02:12:43 UpdateTip: new best=0000000000000000001db75b5bd86d62afc0c122dee00b618c0957e2a7272afc  height=374483  log2_work=83.34518  tx=83670912  date=2015-09-14 11:46:04 progress=0.938355  cache=47.4MiB(20457tx)
    1612015-11-13 02:13:05 UpdateTip: new best=000000000000000012411bd67e74fd519ee0b69cc395a963a03541c6c5e48485  height=374484  log2_work=83.345209  tx=83671972  date=2015-09-14 11:50:16 progress=0.938359  cache=64.3MiB(23850tx)
    1622015-11-13 02:13:21 UpdateTip: new best=000000000000000008f775e0ceeac298796136092dd09a4f9799ca6132d6645e  height=374485  log2_work=83.345237  tx=83672586  date=2015-09-14 11:51:06 progress=0.938360  cache=0.2MiB(0tx)
    1632015-11-13 02:13:21 UpdateTip: new best=0000000000000000127de2cea720f4df05d871e1fe89baf827eb23da8f5912a0  height=374486  log2_work=83.345266  tx=83672588  date=2015-09-14 11:51:36 progress=0.938360  cache=0.2MiB(3tx)
    1642015-11-13 02:13:48 UpdateTip: new best=00000000000000000359511e5c93a208bbf2e7548ab52ac07d8bc544f6514d10  height=374487  log2_work=83.345295  tx=83674146  date=2015-09-14 12:09:48 progress=0.938374  cache=25.5MiB(4243tx)
    1652015-11-13 02:14:11 UpdateTip: new best=0000000000000000068e31065ee3e8b2689240db793860365699f27053fc731d  height=374488  log2_work=83.345323  tx=83675002  date=2015-09-14 12:14:33 progress=0.938377  cache=34.5MiB(9152tx)
    1662015-11-13 02:14:11 UpdateTip: new best=0000000000000000114fab5e190932823c68bb53e96279d6e4f25b950b33cb1b  height=374489  log2_work=83.345352  tx=83675003  date=2015-09-14 12:15:51 progress=0.938378  cache=34.5MiB(9153tx)
    1672015-11-13 02:14:27 UpdateTip: new best=0000000000000000002f081e26a49d5517584ff3558363031208b93ffb51f6ce  height=374490  log2_work=83.345381  tx=83676707  date=2015-09-14 12:39:27 progress=0.938396  cache=37.1MiB(12281tx)
    1682015-11-13 02:14:42 UpdateTip: new best=000000000000000003cc6b9be044b3023d37037a2ae56757ca87bec5fbaeee5c  height=374491  log2_work=83.34541  tx=83677948  date=2015-09-14 12:43:36 progress=0.938400  cache=49.6MiB(15130tx)
    1692015-11-13 02:15:03 Pre-allocating up to position 0xe00000 in rev00336.dat
    1702015-11-13 02:15:03 UpdateTip: new best=00000000000000000b6fd82d8632ccc32bf7cc973dca27b70886c98e5fec64a3  height=374492  log2_work=83.345438  tx=83680121  date=2015-09-14 13:14:25 progress=0.938423  cache=52.2MiB(19028tx)
    1712015-11-13 02:15:19 UpdateTip: new best=00000000000000000e7d6a152f2cfe5d197801894101ea5ce1824c3fa73b43a4  height=374493  log2_work=83.345467  tx=83681538  date=2015-09-14 13:23:31 progress=0.938430  cache=59.2MiB(21936tx)
    1722015-11-13 02:15:37 UpdateTip: new best=0000000000000000046a6d1fbbb6b183a83379efbf536c592eeb40df8a1fa783  height=374494  log2_work=83.345496  tx=83682814  date=2015-09-14 13:32:21 progress=0.938437  cache=0.2MiB(0tx)
    1732015-11-13 02:16:08 UpdateTip: new best=000000000000000007ca1ac7984f0f97da4ba174920fcf72a0b33222e043fde8  height=374495  log2_work=83.345524  tx=83684177  date=2015-09-14 13:45:38 progress=0.938447  cache=19.4MiB(6093tx)
    1742015-11-13 02:16:27 UpdateTip: new best=00000000000000000e43cd91ccd7b838c30e22a699a8aacb24994d13273f4872  height=374496  log2_work=83.345553  tx=83685362  date=2015-09-14 13:57:24 progress=0.938456  cache=35.0MiB(10430tx)
    1752015-11-13 02:16:44 UpdateTip: new best=0000000000000000100570c918f6787bd7722e77a536c34a1a20bd611b390c45  height=374497  log2_work=83.345582  tx=83686568  date=2015-09-14 14:05:05 progress=0.938463  cache=42.2MiB(13712tx)
    1762015-11-13 02:16:59 UpdateTip: new best=00000000000000000c8b3f83d1d22c07183c501334fe209b0ef507a9d0be11ac  height=374498  log2_work=83.345611  tx=83688057  date=2015-09-14 14:17:55 progress=0.938473  cache=47.9MiB(17250tx)
    1772015-11-13 02:17:18 UpdateTip: new best=000000000000000004ac212e10778cae49f88b46d63e40dae3251e0870bdc610  height=374499  log2_work=83.345639  tx=83689891  date=2015-09-14 14:33:13 progress=0.938485  cache=61.7MiB(21123tx)
    1782015-11-13 02:17:47 Pre-allocating up to position 0xf00000 in rev00336.dat
    1792015-11-13 02:17:49 UpdateTip: new best=0000000000000000122dabf23f78e231668a8e2816aa798731581ed2fc83a030  height=374500  log2_work=83.345668  tx=83691247  date=2015-09-14 14:46:00 progress=0.938494  cache=0.2MiB(0tx)
    1802015-11-13 02:18:20 UpdateTip: new best=000000000000000012c28648600c5e6fa0ab69749d93dda943df9b0eaa17271c  height=374501  log2_work=83.345697  tx=83692834  date=2015-09-14 14:55:27 progress=0.938502  cache=21.9MiB(5402tx)
    1812015-11-13 02:18:41 UpdateTip: new best=000000000000000006fa9861736e4096092ee1327775b8b7e6e158b274413666  height=374502  log2_work=83.345726  tx=83694823  date=2015-09-14 15:15:41 progress=0.938518  cache=32.5MiB(9794tx)
    1822015-11-13 02:18:57 UpdateTip: new best=00000000000000000f04b41618ce459783033186663dd757fae7e30c65be9172  height=374503  log2_work=83.345754  tx=83696184  date=2015-09-14 15:26:38 progress=0.938526  cache=43.8MiB(12875tx)
    1832015-11-13 02:19:04 UpdateTip: new best=000000000000000012b3ae57b3f79df9aa3e48d91bf3bc1ea7d03b8ec17060bd  height=374504  log2_work=83.345783  tx=83696877  date=2015-09-14 15:32:10 progress=0.938531  cache=45.4MiB(14238tx)
    1842015-11-13 02:19:34 UpdateTip: new best=00000000000000000d13e3bc1d0553a067bfa1c03d19e8ce771b4036bae38177  height=374505  log2_work=83.345812  tx=83697575  date=2015-09-14 15:34:58 progress=0.938533  cache=58.3MiB(19811tx)
    1852015-11-13 02:20:06 UpdateTip: new best=00000000000000000328c26e90320e0a32ba9a41e4d8c7aba5bd9cc1aa6b2b76  height=374506  log2_work=83.34584  tx=83699600  date=2015-09-14 15:55:20 progress=0.938548  cache=65.1MiB(24065tx)
    1862015-11-13 02:20:43 UpdateTip: new best=000000000000000011a6b05d9f991f255b0ab332066bdf20950268cadf10cb2d  height=374507  log2_work=83.345869  tx=83701165  date=2015-09-14 16:14:42 progress=0.938563  cache=0.2MiB(0tx)
    1872015-11-13 02:21:43 UpdateTip: new best=00000000000000000bf3d649fb32f7e3c74fb369cc1b33735c07bbfa4abef3c8  height=374508  log2_work=83.345898  tx=83701872  date=2015-09-14 16:43:31 progress=0.938582  cache=2.8MiB(6582tx)
    1882015-11-13 02:22:32 Pre-allocating up to position 0x1000000 in rev00336.dat
    1892015-11-13 02:22:32 UpdateTip: new best=0000000000000000005c782001081ada7aa8cc906b226480bfa55e7b8e3c3caf  height=374509  log2_work=83.345927  tx=83701995  date=2015-09-14 16:46:14 progress=0.938584  cache=3.8MiB(12182tx)
    1902015-11-13 02:23:10 UpdateTip: new best=000000000000000009e1180097b6e70ecb3c027cbbb33a25f92b51bee1d5d5dc  height=374510  log2_work=83.345955  tx=83704150  date=2015-09-14 17:04:55 progress=0.938598  cache=8.7MiB(18500tx)
    1912015-11-13 02:23:28 UpdateTip: new best=000000000000000001ea7e48d4cacf2ce8c1dcd9670bbe848dd22925eff55eda  height=374511  log2_work=83.345984  tx=83706158  date=2015-09-14 17:21:04 progress=0.938611  cache=14.9MiB(22559tx)
    1922015-11-13 02:23:49 UpdateTip: new best=00000000000000000353c5881f398412b6e0af40991ec3589d8f7b0ba79ded51  height=374512  log2_work=83.346013  tx=83708876  date=2015-09-14 17:37:08 progress=0.938624  cache=23.9MiB(27354tx)
    1932015-11-13 02:24:05 UpdateTip: new best=00000000000000000f56fcba1ff3b12f05f7cccafe6f4914e310504b195b3499  height=374513  log2_work=83.346041  tx=83710394  date=2015-09-14 17:45:56 progress=0.938632  cache=32.8MiB(30504tx)
    1942015-11-13 02:24:24 UpdateTip: new best=00000000000000000925c936b3831c3c2e3dfdb4489c01623d385688647dec0a  height=374514  log2_work=83.34607  tx=83712295  date=2015-09-14 18:01:15 progress=0.938644  cache=46.1MiB(34830tx)
    1952015-11-13 02:24:43 UpdateTip: new best=00000000000000000eaf5ac6dba59ec5983bae8b76f6b597fbfed072cfc80476  height=374515  log2_work=83.346099  tx=83713597  date=2015-09-14 18:05:54 progress=0.938648  cache=0.2MiB(0tx)
    1962015-11-13 02:25:09 UpdateTip: new best=00000000000000000946de982befd2a5814d2c075e629c3d7f03f9934e7992a6  height=374516  log2_work=83.346127  tx=83714897  date=2015-09-14 18:16:32 progress=0.938656  cache=37.1MiB(4642tx)
    1972015-11-13 02:25:27 Pre-allocating up to position 0x1100000 in rev00336.dat
    1982015-11-13 02:25:27 UpdateTip: new best=00000000000000000e6a47d8f2be5015fee55f21f841495190724c0796646252  height=374517  log2_work=83.346156  tx=83716520  date=2015-09-14 18:23:40 progress=0.938663  cache=46.5MiB(9037tx)
    1992015-11-13 02:25:43 UpdateTip: new best=0000000000000000107faf8030f5c10103a7f245854d5c9e5a19a0e86a817e72  height=374518  log2_work=83.346185  tx=83717544  date=2015-09-14 18:30:37 progress=0.938668  cache=53.5MiB(13001tx)
    2002015-11-13 02:26:06 UpdateTip: new best=000000000000000007e76e4e2a2859f8cafe2ec4e86733410547a3edefa9df22  height=374519  log2_work=83.346214  tx=83719524  date=2015-09-14 18:49:32 progress=0.938683  cache=60.8MiB(17335tx)
    2012015-11-13 02:26:47 UpdateTip: new best=00000000000000000e4865130b5528a0fc51f30fba629f7f45b656dec18308b2  height=374520  log2_work=83.346242  tx=83720007  date=2015-09-14 18:50:59 progress=0.938684  cache=0.2MiB(0tx)
    2022015-11-13 02:27:14 UpdateTip: new best=0000000000000000130fc24ae096c7770edf1afc66ce782603175e091e60bb02  height=374521  log2_work=83.346271  tx=83722188  date=2015-09-14 19:18:25 progress=0.938704  cache=15.2MiB(5588tx)
    2032015-11-13 02:27:41 UpdateTip: new best=00000000000000000aa1ecd528dfb8dab3f98395acf0806a01e518d7aed47659  height=374522  log2_work=83.3463  tx=83723166  date=2015-09-14 19:15:49 progress=0.938703  cache=25.9MiB(11150tx)
    2042015-11-13 02:27:43 UpdateTip: new best=000000000000000002d6a34aa4740e89e438d7083eb1b26e0bb0ca5fe8cb8d98  height=374523  log2_work=83.346328  tx=83723336  date=2015-09-14 19:17:28 progress=0.938705  cache=26.8MiB(11446tx)
    2052015-11-13 02:27:59 UpdateTip: new best=000000000000000001266b2f329165a4fab6ce2dddcc68cd890e5770050371b5  height=374524  log2_work=83.346357  tx=83725504  date=2015-09-14 19:37:06 progress=0.938720  cache=32.5MiB(15818tx)
    2062015-11-13 02:28:17 UpdateTip: new best=00000000000000000adb9029a216705570d80e5acacb267a911ae58f8b17ba98  height=374525  log2_work=83.346386  tx=83727390  date=2015-09-14 19:55:52 progress=0.938734  cache=44.4MiB(19701tx)
    2072015-11-13 02:28:29 Pre-allocating up to position 0x1200000 in rev00336.dat
    2082015-11-13 02:28:29 UpdateTip: new best=000000000000000011f895d470f6cf342e89d87e3184484bdddb0cf99a61b3b1  height=374526  log2_work=83.346415  tx=83728842  date=2015-09-14 20:11:06 progress=0.938746  cache=52.7MiB(22557tx)
    2092015-11-13 02:29:12 UpdateTip: new best=000000000000000000db717b4c076da2d1b9ff8ddbc94132e3a8d008a0fb62b9  height=374527  log2_work=83.346443  tx=83729328  date=2015-09-14 20:12:17 progress=0.938747  cache=63.3MiB(28622tx)
    2102015-11-13 02:29:22 UpdateTip: new best=00000000000000000bd9d02bc3c71d95dc3445c533611ba97093e36e9fb0208b  height=374528  log2_work=83.346472  tx=83729950  date=2015-09-14 20:16:29 progress=0.938750  cache=0.2MiB(0tx)
    2112015-11-13 02:29:46 UpdateTip: new best=00000000000000000a806d622582bb113204fc55ef975f0dc7565d490cea30c6  height=374529  log2_work=83.346501  tx=83731226  date=2015-09-14 20:22:10 progress=0.938755  cache=11.7MiB(4713tx)
    2122015-11-13 02:30:16 UpdateTip: new best=00000000000000000be4facad446f5d3a9a143be6722c7c30a43312601eac66e  height=374530  log2_work=83.346529  tx=83732460  date=2015-09-14 20:32:09 progress=0.938763  cache=19.1MiB(9182tx)
    2132015-11-13 02:30:36 UpdateTip: new best=000000000000000006505d7052e2813e3e30aea97a56c0241fc46dc2cda8a4c4  height=374531  log2_work=83.346558  tx=83733207  date=2015-09-14 20:37:52 progress=0.938767  cache=30.5MiB(14809tx)
    2142015-11-13 02:30:51 Pre-allocating up to position 0x200000 in rev00338.dat
    2152015-11-13 02:30:51 UpdateTip: new best=00000000000000001242d64c44b6f21b462f9529ec6f699733d33ed16c86c62f  height=374532  log2_work=83.346587  tx=83734800  date=2015-09-14 20:54:00 progress=0.938780  cache=36.7MiB(18246tx)
    2162015-11-13 02:30:56 UpdateTip: new best=000000000000000001459e9795ecaade1198a8267dfcd8d20f30ddf5a220bbb9  height=374533  log2_work=83.346615  tx=83735299  date=2015-09-14 21:06:03 progress=0.938788  cache=40.3MiB(19435tx)
    2172015-11-13 02:31:15 UpdateTip: new best=0000000000000000007f5f64c993585398c8aea549f6c0ab4efaf88b6e6d90da  height=374534  log2_work=83.346644  tx=83736208  date=2015-09-14 21:03:10 progress=0.938787  cache=50.3MiB(24404tx)
    2182015-11-13 02:31:37 UpdateTip: new best=00000000000000000a7fb4a5d41aa7d34405e1b07ac89d0ffde5f7507b669132  height=374535  log2_work=83.346673  tx=83737498  date=2015-09-14 21:14:42 progress=0.938796  cache=60.5MiB(29061tx)
    2192015-11-13 02:31:50 UpdateTip: new best=000000000000000010947f1d4a6280f7a7f08a2c048e2f92ec3ef0b24ca7ad54  height=374536  log2_work=83.346702  tx=83738781  date=2015-09-14 21:28:04 progress=0.938806  cache=0.2MiB(0tx)
    2202015-11-13 02:31:57 mapBlockIndex.size() = 383472
    2212015-11-13 02:31:57 nBestHeight = 374536
    2222015-11-13 02:31:57 setKeyPool.size() = 101
    2232015-11-13 02:31:57 mapWallet.size() = 0
    2242015-11-13 02:31:57 mapAddressBook.size() = 1
    225==>2015-11-13 02:31:57 init message: Loading addresses...
    2262015-11-13 02:31:57 Loaded 38273 addresses from peers.dat  344ms
    2272015-11-13 02:31:58 DNS seeding disabled
    2282015-11-13 02:31:58 upnp thread start
    2292015-11-13 02:31:58 addcon thread start
    2302015-11-13 02:31:58 msghand thread start
    2312015-11-13 02:31:58 opencon thread start
    2322015-11-13 02:31:58 net thread start
    2332015-11-13 02:31:58 init message: Done loading
    2342015-11-13 02:31:58 receive version message: /Satoshi:0.9.99/: version 70002, blocks=280006, us=50.136.181.193:8333, peer=2
    2352015-11-13 02:31:58 Added time data, samples 2, offset -3582 (-59 minutes)
    
  7. dthorpe commented at 10:40 pm on November 13, 2015: none

    I don’t see a correlation to #6784. There is no huge memory consumption here. Process memory working set is stable at under 300MB.

    Granted, this machine is not exactly a powerhouse, but a few months ago earlier builds (0.10.x) of the bitcoin client running on this same machine did not exhibit this constant disk thrashing, and did not starve the peer connections.

    Looking at the disk I/O perf stats (Windows Resource Manager), shows nearly all disk I/Os are reads from more than 30 files, 300 to 500bytes/sec per file, with fairly rare write I/Os to one file at about 28,000 bytes/sec.

    This feels like read caching is completely disabled, or read cache size is way too small for the random access data set. This is probably related to the recent switch away from using memory mapped file I/O in favor of discreet file I/O. Is an over-aggressive flush operation clearing the read cache in addition to flushing buffered writes to disk?

    Is there a config file or cmd line switch to change cache memory size?

  8. sipa commented at 10:45 pm on November 13, 2015: member

    Set -dbcache=4000 if you have a 64-bit system and 4 GB to spare. On Nov 13, 2015 11:40 PM, “Danny Thorpe” notifications@github.com wrote:

    I don’t see a correlation to #6784 #6784. There is no huge memory consumption here. Process memory working set is stable at under 300MB.

    Granted, this machine is not exactly a powerhouse, but a few months ago earlier builds (0.10.x) of the bitcoin client running on this same machine did not exhibit this constant disk thrashing, and did not starve the peer connections.

    Looking at the disk I/O perf stats (Windows Resource Manager), shows nearly all disk I/Os are reads from more than 30 files, 300 to 500bytes/sec per file, with fairly rare write I/Os to one file at about 28,000 bytes/sec.

    This feels like read caching is completely disabled, or read cache size is way too small for the random access data set. This is probably related to the recent switch away from using memory mapped file I/O in favor of discreet file I/O. Is an over-aggressive flush operation clearing the read cache in addition to flushing buffered writes to disk?

    Is there a config file or cmd line switch to change cache memory size?

    — Reply to this email directly or view it on GitHub #7001 (comment).

  9. sipa commented at 0:37 am on November 14, 2015: member
    Oh, 32 bit. In that case better limit to max 1 GB (-dbcache=1000) or so.
  10. laanwj added the label Windows on Nov 19, 2015
  11. dthorpe commented at 11:18 pm on January 21, 2016: none

    FYI: After migrating the bitcoin data to a new SSD drive on this same (old) machine, overall system performance while Bitcoin-qt is “catching up” is MUCH better than before. Same system RAM, OS is still running from spinning disk, but bitcoin data is now on SSD.

    I can even run a testnet node at the same time as a mainnet node without starving the testnet peers. (Testnet node is caught up to tip, mainnet node is about 5 weeks behind) Prior to the SSD upgrade, running the mainnet node caused so much system lag that the testnet node would start dropping peers due to excessive ping times.

    CPU utilization is pegged at 99%. Prior to the SSD upgrade, CPU utilization fluttered between 40 and 60% while the disk chattered violently. Adding -dbcache=700 helped only some. Clearly, the process was/is disk-bound. With the SSD upgrade, disk I/O latency is reduced enough that the CPU is now the long pole.

    I can’t yet speak to block verification rate with the SSD upgrade, as the UI is still just sitting there with “Activating Best Chain”. ;>

    -Danny

    p.s. This machine is now running 0.12.0rc1

  12. laanwj added the label Resource usage on Mar 14, 2016
  13. MarcoFalke commented at 11:53 am on May 24, 2016: member
    Activating Best Chain is no longer done during the splash screen. Anything left to do here?
  14. fanquake closed this on May 18, 2017

  15. MarcoFalke locked this on Sep 8, 2021

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-09-29 10:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me