Out of memory error, but there’s plenty of free ram and disk space #21554

issue dynamo-foundation openend this issue on March 30, 2021
  1. dynamo-foundation commented at 8:29 pm on March 30, 2021: none

    Getting “out of memory” error during initial sync.

    To not get an out of memory error

    Got an out of memory error

    Happens every time

    Master branch pulled March 29, 2021. Compiled on Windows 10 64 bit using Visual Studio 2019. PC has 48GB of RAM and 1TB of free space on drive used by BTC. Ran in both debug and release mode - same issue. In debug mode there is no assertion, application just exits. I can restart and the chain will build again then fail after a couple hours. Running with command line option -dbcache=16384 Also tried -dbcache=8192, same issue If I run without -dbcache it seems to work (although didn’t wait the whole 18 hours to build entire chain)

    See above

    Not GUI related

    Console log prior to error…

    2021-03-30T18:37:22Z UpdateTip: new best=0000000000000001e8f6b27e3b62f267084f38a4d5403d69834aa3343fea651a height=282642 version=0x00000002 log2_work=76.074504 tx=31728000 date=‘2014-01-27T02:05:56Z’ progress=0.050509 cache=608.9MiB(5398004txo) 2021-03-30T18:38:05Z UpdateTip: new best=000000000000000087b15e077fd569625cd9c477ef9afc539c45c01ead1e55c8 height=282800 version=0x00000002 log2_work=76.101250 tx=31786000 date=‘2014-01-27T23:30:17Z’ progress=0.050601 cache=614.9MiB(5454255txo) 2021-03-30T18:43:09Z UpdateTip: new best=0000000000000001438931934118d2772deb56efc7be190deb38cc72de267c0a height=284014 version=0x00000002 log2_work=76.291719 tx=32194000 date=‘2014-02-03T21:29:48Z’ progress=0.051250 cache=640.6MiB(5694152txo) 2021-03-30T18:51:28Z UpdateTip: new best=0000000000000000eeea130a6418dc2538690945ab676669c10e569ec5739214 height=285669 version=0x00000002 log2_work=76.550758 tx=32851000 date=‘2014-02-13T20:29:35Z’ progress=0.052296 cache=664.5MiB(5917370txo) 2021-03-30T18:52:20Z UpdateTip: new best=0000000000000001597f7836afd8544c4dc8579f80df1255188d4c4d53d842c0 height=285844 version=0x00000002 log2_work=76.576214 tx=32919000 date=‘2014-02-14T18:08:25Z’ progress=0.052404 cache=666.9MiB(5939396txo) 2021-03-30T18:55:52Z UpdateTip: new best=00000000000000014ac899e0228af437a9752242e7369054877b9bdcd62dc61f height=286636 version=0x00000002 log2_work=76.695542 tx=33198000 date=‘2014-02-19T05:35:45Z’ progress=0.052848 cache=680.5MiB(6065783txo) 2021-03-30T18:57:53Z UpdateTip: new best=00000000000000011eec266001c88f675a3bce478f9525188479b11d02632b95 height=287029 version=0x00000002 log2_work=76.756517 tx=33354000 date=‘2014-02-21T10:23:07Z’ progress=0.053097 cache=687.1MiB(6127357txo) 2021-03-30T19:04:03Z UpdateTip: new best=0000000000000000d5b08fc230ba956a1b0ce35b8a0ee31240e00c92b123589c height=288247 version=0x00000002 log2_work=76.930614 tx=33834000 date=‘2014-02-28T08:46:47Z’ progress=0.053861 cache=711.4MiB(6354539txo) 2021-03-30T19:39:47Z UpdateTip: new best=00000000000000008cdb96a355676dd4cfe4ade7b43707a24be9f44bf14804eb height=295162 version=0x00000002 log2_work=77.887140 tx=36612000 date=‘2014-04-10T21:45:32Z’ progress=0.058282 cache=827.9MiB(7441671txo) 2021-03-30T19:46:06Z UpdateTip: new best=000000000000000040fca501bf00bb045fec41d78bfad8c6b6c2bda19336b9cd height=296473 version=0x00000002 log2_work=78.056940 tx=37089000 date=‘2014-04-18T13:20:23Z’ progress=0.059042 cache=843.0MiB(7581096txo) 2021-03-30T19:51:55Z UpdateTip: new best=000000000000000096cc1c97fb30bce419c4ca56fbdd1589a4eda7accef9d75d height=297632 version=0x00000002 log2_work=78.208146 tx=37541000 date=‘2014-04-25T13:36:24Z’ progress=0.059761 cache=863.7MiB(7774188txo) 2021-03-30T19:53:33Z Error: Out of memory. Terminating.

  2. dynamo-foundation added the label Bug on Mar 30, 2021
  3. jarolrod commented at 3:36 am on April 4, 2021: member

    bitcoin can really tax your hardware, see: #21568 (comment)

    First, make sure that this is not related to a possible hardware issue. Are you overclocking any pieces to the point that instabilities come out because of how taxing bitcoin can be?

  4. dynamo-foundation commented at 4:03 pm on April 4, 2021: none

    bitcoin can really tax your hardware, see: #21568 (comment)

    First, make sure that this is not related to a possible hardware issue. Are you overclocking any pieces to the point that instabilities come out because of how taxing bitcoin can be?

    Yes I’m quite sure my hardware is not “taxed” :)

    Resource manager showed less than 10% hard memory usage (I have 48GB of RAM). Additionally, the error doesnt happen if I simply lower or remove the dbcache parameter. The problem is that there is an assert somewhere which blocks the debugger from breaking at the issue, so it’s effectively masked. “out of memory” is not a string constant anywhere, at least not one that Visual Studio could find.

  5. sipa commented at 4:05 pm on April 4, 2021: member
    Do you also get this problem if you use the release binaries (which aren’t created using msvc)?
  6. dynamo-foundation commented at 4:08 pm on April 4, 2021: none

    Do you also get this problem if you use the release binaries (which aren’t created using msvc)?

    Interesting! I had not tried that - will do it now.

  7. MarcoFalke commented at 4:08 pm on April 4, 2021: member
    Can you use the task manager to see where the memory is being consumed?
  8. dynamo-foundation commented at 4:10 pm on April 4, 2021: none

    Can you use the task manager to see where the memory is being consumed?

    Yes I did - it was bitcoind, but there was like maybe 4GB of reserved RAM and my machine has 48GB of RAM.

  9. MarcoFalke commented at 4:11 pm on April 4, 2021: member
    Is msvc’s default compile mode the debug mode, which might consume more memory at run time?
  10. dynamo-foundation commented at 4:13 pm on April 4, 2021: none

    So I found it! I think maybe VS was not indexing correctly

    line 931 of init.cpp LogPrintf(“Error: Out of memory. Terminating.\n”);

    I will reclone btc core now and set a breakpoint…

  11. sipa commented at 4:16 pm on April 4, 2021: member
    Maybe it’s somehow compiled in 32-bit mode, and can’t access more than 4 GB as a result?
  12. dynamo-foundation commented at 4:18 pm on April 4, 2021: none

    Maybe it’s somehow compiled in 32-bit mode, and can’t access more than 4 GB as a result?

    Testing

  13. dynamo-foundation commented at 7:28 pm on April 4, 2021: none

    After investigation and testing, suggest adding the following “quality of life” code at line 1577 in init.cpp:

    #ifdef _WIN32 if (nTotalCache >= ((int64_t)(2048) « 20)) LogPrintf(“WARNING: Memory allocations over 2GB on Windows x86 builds will fail, adjust dbcache parameter to less than 2048.\n”); #endif

  14. sipa commented at 7:31 pm on April 4, 2021: member
    This is very surprising. Are you sure you’re not building in 32-bit mode?
  15. dynamo-foundation commented at 7:33 pm on April 4, 2021: none

    Yes it is x86, so I’m suggesting the above guard code to inform people of when they make a poor choice for dbcache which will always result in the server crashing.

    x64 runs fine, albeit still slowly. dbcache 8192 did nothing to improve performance of IBD.

  16. sipa commented at 7:38 pm on April 4, 2021: member

    _WIN32 is also set for 64-bit windows, I believe.

    And I thought we already limited dbcache on 32-bit platforms, though MSVC builds in general are not officially supported, and far less tested. So it’s possible that it just doesn’t work there.

    dbcache over a few gigabytes only has diminishing returns, but for a full sync I would expect a speedup between 2048 and 8192; it should result in far fewer flushes to disk.

  17. MarcoFalke added the label Windows on Apr 4, 2021
  18. MarcoFalke added the label UTXO Db and Indexes on Apr 4, 2021
  19. dynamo-foundation commented at 7:55 pm on April 4, 2021: none

    _WIN32 is also set for 64-bit windows, I believe.

    Something being included is defining it…revision below, with proof of testing.

    And I thought we already limited dbcache on 32-bit platforms, though MSVC builds in general are not officially supported, and far less tested. So it’s possible that it just doesn’t work there.

    Other than absolute min/max testing, I dont see any limit on dbcache in init.cpp (lines 1561 to 1565).

    updated to filter for (is windows) and (not 64 bit) #ifdef _WIN32 #ifndef _WIN64 if (nTotalCache >= ((int64_t)(2048) « 20)) LogPrintf(“WARNING: Memory allocations over 2GB on Windows x86 builds will fail, adjust dbcache parameter to less than 2048.\n”); #endif #endif

    image

    Including proper support for windows helps get alot more programmers interested :)

  20. sipa commented at 7:56 pm on April 4, 2021: member
    First of all, do you have the problem also with release builds?
  21. dynamo-foundation commented at 8:04 pm on April 4, 2021: none

    First of all, do you have the problem also with release builds?

    Yes, Windows X86 release build with dbcache > 4096 will run out of memory. I didnt test

    • X64 build (but I suppose that works)
    • dbcache >=2048 (but I suppose that fails)

    If by “release build” you mean the binaries published on the bitcoin core website - those appear to be X64 and wont have this issue.

    Anyway my question is answered, so I’ll close it.

  22. dynamo-foundation closed this on Apr 4, 2021
  23. sipa commented at 8:10 pm on April 4, 2021: member

    If by “release build” you mean the binaries published on the bitcoin core website - those appear to be X64 and wont have this issue.

    Yes, that’s what I meant. I forgot we stopped supporting 32-bit Windows a while ago.

  24. DrahtBot locked this on Aug 18, 2022


dynamo-foundation jarolrod sipa MarcoFalke

Labels
Bug Windows UTXO Db and Indexes

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-31 03:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me