Unbounded reorg memory usage #9027

1. 
   TheBlueMatt commented at 9:48 pm on October 26, 2016: contributor

   #9014 brought to light in issue introduced in #7946 - when we do a reorg, we now keep the full contents of each block we connect in memory until the reorg is complete. Luckily, #9014 makes this a bit eaiser to fix - dont keep more than a few blocks around in shared_ptrs, just re-read them from disk after the reorg is complete when you need to call signalers. This may be complicated due to pruning, but it should be doable.
2. 
   TheBlueMatt commented at 9:50 pm on October 26, 2016: contributor

   Note that this issue has never made it into a released version, so to avoid regression we should fix this before 0.14.
3. fanquake added the label Resource usage on Oct 27, 2016
4. laanwj added this to the milestone 0.14.0 on Oct 27, 2016
5. MarcoFalke added this to the milestone 0.15.0 on Feb 2, 2017
6. MarcoFalke removed this from the milestone 0.14.0 on Feb 2, 2017
7. 
   MarcoFalke commented at 7:08 pm on February 2, 2017: member

   Defer this to 0.15 and remove 0.14 milestone according to current irc log.
8. 
   TheBlueMatt commented at 8:43 pm on June 28, 2017: contributor

   This was partially addressed in #9208. However, the issue of keeping around all blocks connected in the ConnectTrace still exists. Note that, of course, this does not present an issue during normal connect since a ConnectTrace should only ever contain one block (as ABCS will return if it made progress).

   #10286 will present yet another issue here. Because wallet callbacks will not force the ABC thread to pause, callbacks could pile up and even if blocks get freed from the ConnectTrace, they could still sit around in memory waiting for the callback to finish. Luckily since wallet also takes cs_main there is some ability to block the main thread from making progress, assuming locks will switch between threads and not always get taken on the same thread if there is contention, but hopefully eventually wallet can lose much of its cs_main-taking, which means wallet needs a way to pause validation if it is super, super huge and gets far behind.

9. 
   TheBlueMatt commented at 8:04 pm on July 25, 2017: contributor

   Should likely be pushed to 16, as 10708 is not making 15.
10. MarcoFalke added this to the milestone Future on Jul 25, 2017
11. MarcoFalke removed this from the milestone 0.15.0 on Jul 25, 2017
12. MarcoFalke removed this from the milestone Future on May 3, 2018
13. 
    willcl-ark commented at 8:31 pm on October 25, 2022: contributor

    Testing with bitcoin-cli invalidateblock 00000000000000000003618a9a50c352c58fa3a7ae6d0016d2720a9aff2c2db2, a 50k block re-org, shows that bitcoind memory usage peaks at ~ 10GB during this process on my system, although it cannot cause OOM:

    0$ grep ^VmPeak /proc/$(pidof bitcoind)/status
    1
    2VmPeak: 10721532 kB
    

    So it seems like this may still be somewhat present, despite previous comments indicating that it had gone away, and this comment indicating that #15402 had fixed it somewhat more concretely.

    Perhaps we should close this one and keep #14289 open, although it would be nice to address the penultimate comment here first?

    Edit: testing a day’s-worth of blocks re-org (6 * 24) used 729MB

14. 
    MarcoFalke commented at 9:17 am on August 31, 2023: member

    Testing

    Did you test with or without a wallet?

15. 
    willcl-ark commented at 9:34 am on September 6, 2023: contributor

    Testing

    Did you test with or without a wallet?

    I tested this without a wallet, which IIUC should give me the “maximum” unbounded memory usage in this scenario?

    I accidentally -reindex-ed my mainnet dev node earlier this morning as I was adding coinstatsindex to it. When that sorts itself out, I can re-test with a wallet loaded.

Contributors
TheBlueMatt MarcoFalke willcl-ark

Labels
Resource usage