Limit amount of memory used in large re-orgs #5943

1. 
   gavinandresen commented at 8:02 pm on March 24, 2015: contributor

   This pull request was inspired by three things:

   1. Large block testing, where stress testing 100-block-long re-orgs with 100+ megabyte blocks used enormous amounts of memory.
   2. Planning for implementing a memory-limited memory pool.
   3. The general principle that using arbitrarily large amounts of memory is bad.

   Block re-org code before this pull request saves all transactions in the old fork to the memory pool before adding all the blocks on the new fork to the chain. That can use arbitrary amounts of memory if the re-org is arbitrarily long; for ‘reasonable’ re-orgs almost all of the transactions will be in blocks on the new fork, so you eventually end up with a small memory pool, and allocated a lot of memory for nothing.

   This pull request has almost identical behavior for short (6-or-fewer block-deep) re-orgs. But for deeper re-orgs, it does not add transactions to the memory pool– it assumes that the transactions are likely to be on the new fork, and, if they are not, that either the sender or the receiver of the transaction will re-broadcast.

   I made changes to the transaction relaying code that make transaction re-broadcast more robust; if for any reason a transaction is not added to the memory pool during a re-org, it is also cleared from the networking layer’s ‘setInventoryKnown’ so a later re-broadcast will be successful.

   Builds on #5940 and #5945

   Tested using mempool_resurrect_test.py and also manual testing to make sure the wallet’s automatic rebroadcast of unconfirmed transactions Does The Right Thing.

2. gavinandresen force-pushed on Mar 24, 2015
3. gavinandresen force-pushed on Mar 25, 2015
4. laanwj added the label Improvement on Mar 26, 2015
5. laanwj added the label UTXO Db and Indexes on Mar 26, 2015
6. gavinandresen force-pushed on Mar 26, 2015
7. Regression test framework tweaks

   Print out the temporary directory at the end of testing if --nocleanup is
   given (I found this very useful when running --tracerpc --nocleanup when
   developing new tests).
   
   And, by default, wait at most ten seconds for mempools or chains to sync
   before failing a test.

   3f87c7a1ea
8. Net layer: ForgetTransaction()

   Add a ForgetTransaction routine that is the
   opposite of RelayTransaction: it clears a
   transaction from peer nodes setInventoryKnown,
   so if we receive an 'inv' in the future we will
   respond with 'getdata' and un-forget it.
   
   Relies on a new mruset.erase() method (with unit tests).

   b9f56bd376
9. When doing a deep re-org, only add 6 blocks worth of transactions back into mempool

   Deep re-orgs try to put all transactions from all blocks back into the mempool;
   this is not a problem in practice (because deep re-orgs happen approximately never),
   but will be a problem when either the mempool is limited or you run out of
   memory for the mempool.
   
   This change limits the number of blocks worth of transactions to six.

   05697af782
10. gavinandresen force-pushed on May 12, 2015
11. gavinandresen closed this on Sep 28, 2015

    ---

12. MarcoFalke locked this on Sep 8, 2021

Contributors
gavinandresen

Labels
Refactoring UTXO Db and Indexes