Mempool package tracking #6654

sdaftuar commented at 1:23 AM on September 9, 2015: member

This is a preparatory pull to try to make reviewing #6557 easier.

In #6557, I added tracking packages of transactions to the mempool (for each tx, we track all the "descendant" transactions that depend on that tx), in order to make the mempool limiting code more effective. This PR is a standalone implementation of mempool descendant tracking, including the policy limits on transaction chains (limiting ancestors and descendants) proposed in #6557.

I'll rebase that pull off these commits assuming we can agree on this approach.

sdaftuar commented at 1:31 AM on September 9, 2015: member

@sipa I did end up reworking the tracking of in-mempool parents/children to use iterators rather than hashes as you had suggested (which I never pushed up to #6557).

btcdrak commented at 3:11 AM on September 9, 2015: contributor

@laanwj looks like travis crapped out on one of the jobs and needs restarting.

sdaftuar commented at 12:35 PM on September 9, 2015: member

@btcdrak Actually I think this is some kind of problem with the unit test code -- not sure why it fails to compile only in that one job but I was just able to reproduce locally. Working on a fix...

sdaftuar force-pushed on Sep 9, 2015

sdaftuar commented at 1:09 PM on September 9, 2015: member

@btcdrak Fixed and pushed back up, travis is happy now...

in src/main.h:None in d6401c1547 outdated

 473 | @@ -466,5 +474,7 @@ static const unsigned int REJECT_HIGHFEE = 0x100;
 474 |  static const unsigned int REJECT_ALREADY_KNOWN = 0x101;
 475 |  /** Transaction conflicts with a transaction already known */
 476 |  static const unsigned int REJECT_CONFLICT = 0x102;
 477 | +/** Transaction would result in too long in-mempool chain */
 478 | +static const unsigned int REJECT_LONGCHAIN = 0x103;

sdaftuar commented at 7:13 PM on September 14, 2015:

Will move this to not be an internal code (so we send a reject message back).

in src/main.cpp:None in d6401c1547 outdated

 920 | @@ -921,6 +921,17 @@ bool AcceptToMemoryPool(CTxMemPool& pool, CValidationState &state, const CTransa
 921 |                  REJECT_HIGHFEE, "absurdly-high-fee",
 922 |                  strprintf("%d > %d", nFees, ::minRelayTxFee.GetFee(nSize) * 10000));
 923 |  
 924 | +        // Calculate in-mempool ancestors, up to a limit.
 925 | +        CTxMemPool::setEntries setAncestors;
 926 | +        size_t nLimitAncestors = GetArg("-limitancestorcount", DEFAULT_ANCESTOR_LIMIT);
 927 | +        size_t nLimitAncestorSize = GetArg("-limitancestorsize", DEFAULT_ANCESTOR_SIZE_LIMIT)*1000;

petertodd commented at 7:29 PM on September 14, 2015:

As the idea is the ancestor size is set such that the whole string of transactions could be included in a single block, we really want this to be a less than the max block size, as there's always overhead to consider. (e.g. the coinbase, soft-limit, etc)

I'd suggest we set this to 900KB for a 100KB buffer - plenty even in the case of p2pool's large max 50KB coinbase. Another option might be to set it based on the soft-limit - default 750KB - with another 100KB of overhad buffer.

petertodd commented at 7:42 PM on September 14, 2015: contributor

In general, it'd be good to think about separating the reorg case entirely from the main mempool codebase. For instance, keep a simple linear list of reorged transactions, and after a reorg attempt to add them back the mempool one-by-one. This separate code could also handle cases where we might want to remine transactions that we otherwise wouldn't, as a "goodwill" gesture to reduce the impact of large reorgs.

in src/txmempool.cpp:None in d6401c1547 outdated

  58 | +// descendants.
  59 | +bool CTxMemPool::UpdateForDescendants(txiter updateIt, int maxDescendantsToVisit, cacheMap &cachedDescendants, const std::set<uint256> &setExclude)
  60 | +{
  61 | +    // Track the number of entries (outside setExclude) that we'd need to visit
  62 | +    // (will bail out if it exceeds maxDescendantsToVisit)
  63 | +    int nChildrenToVisit = 0;

petertodd commented at 7:45 PM on September 14, 2015:

trailing whitespace

pstratem commented at 7:54 PM on September 14, 2015: contributor

Needs a comment defining ancestor/descendant.

in src/txmempool.cpp:None in d6401c1547 outdated

 190 | +
 191 | +    size_t totalSizeWithAncestors = entry.GetTxSize();
 192 | +
 193 | +    while (!parentHashes.empty()) {
 194 | +        setAncestors.insert(parentHashes.begin(), parentHashes.end());
 195 | +        setEntries stageParentSet;

petertodd commented at 7:54 PM on September 14, 2015:

trailing whitespace

in src/txmempool.cpp:None in d6401c1547 outdated

 216 | +                }
 217 | +                if (stageParentSet.size() + setAncestors.size() + 1 > limitAncestorCount) {
 218 | +                    errString = strprintf("too many unconfirmed ancestors [limit: %u]", limitAncestorCount);
 219 | +                    return false;
 220 | +                }
 221 | +            }

petertodd commented at 7:56 PM on September 14, 2015:

trailing whitespace

in src/txmempool.cpp:None in d6401c1547 outdated

 398 | +    mapTx.erase(it);
 399 | +    nTransactionsUpdated++;
 400 | +    minerPolicyEstimator->removeTx(hash);
 401 | +}
 402 | +
 403 | +// Calculates descendants of hash that are not already in setDescendants, and adds to

petertodd commented at 8:10 PM on September 14, 2015:

trailing whitespace

in src/txmempool.cpp:None in d6401c1547 outdated

 559 |          unsigned int i = 0;
 560 | -        checkTotal += it->second.GetTxSize();
 561 | -        innerUsage += it->second.DynamicMemoryUsage();
 562 | -        const CTransaction& tx = it->second.GetTx();
 563 | +        checkTotal += it->GetTxSize();
 564 | +        innerUsage += it->DynamicMemoryUsage();

petertodd commented at 8:18 PM on September 14, 2015:

trailing whitespace

in src/txmempool.h:None in d6401c1547 outdated

 362 | @@ -138,6 +363,29 @@ class CTxMemPool
 363 |      void ApplyDeltas(const uint256 hash, double &dPriorityDelta, CAmount &nFeeDelta);
 364 |      void ClearPrioritisation(const uint256 hash);
 365 |  
 366 | +public:
 367 | +    void RemoveStaged(std::set<uint256>& stage);

petertodd commented at 8:39 PM on September 14, 2015:

Need to document what RemoveStaged() is.

in src/txmempool.h:None in d6401c1547 outdated

 367 | +    void RemoveStaged(std::set<uint256>& stage);
 368 | +
 369 | +    /** When adding transactions from a disconnected block back to the mempool,
 370 | +     *  new mempool entries may have children in the mempool (which is generally
 371 | +     *  not the case when otherwise adding transactions).
 372 | +     *  UpdateTransactionsFromBlock will find child transactions and update the

petertodd commented at 8:40 PM on September 14, 2015:

Would prefer if comments like this are written as "Name()" rather than just "Name" to make it clear what's a variable and what's a function.

in src/txmempool.h:None in d6401c1547 outdated

 416 | @@ -169,6 +417,48 @@ class CTxMemPool
 417 |      bool ReadFeeEstimates(CAutoFile& filein);
 418 |  
 419 |      size_t DynamicMemoryUsage() const;
 420 | +
 421 | +private:
 422 | +    /** UpdateForDescendants is used by UpdateTransactionsFromBlock to update
 423 | +     *  the descendants for a single transaction that has been added to the
 424 | +     *  mempool but may have child transactions in the mempool, eg during a
 425 | +     *  chain reorg.  setExclude is the set of descendant transactions in the

petertodd commented at 8:41 PM on September 14, 2015:

Double-space after periods? Obvs you're actually Satoshi.

in src/txmempool.h:None in d6401c1547 outdated

 371 | +     *  not the case when otherwise adding transactions).
 372 | +     *  UpdateTransactionsFromBlock will find child transactions and update the
 373 | +     *  descendant state for each transaction in hashesToUpdate (excluding any
 374 | +     *  child transactions present in hashesToUpdate, which are already accounted
 375 | +     *  for).
 376 | +     */

petertodd commented at 8:42 PM on September 14, 2015:

This comment doesn't make it clear whether or not transactions in hashesToUpdate are or are not already in the mempool. :)

sipa commented at 8:45 PM on September 14, 2015: member

Concept ACK.

petertodd commented at 8:54 PM on September 14, 2015: contributor

utACK, modulo nits.

jonasschnelli commented at 9:00 PM on September 14, 2015: contributor

Concept ACK / code review ACK. Passes gitian/osx/debian build/unit-tests/rpc-tests.

petertodd commented at 9:16 PM on September 14, 2015: contributor

Tested with my FSS and Full RBF stress tests and -checkmempool, no unexpected failures. (this pull-req of course doesn't enable any RBF behavior, so replacements failed!)

dcousens commented at 10:00 PM on September 14, 2015: contributor

concept ACK

in src/main.cpp:None in d6401c1547 outdated

 920 | @@ -921,6 +921,17 @@ bool AcceptToMemoryPool(CTxMemPool& pool, CValidationState &state, const CTransa
 921 |                  REJECT_HIGHFEE, "absurdly-high-fee",
 922 |                  strprintf("%d > %d", nFees, ::minRelayTxFee.GetFee(nSize) * 10000));
 923 |  
 924 | +        // Calculate in-mempool ancestors, up to a limit.
 925 | +        CTxMemPool::setEntries setAncestors;
 926 | +        size_t nLimitAncestors = GetArg("-limitancestorcount", DEFAULT_ANCESTOR_LIMIT);

dcousens commented at 10:14 PM on September 14, 2015:

Using GetArg (an application level function) in mempool code feels a bit odd.

sdaftuar commented at 12:12 AM on September 16, 2015:

I agree but is there a better way to do this? I think I'm following the common practice in the code base, though I look forward to a future refactoring when we have better encapsulation.

in src/txmempool.cpp:None in d6401c1547 outdated

  68 | +    while (!stageEntries.empty()) {
  69 | +        setAllDescendants.insert(stageEntries.begin(), stageEntries.end());
  70 | +
  71 | +        setEntries entriesToAdd;
  72 | +        BOOST_FOREACH(const txiter cit, stageEntries) {
  73 | +            if (cit->IsDirty()) {

dcousens commented at 10:25 PM on September 14, 2015:

Dirty implies something changed typically, perhaps instead change this to ShouldSkip?

in src/txmempool.cpp:None in d6401c1547 outdated

 150 | +        std::map<COutPoint, CInPoint>::iterator iter = mapNextTx.lower_bound(COutPoint(hash, 0));
 151 | +        // First calculate the children, and update setMemPoolChildren to
 152 | +        // include them, and update their setMemPoolParents to include this tx.
 153 | +        for (; iter != mapNextTx.end() && iter->first.hash == hash; ++iter) {
 154 | +            const uint256 &childHash = iter->second.ptx->GetHash();
 155 | +            txiter childIter = mapTx.find(childHash);

dcousens commented at 10:28 PM on September 14, 2015:

Do we need to check childIter == mapTx.end()?

sdaftuar commented at 12:31 AM on September 16, 2015:

I'll add an assert to make clear that is not possible.

in src/txmempool.cpp:None in d6401c1547 outdated

 254 | +
 255 | +void CTxMemPool::UpdateForRemoveFromMempool(const std::set<uint256> &hashesToRemove)
 256 | +{
 257 | +    // For each entry, walk back all ancestors and decrement size associated with this
 258 | +    // transaction
 259 | +    uint64_t nNoLimit = std::numeric_limits<uint64_t>::max();

dcousens commented at 10:29 PM on September 14, 2015:

const?

in src/txmempool.cpp:None in d6401c1547 outdated

 279 | +
 280 | +void CTxMemPoolEntry::SetDirty()
 281 | +{
 282 | +    nCountWithDescendants=0;
 283 | +    nSizeWithDescendants=nTxSize;
 284 | +    nFeesWithDescendants=nFee;

dcousens commented at 10:29 PM on September 14, 2015:

nit: Spacing between operators

in src/txmempool.cpp:None in d6401c1547 outdated

 234 | +    BOOST_FOREACH(txiter piter, parentIters) {
 235 | +        UpdateChild(piter, it, add);
 236 | +    }
 237 | +    int64_t updateCount = (add ? 1 : -1);
 238 | +    int64_t updateSize = updateCount * it->GetTxSize();
 239 | +    CAmount updateFee = updateCount * it->GetFee();

dcousens commented at 10:32 PM on September 14, 2015:

nit: const on these?

in src/txmempool.cpp:None in d6401c1547 outdated

 243 | +}
 244 | +
 245 | +// TODO: pass a txiter instead?
 246 | +void CTxMemPool::UpdateChildrenForRemoval(const uint256 &hash)
 247 | +{
 248 | +    txiter it = mapTx.find(hash);

dcousens commented at 10:33 PM on September 14, 2015:

Check/assert it == mapTx.end()?

in src/txmempool.cpp:None in d6401c1547 outdated

 257 | +    // For each entry, walk back all ancestors and decrement size associated with this
 258 | +    // transaction
 259 | +    uint64_t nNoLimit = std::numeric_limits<uint64_t>::max();
 260 | +    BOOST_FOREACH(const uint256& removeHash, hashesToRemove) {
 261 | +        setEntries setAncestors;
 262 | +        const CTxMemPoolEntry &entry = *mapTx.find(removeHash);

dcousens commented at 10:33 PM on September 14, 2015:

Check/assert it == mapTx.end(), especially before de-reference

in src/txmempool.cpp:None in d6401c1547 outdated

 417 | +    // already been walked, or will be walked in this iteration).
 418 | +    while (!stage.empty()) {
 419 | +        setDescendants.insert(stage.begin(), stage.end());
 420 | +        std::set<uint256> setNext;
 421 | +        BOOST_FOREACH(const uint256 &stagehash, stage) {
 422 | +            indexed_transaction_set::iterator it = mapTx.find(stagehash);

dcousens commented at 10:35 PM on September 14, 2015:

Check/assert it == mapTx.end()? Would only be possible on the first stagehash AFAIK

Also, const/const_iterator?

in src/txmempool.cpp:None in d6401c1547 outdated

 476 | -            minerPolicyEstimator->removeTx(hash);
 477 | +        } else {
 478 | +            setAllRemoves = txToRemove;
 479 | +        }
 480 | +        BOOST_FOREACH(const uint256& hash, setAllRemoves) {
 481 | +            removed.push_back(mapTx.find(hash)->GetTx());

dcousens commented at 10:36 PM on September 14, 2015:

Again, assumption of hash in mapTx, worth asserting?

in src/txmempool.cpp:None in d6401c1547 outdated

 587 | +        CTxMemPool::setEntries setChildrenCheck;
 588 | +        std::map<COutPoint, CInPoint>::const_iterator iter = mapNextTx.lower_bound(COutPoint(it->GetTx().GetHash(), 0));
 589 | +        int64_t childSizes = 0;
 590 | +        CAmount childFees = 0;
 591 | +        for (; iter != mapNextTx.end() && iter->first.hash == it->GetTx().GetHash(); ++iter) {
 592 | +            txiter childit = mapTx.find(iter->second.ptx->GetHash());

dcousens commented at 10:39 PM on September 14, 2015:

Again, find not checked, if this assumption can hold true, can it be documented?

sdaftuar commented at 8:06 PM on September 15, 2015:

mapNextTx contains exactly the in-mempool children of a given transaction, so this code should be safe. I'll add an assert to make the assumption clearer (the semantics of mapNextTx weren't changed by this pull).

in src/txmempool.cpp:None in d6401c1547 outdated

 561 | -        innerUsage += it->second.DynamicMemoryUsage();
 562 | -        const CTransaction& tx = it->second.GetTx();
 563 | +        checkTotal += it->GetTxSize();
 564 | +        innerUsage += it->DynamicMemoryUsage(); 
 565 | +        const CTransaction& tx = it->GetTx();
 566 | +        const TxLinks &links = mapLinks.find(it)->second;

dcousens commented at 10:40 PM on September 14, 2015:

Is mapLinks in perfect sync with mapTx?

sdaftuar commented at 7:13 PM on September 15, 2015:

Yes -- I'll be adding some additional asserts to make this more clear.

sdaftuar commented at 1:44 PM on September 16, 2015: member

Thanks everyone for reviewing! I've pushed up a series of commits to address everyone's feedback; I believe all comments should have been addressed.

These cleanups need to be squashed, and this pull now needs to be rebased (a merge conflict crept in now that I'm outputting additional information in getrawmempool). Reviewers -- please let me know if you prefer I leave these commits unsquashed/unrebased so you can review the changes. (In the absence of any expressed preferences, I'll plan to squash/rebase in a day or two so that this can become mergeable.) @petertodd I added the extra information to the getrawmempool RPC call (good idea!). Now that the RPC call is there, I realized that adding an RPC test that exercises the new limits is a good idea, so I've started work on that, but it's not yet complete.

dcousens commented at 3:16 PM on September 16, 2015: contributor

utACK, thanks @sdaftuar!

edit: wait up, might need to re-evaluate the policy options in regards to comments in #6403

in src/txmempool.h:None in 38e9d0da46 outdated

 300 | +
 301 |      mutable CCriticalSection cs;
 302 | -    std::map<uint256, CTxMemPoolEntry> mapTx;
 303 | +    indexed_transaction_set mapTx;
 304 | +    typedef indexed_transaction_set::iterator txiter;
 305 | +    struct CompareIteratorByHash {

morcos commented at 4:02 PM on September 16, 2015:

Can all these new members from here through to UpdateChild be made private?

in src/txmempool.cpp:None in 38e9d0da46 outdated

 769 | +        removeUnchecked(it);
 770 | +    }
 771 | +}
 772 | +
 773 | +bool CTxMemPool::addUnchecked(const uint256&hash, const CTxMemPoolEntry &entry, bool fCurrentEstimate)
 774 | +{

morcos commented at 4:04 PM on September 16, 2015:

needs LOCK(cs)?

in src/txmempool.cpp:None in 38e9d0da46 outdated

 757 | @@ -429,5 +758,58 @@ bool CCoinsViewMemPool::HaveCoins(const uint256 &txid) const {
 758 |  
 759 |  size_t CTxMemPool::DynamicMemoryUsage() const {
 760 |      LOCK(cs);
 761 | -    return memusage::DynamicUsage(mapTx) + memusage::DynamicUsage(mapNextTx) + memusage::DynamicUsage(mapDeltas) + cachedInnerUsage;
 762 | +    // Estimate the overhead of mapTx to be 9 pointers + an allocation, as no exact formula for boost::multi_index_contained is implemented.
 763 | +    return memusage::MallocUsage(sizeof(CTxMemPoolEntry) + 9 * sizeof(void*)) * mapTx.size() + memusage::DynamicUsage(mapNextTx) + memusage::DynamicUsage(mapDeltas) + memusage::DynamicUsage(mapLinks) + cachedInnerUsage;
 764 | +}
 765 | +
 766 | +void CTxMemPool::RemoveStaged(setEntries &stage) {

morcos commented at 4:05 PM on September 16, 2015:

Needs LOCK(cs) if ever called from somewhere else.

TheBlueMatt commented at 1:23 AM on September 19, 2015:

You can also AssertLockHeld(cs), otherwise, to make things more readable.

On September 16, 2015 11:05:34 AM CDT, Alex Morcos notifications@github.com wrote:

@@ -429,5 +758,58 @@ bool CCoinsViewMemPool::HaveCoins(const uint256 &txid) const {

size_t CTxMemPool::DynamicMemoryUsage() const { LOCK(cs);

return memusage::DynamicUsage(mapTx) + memusage::DynamicUsage(mapNextTx) + memusage::DynamicUsage(mapDeltas) + cachedInnerUsage;

// Estimate the overhead of mapTx to be 9 pointers + an allocation, as no exact formula for boost::multi_index_contained is implemented.

return memusage::MallocUsage(sizeof(CTxMemPoolEntry) + 9 * sizeof(void)) \ mapTx.size() + memusage::DynamicUsage(mapNextTx) + memusage::DynamicUsage(mapDeltas) + memusage::DynamicUsage(mapLinks) + cachedInnerUsage; +} + +void CTxMemPool::RemoveStaged(setEntries &stage) {

Needs LOCK(cs) if ever called from somewhere else.

Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/pull/6654/files#r39649424

in src/txmempool.cpp:None in 38e9d0da46 outdated

 112 | +// vHashesToUpdate is the set of transaction hashes from a disconnected block
 113 | +// which has been re-added to the mempool.
 114 | +// for each entry, look for descendants that are outside hashesToUpdate, and
 115 | +// add fee/size information for such descendants to the parent.
 116 | +void CTxMemPool::UpdateTransactionsFromBlock(const std::vector<uint256> &vHashesToUpdate)
 117 | +{

morcos commented at 4:06 PM on September 16, 2015:

Needs LOCK(cs)

in src/txmempool.h:None in 38e9d0da46 outdated

 305 | +    struct CompareIteratorByHash {
 306 | +        bool operator()(const txiter &a, const txiter &b) const {
 307 | +            return a->GetTx().GetHash() < b->GetTx().GetHash();
 308 | +        }
 309 | +    };
 310 | +    typedef std::set<txiter, CompareIteratorByHash> setEntries;

morcos commented at 4:15 PM on September 16, 2015:

oops, setEntries needs to be public, the rest can be private though.

TheBlueMatt commented at 1:23 AM on September 19, 2015: member

My computer's battery died, but in txmempool.h, the definition of txiter as "boost::multi_index_container::iterator" is "implementation defined" according to boost's docs. In the latest versions it is defined to an equivalent of "boost::multi_index_container::nth_index<0>::type::iterator", so you should use that.

On September 16, 2015 8:44:26 AM CDT, Suhas Daftuar notifications@github.com wrote:

Thanks everyone for reviewing! I've pushed up a series of commits to address everyone's feedback; I believe all comments should have been addressed.

These cleanups need to be squashed, and this pull now needs to be rebased (a merge conflict crept in now that I'm outputting additional information in getrawmempool). Reviewers -- please let me know if you prefer I leave these commits unsquashed/unrebased so you can review the changes. (In the absence of any expressed preferences, I'll plan to squash/rebase in a day or two so that this can become mergeable.)

@petertodd I added the extra information to the getrawmempool RPC call (good idea!). Now that the RPC call is there, I realized that adding an RPC test that exercises the new limits is a good idea, so I've started work on that, but it's not yet complete.

Reply to this email directly or view it on GitHub: #6654 (comment)

dcousens commented at 3:04 AM on September 19, 2015: contributor

@sdaftuar needs rebase.

TxMemPool: Change mapTx to a boost::multi_index_container

Indexes on:
- Tx Hash
- Fee Rate (fee-per-kb)

34628a1807

Track transaction packages in CTxMemPoolEntry

Associate with each CTxMemPoolEntry all the size/fees of descendant
mempool transactions.  Sort mempool by max(feerate of entry, feerate
of descendants).  Update statistics on-the-fly as transactions enter
or leave the mempool.

Also add ancestor and descendant limiting, so that transactions can
be rejected if the number or size of unconfirmed ancestors exceeds
a target, or if adding a transaction would cause some other mempool
entry to have too many (or too large) a set of unconfirmed in-
mempool descendants.

5add7a74a6

sdaftuar force-pushed on Sep 19, 2015

sdaftuar commented at 5:40 PM on September 19, 2015: member

I addressed the latest comments from @morcos and @TheBlueMatt, added an rpc test (it only tests the ancestor/descendant length limits, not the size limits, nor the handling of reorgs -- so that still can be improved), and then squashed everything down and rebased on master to get rid of the merge conflict in rpcblockchain.cpp.

Mirobit commented at 6:39 PM on September 19, 2015: contributor

Does anyone else think the ancestor limit is way to generous? In what other case than spamming do txs have 100 ancestors in mempool? Shouldn't 5 be enough?

petertodd commented at 12:53 AM on September 20, 2015: contributor

@Mirobit The limits are there to limit computational costs in determining things like dependent fees/size etc. There's no need to set them low unless the algorithms take a long time to compute those sums; the data structures in the mempool are mainly pointer following so it's fairly fast.

laanwj commented at 12:26 PM on September 21, 2015: member

utACK

laanwj merged this on Sep 21, 2015

laanwj closed this on Sep 21, 2015

laanwj referenced this in commit b0ce4508b0 on Sep 21, 2015

sipa commented at 4:47 PM on September 22, 2015: member

My node running 5add7a7 with -checkmempool crashed. Last debug.log message was:

2015-09-22 13:06:19 - Disconnect block: 90.04ms

morcos commented at 5:13 PM on September 22, 2015: member

@sipa can you give us some more details? were you running will all -debug options? for instance I assume you were not running with estimatefee debugging? it helps to narrow down where it crashed based on what didn't get printed to the debug log.

zkbot referenced this in commit 6d6b780969 on Feb 20, 2018

zkbot referenced this in commit 49274558c6 on Feb 20, 2018

furszy referenced this in commit eb00d0f62f on Jun 14, 2020

str4d referenced this in commit bd2c35a93f on Aug 10, 2021

str4d referenced this in commit 55ee6e5415 on Aug 10, 2021

str4d referenced this in commit 2a7ebb3aba on Aug 11, 2021

MarcoFalke locked this on Sep 8, 2021