Compute ‘short id’ when transaction joins mempool #27706

issue brunoerg openend this issue on May 20, 2023
  1. brunoerg commented at 11:38 am on May 20, 2023: contributor

    When a node receives a cmpctblock it has to verify to its check mempool in order to know whether it has all the required transactions to construct that block. If it doesn’t, it will send getblocktxn to fetch the missing tx{s}.

    PartiallyDownloadedBlock::InitData shows we have to iterate the whole mempool in order to get the short id and do the verifications, see:

     0for (size_t i = 0; i < pool->vTxHashes.size(); i++) {
     1    uint64_t shortid = cmpctblock.GetShortID(pool->vTxHashes[i].first);
     2    std::unordered_map<uint64_t, uint16_t>::iterator idit = shorttxids.find(shortid);
     3    if (idit != shorttxids.end()) {
     4        if (!have_txn[idit->second]) {
     5            txn_available[idit->second] = pool->vTxHashes[i].second->GetSharedTx();
     6            have_txn[idit->second]  = true;
     7            mempool_count++;
     8        } else {
     9            // If we find two mempool txn that match the short id, just request it.
    10            // This should be rare enough that the extra bandwidth doesn't matter,
    11            // but eating a round-trip due to FillBlock failure would be annoying
    12            if (txn_available[idit->second]) {
    13                txn_available[idit->second].reset();
    14                mempool_count--;
    15            }
    16        }
    17    }
    18    // Though ideally we'd continue scanning for the two-txn-match-shortid case,
    19    // the performance win of an early exit here is too good to pass up and worth
    20    // the extra risk.
    21    if (mempool_count == shorttxids.size())
    22        break;
    23}
    

    This means that every time we receive a compact block we have to iterate the whole mempool and calculate the “short id"s all over again. Couldn’tCTxMemPool have a hashmap where we could store the transactions’ short id right after joining the mempool and remove it once the tx gets confirmed/out of mempool?

  2. brunoerg commented at 11:38 am on May 20, 2023: contributor
  3. recursive-rat4 commented at 12:45 pm on May 20, 2023: none
    Short IDs, as they are specified in BIP 152, cannot be computed in advance because their formula includes a block header (and a random nonce) in addition to a transaction content.
  4. sipa commented at 12:57 pm on May 20, 2023: member

    Yeah, what @pavel-vasin says.

    Salting the hash computation using the block header is needed to prevent intentional collisions, and is the price for using short ids.

  5. brunoerg commented at 1:56 pm on May 20, 2023: contributor
    Makes sense, thanks for the clarification!
  6. maflcko added the label Questions and Help on May 20, 2023
  7. Davidson-Souza commented at 3:01 pm on May 20, 2023: none

    @pavel-vasin Yes, you’re right. The compact block relay design, as it currently exists, does not permit pre computations.

    Salting the hash computation using the block header is needed to prevent intentional collisions, and is the price for using short ids.

    Does it really need to be the block header? If we kept a random, per connection byte-array used for salting, how could an attacker cause intentional collisions?

    (Thinking outside BIP152 context) If we hold a per connection salt, and a rolling hash map for mempool transactions, we could avoid the mempool iteration entirely. Since we only keep a few high-bandwidth connections, and hash sets are very small, it’s not really a memory usage penalty, IMO. If our peer misbehave, we can detect it, once they would create an uncommon number of collisions.

    I guess a first good step would be benchmarking this code path, and see how it impacts the whole block compact propagation logic.

  8. recursive-rat4 commented at 9:11 am on May 21, 2023: none

    Does it really need to be the block header? If we kept a random, per connection byte-array used for salting, how could an attacker cause intentional collisions?

    If PoW or something similarly strong is not involved then the attacker can create collisions for a chosen set of nonces. An algorithm of the attack is straightforward:

    1. Connect to a next victim
    2. Negotiate a nonce
    3. Bruteforce transactions that collide under the nonce
    4. Go to step 1
  9. fanquake commented at 8:56 am on May 22, 2023: member
    Ok. Closing for now. Discussion can continue, but it’s not clear why this is an issue. General questions /discussion can also be asked/happen in IRC etc.
  10. fanquake closed this on May 22, 2023

  11. bitcoin locked this on May 21, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me