log: log wtxids when possible, add TXPACKAGES category #28364

pull glozow wants to merge 4 commits into bitcoin:master from glozow:2023-08-log-wtxids changing 6 files +63 −35
  1. glozow commented at 1:32 pm on August 29, 2023: member

    This was taken from #28031 (see #27463 for project tracking).

    • Log wtxids in addition to txids when possible. This allows us to track the fate of a transaction from inv to mempool accept/reject through logs.
    • Additional orphan-related logging to make testing and debugging easier. Suggested in #28031#pullrequestreview-1531022386 and #28031 (review)
    • Add TXPACKAGES category for logging.
    • Move a nearby comment block that was in the wrong place.
  2. glozow added the label Utils/log/libs on Aug 29, 2023
  3. DrahtBot commented at 1:32 pm on August 29, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK instagibbs, brunoerg, mzumsande, achow101
    Concept ACK jonatack

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #28107 (util: Type-safe transaction identifiers by dergoegge)
    • #28031 (Package Relay 1/3: Introduce TxPackageTracker as Orphan Resolution Module by glozow)
    • #26697 (logging: use bitset for categories by LarryRuane)
    • #25038 (policy: nVersion=3 and Package RBF by glozow)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. in src/net_processing.cpp:2919 in 3c2b29046c outdated
    2918@@ -2919,9 +2919,10 @@ bool PeerManagerImpl::ProcessOrphanTx(Peer& peer)
    2919         const MempoolAcceptResult result = m_chainman.ProcessTransaction(porphanTx);
    


    instagibbs commented at 2:13 pm on August 29, 2023:

    in [log] include wtxid in tx {relay,validation,orphanage} logging

    looks like we’re missing one wtxid logging event here:

    https://github.com/glozow/bitcoin/blob/2023-08-log-wtxids/src/net_processing.cpp#L4224


    glozow commented at 3:20 pm on August 29, 2023:
    thanks, added wtxid to that log :+1:

    glozow commented at 3:42 pm on August 29, 2023:
    Oops, I added it with wtxid before txid :scream: . Fixed. Good thing we added tests eh
  5. in src/net_processing.cpp:4138 in 17e327d95a outdated
    4134@@ -4135,6 +4135,21 @@ void PeerManagerImpl::ProcessMessage(CNode& pfrom, const std::string& msg_type,
    4135                     RelayTransaction(tx.GetHash(), tx.GetWitnessHash());
    4136                 }
    4137             }
    4138+            // If a tx has been detected by m_recent_rejects, it is ignored. Because we haven't
    


    instagibbs commented at 2:59 pm on August 29, 2023:

    Ok I finally understand what this comment is saying, yay. In my words:

    0We stop processing a transaction if we currently have it in mempool or it is detected in m_recent_rejects.
    1
    2
    3Recall that m_recent_rejects can have false positives, so in this case we cannot safely
    4punish the peer without further processing, which we are explicitly avoiding with this cache.
    

    glozow commented at 3:20 pm on August 29, 2023:
    :+1:
  6. instagibbs approved
  7. instagibbs commented at 3:07 pm on August 29, 2023: member

    ACK 17e327d95ae9e55353f94acee126b227cf906c3f

    modulo that one missing log, for completeness

  8. DrahtBot added the label CI failed on Aug 29, 2023
  9. glozow force-pushed on Aug 29, 2023
  10. [log] include wtxid in tx {relay,validation,orphanage} logging a33dde1e41
  11. [log] add category TXPACKAGES for orphanage and package relay 51b3275cd1
  12. [log] add more logs related to orphan handling
    - Whenever a tx is erased. Allows somebody to see which transactions
      have been erased due to expiry/overflow, not just how many.
    - Whenever a tx is added to a peer's workset.
    - AcceptToMemoryPool when a tx is accepted, mirroring the one logged for
      a tx received from a peer. This allows someone to see all of the
      transactions that are accepted to mempool just by looking for ATMP logs.
    - MEMPOOLREJ when a tx is rejected, mirroring the one logged for
      a tx received from a peer. This allows someone to see all of the
      transaction rejections by looking at MEMPOOLREJ logs.
    3b8c17838a
  13. [doc] move comment about AlreadyHaveTx DoS score to the right place
    This comment isn't in the right place, as detection of a tx in
    recent_rejects would cause the function to exit much earlier.
    Move the comment to the right place and tweak the first sentence for
    accuracy.
    a3b55c94b9
  14. glozow force-pushed on Aug 29, 2023
  15. DrahtBot removed the label CI failed on Aug 29, 2023
  16. in test/functional/p2p_permissions.py:122 in a33dde1e41 outdated
    118@@ -119,14 +119,14 @@ def check_tx_relay(self):
    119             [tx],
    120             self.nodes[1],
    121             success=False,
    122-            reject_reason='{} from peer=0 was not accepted: txn-mempool-conflict'.format(txid)
    123+            reject_reason='{} (wtxid={}) from peer=0 was not accepted: txn-mempool-conflict'.format(txid, tx.getwtxid())
    


    brunoerg commented at 12:52 pm on August 30, 2023:
    nit: we could also change it to use f-strings instead of format given it touches this line.

    glozow commented at 2:53 pm on August 31, 2023:
    Will keep this in mind for the future. I’m just sticking with how this string was formatted previously.
  17. brunoerg approved
  18. brunoerg commented at 1:35 pm on August 30, 2023: contributor

    crACK a3b55c94b9ffde07386aa887149ddca91b364967

    Code looks great and m_recent_rejects check is done into AlreadyHaveTx so the local change makes sense for me. I just was in doubt if this comment could be in AlreadyHaveTx but seems ok to leave it in net_processing side.

  19. in src/txorphanage.cpp:90 in 3b8c17838a outdated
    83@@ -84,6 +84,8 @@ int TxOrphanage::EraseTxNoLock(const uint256& txid)
    84         m_orphan_list[old_pos] = it_last;
    85         it_last->second.list_pos = old_pos;
    86     }
    87+    const auto& wtxid = it->second.tx->GetWitnessHash();
    88+    LogPrint(BCLog::TXPACKAGES, "   removed orphan tx %s (wtxid=%s)\n", txid.ToString(), wtxid.ToString());
    89     m_orphan_list.pop_back();
    90     m_wtxid_to_orphan_it.erase(it->second.tx->GetWitnessHash());
    


    mzumsande commented at 3:28 pm on August 30, 2023:
    nit: could use the newly introduced wtxid in this line too.

    glozow commented at 1:22 pm on August 31, 2023:
    Will do in the next PR, as it touches this line :+1:

    glozow commented at 3:10 pm on September 14, 2023:
    Done in #28481
  20. mzumsande commented at 3:50 pm on August 30, 2023: contributor
    Code review ACK a3b55c94b9ffde07386aa887149ddca91b364967
  21. in src/net_processing.cpp:2926 in a3b55c94b9
    2918@@ -2919,9 +2919,15 @@ bool PeerManagerImpl::ProcessOrphanTx(Peer& peer)
    2919         const MempoolAcceptResult result = m_chainman.ProcessTransaction(porphanTx);
    2920         const TxValidationState& state = result.m_state;
    2921         const uint256& orphanHash = porphanTx->GetHash();
    2922+        const uint256& orphan_wtxid = porphanTx->GetWitnessHash();
    2923 
    2924         if (result.m_result_type == MempoolAcceptResult::ResultType::VALID) {
    2925-            LogPrint(BCLog::MEMPOOL, "   accepted orphan tx %s\n", orphanHash.ToString());
    2926+            LogPrint(BCLog::TXPACKAGES, "   accepted orphan tx %s (wtxid=%s)\n", orphanHash.ToString(), orphan_wtxid.ToString());
    2927+            LogPrint(BCLog::MEMPOOL, "AcceptToMemoryPool: peer=%d: accepted %s (wtxid=%s) (poolsz %u txn, %u kB)\n",
    


    jonatack commented at 5:03 pm on August 30, 2023:

    For added or updated logging, it would help the migration to severity level logging if LogPrintLevel instead of LogPrint was used, either with debug or trace levels depending on e.g. log frequency.

    For more context, LogPrintLevel may eventually become the principal or only logging macro, and at some point could be renamed to just Log, so using it for new logging would make the transition easier.


    glozow commented at 8:15 am on August 31, 2023:
    I’m not familiar with the migration you’re referring to, could you link a PR/issue where I can see the changes and motivation? It’s not clear to me why I should change this PR for it.

    jonatack commented at 12:31 pm on August 31, 2023:

    #25203 has been ongoing since a year and a half and several initial steps have been merged. The blocker since March has been #27231, though given how long it has remained open I’ll probably begin to work around it.

    By using the new macros, there is less to update later on, and you are able to decide now the log level per message of your added logging.


    glozow commented at 2:46 pm on August 31, 2023:
    I’m going to leave this as is. It can very easily be changed later on if we decide on a logging api. Doesn’t seem like there is consensus to do it in the #25203-style. This PR has 3 ACKs and would unblock package relay work.

    jonatack commented at 5:34 pm on August 31, 2023:

    Doesn’t seem like there is consensus to do it in the #25203-style.

    The suggested macro, LogPrintLevel, was merged 15 months ago in #24464.

  22. in src/logging.h:71 in a3b55c94b9
    67@@ -68,6 +68,7 @@ namespace BCLog {
    68         BLOCKSTORAGE = (1 << 26),
    69         TXRECONCILIATION = (1 << 27),
    70         SCAN        = (1 << 28),
    71+        TXPACKAGES  = (1 << 29),
    


    jonatack commented at 5:04 pm on August 30, 2023:

    A possible alternative to adding a category would be to distinguish these from the other mempool logs via debug vs trace levels, if applicable.

    User-facing naming suggestion, here and for "txpackages" in logging.cpp, categories tend to be named in the singular.

    0        TXPACKAGE  = (1 << 29),
    

    glozow commented at 2:52 pm on August 31, 2023:
    I prefer txpackages. There’s plenty of places where a log pertains to multiple packages.
  23. jonatack commented at 5:08 pm on August 30, 2023: member
    Concept ACK, happy to see improved or more useful logging.
  24. achow101 commented at 4:47 pm on August 31, 2023: member
    ACK a3b55c94b9ffde07386aa887149ddca91b364967
  25. achow101 merged this on Aug 31, 2023
  26. achow101 closed this on Aug 31, 2023

  27. glozow deleted the branch on Aug 31, 2023
  28. Frank-GER referenced this in commit e6450d7d8d on Sep 8, 2023
  29. Fabcien referenced this in commit 2de0e104d3 on Jun 10, 2024
  30. bitcoin locked this on Dec 5, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-22 00:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me