Rewrite DoS interface between validation and net_processing #15141

pull sdaftuar wants to merge 20 commits into bitcoin:master from sdaftuar:2019-01-rewrite-validation-interface changing 9 files +326 −223
  1. sdaftuar commented at 4:57 pm on January 10, 2019: member

    This is a rebase of #11639 with some fixes for the last few comments which were not yet addressed.

    The original PR text, with some strikethroughs of text that is no longer correct:

    This cleans up an old main-carryover - it made sense that main could decide what DoS scores to assign things because the DoS scores were handled in a different part of main, but now validation is telling net_processing what DoS scores to assign to different things, which is utter nonsense. Instead, we replace CValidationState’s nDoS and CorruptionPossible with a general ValidationInvalidReason, which net_processing can handle as it sees fit. I keep the behavior changes here to a minimum, but in the future we can utilize these changes for other smarter behavior, such as disconnecting/preferring to rotate outbound peers based on them providing things which are invalid due to SOFT_FORK because we shouldn’t ban for such cases.

    This is somewhat complementary with, though obviously conflicts heavily with #11523, which added enums in place of DoS scores, as well as a few other cleanups (which are still relevant).

    Compared with previous bans, the following changes are made:

    Txn with empty vin/vout or null prevouts move from 10 DoS points to 100. Loose transactions with a dependency loop now result in a ban instead of 10 DoS points. BIP68-violation no longer results in a ban as it is SOFT_FORK. Non-SegWit SigOp violation no longer results in a ban as it considers P2SH sigops and is thus SOFT_FORK. Any script violation in a block no longer results in a ban as it may be the result of a SOFT_FORK. This should likely be fixed in the future by differentiating between them. Proof of work failure moves from 50 DoS points to a ban. Blocks with timestamps under MTP now result in a ban, blocks too far in the future continue to not result in a ban. Inclusion of non-final transactions in a block now results in a ban instead of 10 DoS points.

    Note: The change to ban all peers for consensus violations is actually NOT the change I’d like to make – I’d prefer to only ban outbound peers in those situations. The current behavior is a bit of a mess, however, and so in the interests of advancing this PR I tried to keep the changes to a minimum. I plan to revisit the behavior in a followup PR.

    EDIT: One reviewer suggested I add some additional context for this PR:

    The goal of this work was to make net_processing aware of the actual reasons for validation failures, rather than just deal with opaque numbers instructing it to do something.

    In the future, I’d like to make it so that we use more context to decide how to punish a peer. One example is to differentiate inbound and outbound peer misbehaviors. Another potential example is if we’d treat RECENT_CONSENSUS_CHANGE failures differently (ie after the next consensus change is implemented), and perhaps again we’d want to treat some peers differently than others.

  2. laanwj added the label Validation on Jan 10, 2019
  3. DrahtBot commented at 6:31 pm on January 10, 2019: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #15681 ([mempool] Allow one extra single-ancestor transaction per package by TheBlueMatt)
    • #15505 ([p2p] Request NOTFOUND transactions immediately from other outbound peers, when possible by sdaftuar)
    • #15253 (Net: Consistently log outgoing INV messages by Empact)
    • #13868 (Remove unused fScriptChecks parameter from CheckInputs by Empact)
    • #13525 (Report reason inputs are nonstandard from AreInputsStandard by Empact)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. practicalswift commented at 6:34 pm on January 10, 2019: contributor
    Concept ACK
  5. laanwj added this to the "Blockers" column in a project

  6. sipa commented at 6:17 pm on January 11, 2019: member
    Concept ACK; I’ll review for changes in behavior for specific validation reasons later.
  7. laanwj commented at 2:23 pm on January 14, 2019: member
    Concept ACK
  8. in src/consensus/validation.h:77 in 94874ddfdf outdated
    73@@ -30,32 +74,24 @@ class CValidationState {
    74         MODE_INVALID, //!< network rule violation (DoS value may be set)
    75         MODE_ERROR,   //!< run-time error
    76     } mode;
    77-    int nDoS;
    78+    ValidationInvalidReason reason;
    


    jnewbery commented at 7:52 pm on January 14, 2019:
    nit: change to m_reason and avoid all the non-shadowing naming tricks below (reasonIn and _reason)

    sdaftuar commented at 9:40 pm on January 31, 2019:
    Fixed in latest commit.
  9. in src/consensus/validation.h:61 in 94874ddfdf outdated
    55+     * activation, or witness may have been malleated (which includes
    56+     * non-standard witnesses).
    57+     */
    58+    TX_WITNESS_MUTATED,
    59+    /**
    60+     * Tx already in mempool or conflicts with a tx in the chain
    


    jnewbery commented at 8:03 pm on January 14, 2019:
    nit: s/conflicts with a tx in the chain/conflicts with a confirmed transaction. Same comment below for “exists in the mempool or on chain”

    sdaftuar commented at 9:40 pm on January 31, 2019:

    I don’t really think “confirmed transaction” is any clearer than “tx in the chain” – if anything, the latter seems more specific to me, as “confirmed” is a concept that only makes sense in the context of the chain that you’re on, which “tx in the chain” is more explicit about.

    I’m going to leave this comment intact, pending other opinions.


    sipa commented at 8:56 pm on February 8, 2019:
    I think the current wording is fine.
  10. in src/validation.cpp:3344 in 94874ddfdf outdated
    3340@@ -3333,7 +3341,9 @@ static bool ContextualCheckBlock(const CBlock& block, CValidationState& state, c
    3341     // the block hash, so we couldn't mark the block as permanently
    3342     // failed).
    3343     if (GetBlockWeight(block) > MAX_BLOCK_WEIGHT) {
    3344-        return state.DoS(100, false, REJECT_INVALID, "bad-blk-weight", false, strprintf("%s : weight limit failed", __func__));
    3345+        // We can call this a consensus failure as any data-providers who provided
    


    jnewbery commented at 10:09 pm on January 14, 2019:

    This seems entirely obvious and not requiring a comment to me, which makes me think there’s some subtlety I’ve missed. Is this just saying that if we receive a block with witness data, it should be valid-according-to-BIP141?

    Pedantic nit: I’d also avoid talking about ‘data-providers’ in validation.cpp. After this PR, validation should be unconcerned with data-providers and only be validating blocks based on the block data.


    sdaftuar commented at 1:23 am on January 15, 2019:

    I believe this comment is contrasting a CONSENSUS failure from a RECENT_CONSENSUS_CHANGE – I think in @TheBlueMatt’s original PR he had some validation failures marked as RECENT_CONSENSUS_CHANGE, but eventually we decided to switch them all out (and reserve RECENT_CONSENSUS_CHANGE as something we might do in the future).

    I think I agree with you philosophically that validation ought not be very concerned with ‘data providers’, but I think the ValidationReasons interface is also driven by the needs of net_processing, so sometimes we may need to explain reasons that maybe don’t make sense in a totally neutral validation library because our application requires it. RECENT_CONSENSUS_CHANGE is one such possible example (though we’re not using it in this PR and I am not sure we ever will); the BLOCK_INVALID_HEADER enum I added is another (net_processing needs to be able to distinguish some headers failures from others, in order to maintain the current ban behavior).

    Anyway I’ll update this comment to be clearer.


    ajtowns commented at 5:31 am on January 15, 2019:
    I think this comment is justifying upgrading the (at the time recent) segwit test from a RECENT_CONSENSUS_CHANGE to just CONSENSUS_CHANGE, the reason being that either you’ve got an old client that didn’t provide segwit data – in which case this test won’t trigger because the bad-blk-length test will already have failed – or it is providing segwit data but doing it wrong, in which case there’s no reason to use the more forgiving RECENT version. So I think just dropping the comment (now that segwit isn’t recent) is fine, fwiw.
  11. in src/net_processing.cpp:982 in 94874ddfdf outdated
    837+        // building off an invalid or missing block -- are punished regardless
    838+        // (see below).
    839+        return !via_compact_block;
    840+    case ValidationInvalidReason::BLOCK_INVALID_HEADER:
    841+    case ValidationInvalidReason::BLOCK_CHECKPOINT:
    842+    case ValidationInvalidReason::BLOCK_INVALID_PREV:
    


    jnewbery commented at 10:42 pm on January 14, 2019:

    It’s unclear to me whether peers should always be punished for BLOCK_INVALID_PREV. For example, if the previous block was invalid because of RECENT_CONSENSUS_CHANGE and the peer wasn’t punished, should it be punished for relaying this descendant block?

    Should compact block peers be punished for relaying the block if its parent is invalid? My reading of https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#pre-validation-relay-and-consistency-considerations is that the answer is no.

    Same question for MaybePunishNode() below.


    ajtowns commented at 5:56 am on January 15, 2019:

    If miners have mostly upgraded then building on top of a RECENT_CONSENSUS_CHANGE block should be rare enough for this not to be a huge problem.

    If not, and we want to cope with a moderately controversial consensus upgrade, then we probably want to track whether blocks failed due to RECENT_CONSENSUS_CHANGE and mark their children as also failing due to RECENT_CONSENSUS_CHANGE (after checking PoW at least). Working all that out doesn’t seem necessary for this patchset to me though.


    sdaftuar commented at 6:07 pm on January 15, 2019:

    With respect to the issue you’re bringing up, I believe the behavior in this PR matches existing behavior, in which case I’d prefer to defer improvement to a future PR. If I’m missing some way that we’ve made things different or worse though let me know.

    As for BIP 152:

    A node MUST NOT send a cmpctblock message without having validated that the header properly commits to each transaction in the block, and properly builds on top of the existing, fully-validated chain with a valid proof-of-work either as a part of the current most-work valid chain, or building directly on top of it. A node MAY send a cmpctblock before validating that each transaction in the block validly spends existing UTXO set entries.

  12. jnewbery commented at 10:44 pm on January 14, 2019: member
    I’ve reviewed a5415e85c. A few nits/questions inline.
  13. in src/net_processing.cpp:1532 in 94874ddfdf outdated
    1528@@ -1453,6 +1529,7 @@ bool static ProcessHeadersMessage(CNode *pfrom, CConnman *connman, const std::ve
    1529                 // etc), and not just the duplicate-invalid case.
    1530                 pfrom->fDisconnect = true;
    1531             }
    1532+            MaybePunishNode(pfrom->GetId(), state, /*via_compact_block*/ false, "invalid header received");
    


    sdaftuar commented at 1:29 am on January 15, 2019:
    Self-review: I think adding this line here may be a bug. At any rate, there is a serious confusion between the hacky punish_invalid bool in the existing code and the introduction of MaybePunishNode in this PR that ought to be cleaned up.

    ajtowns commented at 2:48 pm on January 15, 2019:
    This replaces the Misbehaving(.., "invalid header received"); from earlier; shouldn’t be introducing a bug (unless the move to below the if introduces one)?

    sdaftuar commented at 5:54 pm on January 15, 2019:

    I believe there is an unintended behavior change here – previously, “duplicate invalid” headers were not assigned DoS points. We added a bunch of logic (just above this line of code) to punish outbound peers for providing invalid headers.

    After the rewrite in this PR, CACHED_INVALID is a bannable offense from any peer (other than in HB compact block relay).

    I’ll rework this…


    ryanofsky commented at 6:03 pm on February 12, 2019:

    re: #15141 (review)

    I’ll rework this…

    This is resolved now?

  14. in src/consensus/validation.h:83 in 6ee2c4551d outdated
    106@@ -117,14 +107,7 @@ class CValidationState {
    107     bool IsError() const {
    108         return mode == MODE_ERROR;
    109     }
    110-    bool CorruptionPossible() const {
    111-        return corruptionPossible;
    112-    }
    


    ajtowns commented at 6:07 am on January 15, 2019:
    Maybe having inline bool CorruptionPossible() const { return reason == BLOCK_MUTATED; } would make for nicer code elsewhere?
  15. in src/validation.cpp:906 in 6ee2c4551d outdated
    888@@ -889,7 +889,6 @@ static bool AcceptToMemoryPoolWorker(const CChainParams& chainparams, CTxMemPool
    889                 // Only the witness is missing, so the transaction itself may be fine.
    890                 state.Invalid(ValidationInvalidReason::TX_WITNESS_MUTATED, false,
    891                           state.GetRejectCode(), state.GetRejectReason(), state.GetDebugMessage());
    892-                state.SetCorruptionPossible();
    


    ajtowns commented at 12:57 pm on January 15, 2019:
    This change isn’t a clean refactor – !state.CorruptionPossible() would have returned false after this, but its replacement in this commit (ie, state.GetReason() != BLOCK_MUTATED) will return true. I think this is okay though, since CorruptionPossible() is only checked for block updates, and this just deals with mempool tx’s, and the uses of state.CorruptionPossible() that this would have effected were already changed to state.GetReason() != TX_WITNESS_MUTATED in an earlier commit.
  16. in src/blockencodings.cpp:206 in 963699d131 outdated
    202@@ -203,7 +203,7 @@ ReadStatus PartiallyDownloadedBlock::FillBlock(CBlock& block, const std::vector<
    203         // but that is expensive, and CheckBlock caches a block's
    204         // "checked-status" (in the CBlock?). CBlock should be able to
    205         // check its own merkle root and cache that check.
    206-        if (state.CorruptionPossible())
    207+        if (state.GetReason() == ValidationInvalidReason::BLOCK_MUTATED)
    


    ajtowns commented at 2:18 pm on January 15, 2019:
    Seems like this change could be squashed into “Remove references to CValidationState’s DoS and CorruptionPossible” ?
  17. ajtowns commented at 2:40 pm on January 15, 2019: member
    Looks good to me; haven’t fully looked through “Use new reason-based DoS/disconnect logic instead of state.nDoS” though.
  18. in src/consensus/tx_verify.cpp:163 in a5415e85ca outdated
    159@@ -160,24 +160,24 @@ bool CheckTransaction(const CTransaction& tx, CValidationState &state, bool fChe
    160 {
    161     // Basic checks that don't depend on any context
    162     if (tx.vin.empty())
    163-        return state.DoS(10, false, REJECT_INVALID, "bad-txns-vin-empty");
    164+        return state.DoS(10, ValidationInvalidReason::CONSENSUS, false, REJECT_INVALID, "bad-txns-vin-empty");
    


    ryanofsky commented at 7:30 pm on January 15, 2019:

    In commit “Add useful-for-dos “reason” field to CValidationState” (a5415e85caaf2f5a77d6bae9574bb6d21139ee34)

    Note: word-diff is useful here to review new function arguments:

    0git log -p -n1 -U0 --word-diff-regex=. a5415e85caaf2f5a77d6bae9574bb6d21139ee34
    
  19. in src/validation.cpp:724 in a5415e85ca outdated
    715@@ -716,27 +716,22 @@ static bool AcceptToMemoryPoolWorker(const CChainParams& chainparams, CTxMemPool
    716                               fSpendsCoinbase, nSigOpsCost, lp);
    717         unsigned int nSize = entry.GetTxSize();
    718 
    719-        // Check that the transaction doesn't have an excessive number of
    


    ryanofsky commented at 7:36 pm on January 15, 2019:

    In commit “Add useful-for-dos “reason” field to CValidationState” (a5415e85caaf2f5a77d6bae9574bb6d21139ee34):

    Why remove this comment?


    kallewoof commented at 5:45 am on March 5, 2019:
    Good question.

    Sjors commented at 1:37 pm on March 6, 2019:
    Note that MAX_BLOCK_SIGOPS has been renamed / replaced by MAX_STANDARD_TX_SIGOPS_COST as part of SegWit in 2b1f6f9ccf36f1e0a2c9d99154e1642f796d7c2b. In addition to this comment, MAX_BLOCK_SIGOPS is also still mentioned in the function test framework. But that doesn’t explain why the comment can be removed.

    sdaftuar commented at 3:16 pm on March 7, 2019:

    I think this comment is not very helpful. It was originally added in #4150, and in the review on that PR people complained that the phrasing in this comment is confusing (“invalid rather than merely non-standard” - huh?).

    If reviewers prefer it, then I can just improve this comment rather than delete it – it just seems to me like the code reads just fine on its own.


    ajtowns commented at 8:11 am on April 1, 2019:

    MAX_BLOCK_SIGOPS is replaced by MAX_BLOCK_SIGOPS_COST (which is just multiplied by the witness scale factor of 4) as part of segwit. MAX_STANDARD_TX_SIGOPS_COST is just a separate rule at the relay/mempool level saying “you have to use at least 5 tx’s to hit the sigop limit”.

    I agree that the comment’s just confusing given how we understand “invalid” (breaks consensus rules) vs “non-standard” (not interesting for mempool/relay, but not too strongly punished either).

  20. in src/validation.cpp:1994 in a5415e85ca outdated
    1987@@ -1988,11 +1988,17 @@ bool CChainState::ConnectBlock(const CBlock& block, CValidationState& state, CBl
    1988         {
    1989             CAmount txfee = 0;
    1990             if (!Consensus::CheckTxInputs(tx, state, view, pindex->nHeight, txfee)) {
    1991+                if (state.GetReason() == ValidationInvalidReason::TX_MISSING_INPUTS) {
    1992+                    // CheckTxInputs may return MISSING_INPUTS but we can't return that, as
    1993+                    // it's not defined for a block, so we reset the reason flag to CONSENSUS here.
    1994+                    state.DoS(state.GetDoS(), ValidationInvalidReason::CONSENSUS, false,
    


    ryanofsky commented at 7:44 pm on January 15, 2019:

    In commit “Add useful-for-dos “reason” field to CValidationState” (a5415e85caaf2f5a77d6bae9574bb6d21139ee34)

    This seems like it is doubling the state.nDoS level, in addition to updating the reason enum:

    https://github.com/bitcoin/bitcoin/blob/a5415e85caaf2f5a77d6bae9574bb6d21139ee34/src/consensus/validation.h#L96

    Would suggest replacing this change something more straightforward like state.UpdateReason(ValidationInvalidReason::CONSENSUS)

  21. in src/validation.cpp:2018 in a5415e85ca outdated
    2036+                if (state.GetReason() == ValidationInvalidReason::TX_NOT_STANDARD) {
    2037+                    // CheckInputs may return NOT_STANDARD for extra flags we passed,
    2038+                    // but we can't return that, as it's not defined for a block, so
    2039+                    // we reset the reason flag to CONSENSUS here.
    2040+                    // (note that this may not be the case until we add additional
    2041+                    // soft-fork flags to our script flags, in which case we  need to
    


    ryanofsky commented at 7:45 pm on January 15, 2019:

    In commit “Add useful-for-dos “reason” field to CValidationState” (a5415e85caaf2f5a77d6bae9574bb6d21139ee34)

    Extra space on this line

  22. in src/validation.cpp:2042 in a5415e85ca outdated
    2039+                    // we reset the reason flag to CONSENSUS here.
    2040+                    // (note that this may not be the case until we add additional
    2041+                    // soft-fork flags to our script flags, in which case we  need to
    2042+                    // be careful to differentiate RECENT_CONSENSUS_CHANGE and
    2043+                    // CONSENSUS)
    2044+                    state.DoS(state.GetDoS(), ValidationInvalidReason::CONSENSUS, false,
    


    ryanofsky commented at 7:47 pm on January 15, 2019:

    In commit “Add useful-for-dos “reason” field to CValidationState” (a5415e85caaf2f5a77d6bae9574bb6d21139ee34)

    This also seems to double state.nDoS.

  23. in src/validation.cpp:1989 in a5415e85ca outdated
    1987@@ -1988,11 +1988,17 @@ bool CChainState::ConnectBlock(const CBlock& block, CValidationState& state, CBl
    1988         {
    1989             CAmount txfee = 0;
    1990             if (!Consensus::CheckTxInputs(tx, state, view, pindex->nHeight, txfee)) {
    1991+                if (state.GetReason() == ValidationInvalidReason::TX_MISSING_INPUTS) {
    1992+                    // CheckTxInputs may return MISSING_INPUTS but we can't return that, as
    1993+                    // it's not defined for a block, so we reset the reason flag to CONSENSUS here.
    


    ryanofsky commented at 7:52 pm on January 15, 2019:

    In commit “Add useful-for-dos “reason” field to CValidationState” (a5415e85caaf2f5a77d6bae9574bb6d21139ee34)

    Is there a check for the requirement that MISSING_INPUTS is not used for a block? I would expect to see an assert(reason != MISSING_INPUTS) or assert(ValidForBlock(reason)) or something like that somewhere.

  24. in src/validation.cpp:3106 in a5415e85ca outdated
    3137+            return state.DoS(100, ValidationInvalidReason::CONSENSUS, false, REJECT_INVALID, "bad-cb-multiple", false, "more than one coinbase");
    3138 
    3139     // Check transactions
    3140-    for (const auto& tx : block.vtx)
    3141-        if (!CheckTransaction(*tx, state, true))
    3142-            return state.Invalid(false, state.GetRejectCode(), state.GetRejectReason(),
    


    ryanofsky commented at 7:57 pm on January 15, 2019:

    In commit “Add useful-for-dos “reason” field to CValidationState” (a5415e85caaf2f5a77d6bae9574bb6d21139ee34)

    Note: I guess this line used to set state.corruptionPossible = false but no longer does.

    https://github.com/bitcoin/bitcoin/blob/cebe910718ae4f099f292736192a4e725ad02b94/src/consensus/validation.h#L54-L58

    New way seems better.


    Sjors commented at 1:28 pm on March 6, 2019:

    So if I understand this correctly:

    • state.Invalid() just calls state.DoS() with level=0andcorruptionIn=false` (default).
    • CheckTransaction() can currently fail in various ways, calling:
      • state.DoS with:
        • level 10 or 100: why isn’t this higher level a problem?
        • corruptionIn not specified (so defaults to false)

    sdaftuar commented at 3:04 pm on March 7, 2019:

    level 10 or 100: why isn’t this higher level a problem? @sjors I don’t understand your question – can you rephrase?


    ajtowns commented at 9:29 am on April 1, 2019:
    If I understand correctly: the higher level (ie, changing the 10’s to 100’s in 96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5 - the “clean up banning levels” commit) isn’t a problem because these failures are all consensus ones, so any reasonable implementation shouldn’t be making them. (Except for immature coinbase and missing inputs at the mempool level which are downgraded elsewhere)
  25. in src/net_processing.cpp:1233 in 6bdc4491e0 outdated
    1084         if (it != mapBlockSource.end() && State(it->second.first) && state.GetRejectCode() > 0 && state.GetRejectCode() < REJECT_INTERNAL) {
    1085             CBlockReject reject = {(unsigned char)state.GetRejectCode(), state.GetRejectReason().substr(0, MAX_REJECT_MESSAGE_LENGTH), hash};
    1086             State(it->second.first)->rejects.push_back(reject);
    1087-            if (nDoS > 0 && it->second.second)
    1088-                Misbehaving(it->second.first, nDoS);
    1089+            MaybePunishNode(/*nodeid=*/ it->second.first, state, /*via_compact_block=*/ !it->second.second);
    


    ryanofsky commented at 8:21 pm on January 15, 2019:

    In commit “Use state reason field to check for collisions in cmpctblocks” (963699d1316f6b14c98a4624f766393379db85e1)

    Since the mapBlockSource bool is now being passed as !via_compact_block, it seems like the field description should mention something about setting it based on whether the source was a compact or full block:

    https://github.com/bitcoin/bitcoin/blob/6bdc4491e06433eb380ca3b8bc3e7c15f06aee8b/src/net_processing.cpp#L104-L105

  26. ryanofsky commented at 9:27 pm on January 15, 2019: member

    Started reviewing this, but IMO, the way this PR is structured makes it difficult to verify that it doesn’t unintentionally change behavior.

    I think a nicer way to write this would be to have one commit adding empty ValidationInvalidReason, MayResultInDisconnect, and MaybePunishNode definitions, and adding a state.Invalid() overload taking an optional ValidationInvalidReason argument. Then have a sequence of small followup commits which each add a few enum values at a time, passing them through state.Invalid() and translating them into Misbehaving() calls, where each commit is self contained and is deals with related reasons so it is easy to spot and understand changes in behavior.

    If this is a bad idea, or too much work, I’d be ok with trying to review this PR as it is, but I wanted to suggest something to be able to have more confidence in it, and to maybe make it easier to find other reviewers.

    • a5415e85caaf2f5a77d6bae9574bb6d21139ee34 Add useful-for-dos “reason” field to CValidationState (1/8)
    • 33213ad4ed9c5cef893285a7880ca708fb86a4ff Add functions to convert CValidationInterface’s reason to DoS info (2/8)
    • 6bdc4491e06433eb380ca3b8bc3e7c15f06aee8b Use new reason-based DoS/disconnect logic instead of state.nDoS (3/8)
    • 963699d1316f6b14c98a4624f766393379db85e1 Use state reason field to check for collisions in cmpctblocks (4/8)
    • 06e4247ede5d052c9680f9bacee6ec52b83cc097 Prep for scripted-diff by removing some \ns which annoy sed. (5/8)
    • 81318965979971dbcf04df5a216ee1a687d8173f scripted-diff: Remove DoS calls to CValidationState (6/8)
    • 6ee2c4551d055dd3c7bf28ed4bde7c566d75dfef Remove references to CValidationState’s DoS and CorruptionPossible (7/8)
    • 94874ddfdf4ae9c4b10f3f91d5e3280e2c24c371 Update some comments in validation.cpp as we arent doing DoS there (8/8)
  27. in src/test/txvalidation_tests.cpp:56 in a642744cc5 outdated
    52@@ -53,9 +53,8 @@ BOOST_FIXTURE_TEST_CASE(tx_mempool_reject_coinbase, TestChain100Setup)
    53     BOOST_CHECK(state.IsInvalid());
    54     BOOST_CHECK_EQUAL(state.GetRejectReason(), "coinbase");
    55 
    56-    int nDoS;
    57-    BOOST_CHECK_EQUAL(state.IsInvalid(nDoS), true);
    58-    BOOST_CHECK_EQUAL(nDoS, 100);
    59+    BOOST_CHECK_EQUAL(state.IsInvalid(), true);
    


    ajtowns commented at 1:19 pm on January 18, 2019:
    We checked state.IsInvalid() a couple of lines earlier, so this addition is redundant.
  28. ajtowns commented at 1:42 pm on January 18, 2019: member

    Started reviewing this, but IMO, the way this PR is structured makes it difficult to verify that it doesn’t unintentionally change behavior.

    FWIW, I’ve had a go at redoing the patchset to try to make the (potential) functionality changes more clear: https://github.com/ajtowns/bitcoin/commits/201901-dosreasons

    This has (I think) all the behaviour changes first:

    d9451de0d0 drop obsolete comment
    acdb469525 [refactor] stateDummy -> orphan_state
    5cd7a4d338 [refactor] Use maybepunish etc
    0d1d471eac [refactor] drop IsInvalid(nDoSOut)
    a0776a5d8a set nDoS rather than bumping it
    e0cff4e133 Clean up banning levels
    

    before introducing the new reason field, along with checks that the implied DoS value for each reason matches the actual DoS values presented/used:

    89e8dea284 [refactor] Add useful-for-dos "reason" field to CValidationState
    

    which then allows dropping the instance variables:

    27089e55be [refactor] Drop redundant nDoS, corruptionPossible, SetCorruptionPossible
    

    Then the code is changed to use reasons directly:

    15d9023106 LookupBlockIndex -> CACHED_INVALID
    519fb78934 CorruptionPossible -> TX_WITNESS_MUTATED
    221d17f332 CorruptionPossible -> BLOCK_MUTATED
    32747d0746 [refactor] Use Reasons directly instead of DoS codes
    

    And the now obsolete DoS/etc stuff is dropped:

    4d110a59c6 [refactor] Prep for scripted-diff by removing some \ns which annoy sed.
    9a89a47257 scripted-diff: Remove DoS calls to CValidationState
    327591b016 [refactor] Drop unused state.DoS(), state.GetDoS(), state.CorruptionPossible()
    

    That leaves a couple more things:

    94cf0deffb [refactor] Update some comments in validation.cpp as we arent doing DoS there
    04c6b24a66 [refactor] swap if/else order
    96f0ee075d remaining commits vs 94874ddfdf4ae9c4b10f3f91d5e3280e2c24c371
    

    but finally ends up with the same code as this PR (minus the latest commit anyway).

    Anyway I think this approach might be easier to review? It could also allow splitting the PR into two – one making the changes to DoS behaviour but not changing the way DoS works; followed by a second PR that actually adds the Reasons and refactors but doesn’t change behaviour.

    (Proof of concept only: bunches of these commits should probably be combined, commit messages need improvement, and I think I lost a bunch of authorship info)

    EDIT:

    I’ve added an extra commit prior to the DoS->Invalid refactor, namely “5b15205883 Allow use of state.Invalid() for all reasons” that avoids assertions that Invalid() is only used for DoS-level-0 problems failing.

    That just leaves one test failure in the intermediate commits; feature_block.py fails after the changing the DoS levels but before adding the “reason” code. I think this is due to lowering bad-txns-inputs-missingorspent from 100 to 0 with the tests still expecting a disconnect when that happens in a block. Adding the reasons fixes this because that includes code to update that problem from 0/TX_MISSING_INPUTS to 100/CONSENSUS when it affects a block rather than a loose transaction. (And similarly, the tests are adujsted to expect disconnects due to premature coinbase spends, but that functionality only occurs as part of the 0/TX to 100/CONS step)

  29. sdaftuar commented at 2:46 pm on January 18, 2019: member

    Thanks all for the review so far!

    I’d started taking a stab at rewriting this; I’ll continue with my approach to see how it ends up but @ajtowns thank you for your help – @ryanofsky if you have any thoughts on @ajtowns’s rework please let me know, happy to adapt his breakdown and include here if that approach looks good.

  30. ryanofsky commented at 3:12 pm on January 18, 2019: member

    @ryanofsky if you have any thoughts on @ajtowns’s rework please let me know

    Took a quick look, and I think ajtowns’s refactor is great. It’s a slightly different approach than I suggested in that the 32747d0746d91a8f63e39cedfb232f8c36b33bc6 commit which starts using reason codes is done all at once instead of incrementally as reasons are added, so it requires a little bit of grepping to verify, but this is easy to do and I think it’s a huge improvement.

    I think it would be best to use ajtown’s branch here, unless you’ve done a lot of work on your own already or see problems I’m missing.

  31. naumenkogs commented at 7:16 pm on January 21, 2019: member
    Concept ACK, I will take a closer look once the code is updated per comments above I guess.
  32. jnewbery removed this from the "Blockers" column in a project

  33. sdaftuar force-pushed on Jan 24, 2019
  34. sdaftuar commented at 6:26 pm on January 24, 2019: member

    I have redone this along the lines of @ajtowns branch, and cleaned up each commit (I think!) so that each one should be logically correct, pass tests, etc.

    I’ve saved the original version of this PR here: https://github.com/sdaftuar/bitcoin/commits/15141.original

    The diff between the two is pretty small (just some formatting changes that were getting tedious to resolve, and I removed a couple lines that some reviewers had commented on as being unnecessary):

     0diff --git a/src/consensus/tx_verify.cpp b/src/consensus/tx_verify.cpp
     1index fb04c1c0abf..a7b31ff7c56 100644
     2--- a/src/consensus/tx_verify.cpp
     3+++ b/src/consensus/tx_verify.cpp
     4@@ -221,8 +221,7 @@ bool Consensus::CheckTxInputs(const CTransaction& tx, CValidationState& state, c
     5 
     6         // If prev is coinbase, check that it's matured
     7         if (coin.IsCoinBase() && nSpendHeight - coin.nHeight < COINBASE_MATURITY) {
     8-            return state.Invalid(ValidationInvalidReason::TX_MISSING_INPUTS, false,
     9-                REJECT_INVALID, "bad-txns-premature-spend-of-coinbase",
    10+            return state.Invalid(ValidationInvalidReason::TX_MISSING_INPUTS, false, REJECT_INVALID, "bad-txns-premature-spend-of-coinbase",
    11                 strprintf("tried to spend coinbase at depth %d", nSpendHeight - coin.nHeight));
    12         }
    13 
    14diff --git a/src/consensus/validation.h b/src/consensus/validation.h
    15index daf8b9b87cc..09a5630a4f3 100644
    16--- a/src/consensus/validation.h
    17+++ b/src/consensus/validation.h
    18@@ -81,8 +81,8 @@ private:
    19 public:
    20     CValidationState() : mode(MODE_VALID), reason(ValidationInvalidReason::NONE), chRejectCode(0) {}
    21     bool Invalid(ValidationInvalidReason reasonIn, bool ret = false,
    22-             unsigned int chRejectCodeIn=0, const std::string &strRejectReasonIn="",
    23-             const std::string &strDebugMessageIn="") {
    24+            unsigned int chRejectCodeIn=0, const std::string &strRejectReasonIn="",
    25+            const std::string &strDebugMessageIn="") {
    26         reason = reasonIn;
    27         chRejectCode = chRejectCodeIn;
    28         strRejectReason = strRejectReasonIn;
    29diff --git a/src/test/txvalidation_tests.cpp b/src/test/txvalidation_tests.cpp
    30index 00fd7fef12a..aa30129361f 100644
    31--- a/src/test/txvalidation_tests.cpp
    32+++ b/src/test/txvalidation_tests.cpp
    33@@ -52,8 +52,6 @@ BOOST_FIXTURE_TEST_CASE(tx_mempool_reject_coinbase, TestChain100Setup)
    34     // Check that the validation state reflects the unsuccessful attempt.
    35     BOOST_CHECK(state.IsInvalid());
    36     BOOST_CHECK_EQUAL(state.GetRejectReason(), "coinbase");
    37-
    38-    BOOST_CHECK_EQUAL(state.IsInvalid(), true);
    39     BOOST_CHECK(state.GetReason() == ValidationInvalidReason::CONSENSUS);
    40 }
    41 
    42diff --git a/src/validation.cpp b/src/validation.cpp
    43index e1f562ffbfd..20759bf96e7 100644
    44--- a/src/validation.cpp
    45+++ b/src/validation.cpp
    46@@ -888,7 +888,7 @@ static bool AcceptToMemoryPoolWorker(const CChainParams& chainparams, CTxMemPool
    47                 !CheckInputs(tx, stateDummy, view, true, scriptVerifyFlags & ~SCRIPT_VERIFY_CLEANSTACK, true, false, txdata)) {
    48                 // Only the witness is missing, so the transaction itself may be fine.
    49                 state.Invalid(ValidationInvalidReason::TX_WITNESS_MUTATED, false,
    50-                          state.GetRejectCode(), state.GetRejectReason(), state.GetDebugMessage());
    51+                        state.GetRejectCode(), state.GetRejectReason(), state.GetDebugMessage());
    52             }
    53             return false; // state filled in by CheckInputs
    54         }
    55@@ -1980,7 +1980,7 @@ bool CChainState::ConnectBlock(const CBlock& block, CValidationState& state, CBl
    56                     // CheckTxInputs may return MISSING_INPUTS but we can't return that, as
    57                     // it's not defined for a block, so we reset the reason flag to CONSENSUS here.
    58                     state.Invalid(ValidationInvalidReason::CONSENSUS, false,
    59-                              state.GetRejectCode(), state.GetRejectReason(), state.GetDebugMessage());
    60+                            state.GetRejectCode(), state.GetRejectReason(), state.GetDebugMessage());
    61                 }
    62                 return error("%s: Consensus::CheckTxInputs: %s, %s", __func__, tx.GetHash().ToString(), FormatStateMessage(state));
    63             }
    64@@ -3341,8 +3341,6 @@ static bool ContextualCheckBlock(const CBlock& block, CValidationState& state, c
    65     // the block hash, so we couldn't mark the block as permanently
    66     // failed).
    67     if (GetBlockWeight(block) > MAX_BLOCK_WEIGHT) {
    68-        // We can call this a consensus failure as any data-providers who provided
    69-        // us with witness data can be expected to support SegWit validation.
    70         return state.Invalid(ValidationInvalidReason::CONSENSUS, false, REJECT_INVALID, "bad-blk-weight", strprintf("%s : weight limit failed", __func__));
    71     }
    

    Also if this version is not actually easier to review I’m happy to go back to the original or try another approach.

  35. jnewbery added this to the "Blockers" column in a project

  36. in src/validation.cpp:3123 in 94c2cdb880 outdated
    3120-        if (!CheckTransaction(*tx, state, true))
    3121-            return state.Invalid(false, state.GetRejectCode(), state.GetRejectReason(),
    3122-                                 strprintf("Transaction check failed (tx hash %s) %s", tx->GetHash().ToString(), state.GetDebugMessage()));
    3123+    for (const auto& tx : block.vtx) {
    3124+        if (!CheckTransaction(*tx, state, true)) {
    3125+            LogPrintf("Transaction check failed (tx hash %s) %s\n", tx->GetHash().ToString(), state.GetDebugMessage());
    


    ryanofsky commented at 10:22 pm on January 24, 2019:

    In commit “Check transactions just logs a message” (94c2cdb88049af5283a7c1f52ea6e52ac2946686)

    Could you update the commit message to say whether this commit changes behavior at all, and what the motivation is? At first glance it seems like this probably doesn’t change behavior, and the only motivation is to simplify code. But I could easily be missing something.


    sdaftuar commented at 4:50 pm on January 29, 2019:
    Done

    ryanofsky commented at 7:35 pm on March 5, 2019:

    re: #15141 (review)

    Done

    Thanks, new commit is “Remove redundant state.Invalid() call after CheckTransaction()” (59ff8e67c2c62ec11d76d3d1b54dc4829363ad5e)

  37. in src/net_processing.cpp:826 in 8226bed419 outdated
    821+}
    822+
    823+static bool MaybePunishNode(NodeId nodeid, const CValidationState& state, bool via_compact_block, const std::string& message = "") {
    824+    int nDoS = state.GetDoS();
    825+    if (nDoS > 0 && !via_compact_block) {
    826+         LOCK(cs_main);
    


    ryanofsky commented at 10:31 pm on January 24, 2019:

    In commit “[refactor] Use maybepunish etc” (8226bed4191a50129ac6fdbcb8fad5e1c6b7cacd)

    Note: This acquires lock recursively in PeerLogicValidation::BlockChecked. Seems fine, but just wanted to note it wasn’t happening before.

  38. in src/validation.cpp:1434 in e534b0b78b outdated
    1431-                    // invalid in new blocks, e.g. an invalid P2SH. We DoS ban
    1432-                    // such nodes as they are not following the protocol. That
    1433-                    // said during an upgrade careful thought should be taken
    1434-                    // as to the correct behavior - we may want to continue
    1435-                    // peering with non-upgraded nodes even after soft-fork
    1436-                    // super-majority signaling has occurred.
    


    ryanofsky commented at 10:46 pm on January 24, 2019:

    In commit “[refactor] Update some comments in validation.cpp as we arent doing DoS there” (e534b0b78bec49750421b5f52012b857df197e24)

    Why remove this comment entirely?


    sdaftuar commented at 4:51 pm on January 29, 2019:
    Updated with a new comment.

    ryanofsky commented at 7:36 pm on March 5, 2019:

    re: #15141 (review)

    Updated with a new comment.

    Thanks, new commit is “[refactor] Update some comments in validation.cpp as we arent doing DoS there” (9b7978efe3d127fa7833d6561a9d053c6820dc1b)

  39. ryanofsky commented at 10:49 pm on January 24, 2019: member

    Started review (will update list below with progress).

    • 59ff8e67c2c62ec11d76d3d1b54dc4829363ad5e Remove redundant state.Invalid() call after CheckTransaction() (1/27)
    • e7edc15f86fcd247d92273cb7ea094d59e1faefa drop obsolete comment (2/27)
    • f3fd64cc049a2056b2fcbed136ff9e487db57a25 [refactor] stateDummy -> orphan_state (3/27)
    • 4159f7ca7b449195f0e8a3f67a4045409c703e9d [refactor] Use maybepunish etc (4/27)
    • bfa94c76c607c741e1eca7ffdd1bef99271ea37d Update comment to reference MaybePunishNode (5/27)
    • 70906c5b410027919e7eda2932093be3e69b18f3 [refactor] drop IsInvalid(nDoSOut) (6/27)
    • 858bae6104ba7d16a637920d7802bb4be4c64994 set nDoS rather than bumping it (7/27)
    • 96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5 Clean up banning levels (8/27)
    • 6e27c500f254ffb009cb54a06bb9861585c0d126 === end of functionality changes (9/27)
    • ac3873e2a92457995f7e5a9e5fc24352af360c6b [refactor] Add useful-for-dos “reason” field to CValidationState (10/27)
    • bee1d4f5e29c8c447ac47a608240b38216750072 TX_MISSING_INPUTS now has a DoS score of 0 (11/27)
    • 95d7de9ab85f41aafed8a3cccbd09481497a5cb8 ==== start switch to reasons (12/27)
    • 6e3332a76f227b6dd6068513807934fd1b3d936b [refactor] Drop redundant nDoS, corruptionPossible, SetCorruptionPossible (13/27)
    • c558ebaa6d02154eaf762a28d2a7e954acee0661 LookupBlockIndex -> CACHED_INVALID (14/27)
    • 8ed1801e06bcac1ce5dce594a9bc548db3b54fc2 CorruptionPossible -> TX_WITNESS_MUTATED (15/27)
    • 3048533275227e67ce22931c6360513bddbd1767 CorruptionPossible -> BLOCK_MUTATED (16/27)
    • 346699322ca820c5d95c255386df3ce1fb1f3d11 [refactor] Use Reasons directly instead of DoS codes (17/27)
    • 9dd6fc18658b36b63b9f264676ac484879597b83 Fix handling of invalid headers (18/27)
    • 5e85f548fb305dde9de0a4f9a309ef1ff9f5b764 ==== drop nDoS info (19/27)
    • e5f43f3239dc60680646b79f65e9cde745abd760 Allow use of state.Invalid() for all reasons (20/27)
    • 5507feabe7bfc8fe599c0505cb64bf33ddb0ded6 [refactor] Prep for scripted-diff by removing some \ns which annoy sed. (21/27)
    • c664daf1530f58feb1a1fccd2e5ed80563389126 scripted-diff: Remove DoS calls to CValidationState (22/27)
    • 8e4590e522d4903d970cdaafb95e4fdfccf792fb [refactor] Drop unused state.DoS(), state.GetDoS(), state.CorruptionPossible() (23/27)
    • f494f78a1a8b82cef5e908588ec362296dba2188 ==== cleanup (24/27)
    • 9b7978efe3d127fa7833d6561a9d053c6820dc1b [refactor] Update some comments in validation.cpp as we arent doing DoS there (25/27)
    • 99d96897b2eb7300cc09389edaf2537f4d45f95b [refactor] swap if/else order (26/27)
    • 7682566acd7b04aee9425772823e77f230268ad8 nit: reason -> m_reason (27/27)
  40. sdaftuar force-pushed on Jan 29, 2019
  41. sdaftuar commented at 4:52 pm on January 29, 2019: member
    I addressed @ryanofsky’s comments so far (which rewrote the git history, since one of the commit messages changed, so I also squashed in a comment change as well). Previous version of this PR is now here: https://github.com/sdaftuar/bitcoin/commits/15141.1.
  42. DrahtBot added the label Needs rebase on Feb 8, 2019
  43. sdaftuar commented at 5:57 pm on February 8, 2019: member
    This needs a simple rebase, but can I get concept ACK/NACK from more reviewers on whether the reworked form of this PR (which broke things up into many more commits) is preferable compared to the original formulation?
  44. sipa commented at 6:43 pm on February 8, 2019: member
    I haven’t reviewed the last few commits yet (only up to “[refactor] Use Reasons directly instead of DoS codes”), but so far the structure is very clear. Concept ACK on that.
  45. sdaftuar force-pushed on Feb 8, 2019
  46. sdaftuar commented at 7:31 pm on February 8, 2019: member
    Thanks @sipa. Rebased. Prior version is here: 15141.2
  47. DrahtBot removed the label Needs rebase on Feb 8, 2019
  48. sipa commented at 8:19 pm on February 8, 2019: member
    One overall comment: it seems there is a subset of ValidationInvalidReasons that are valid for transactions, and another subset that is valid for blocks. Perhaps it’s useful to have functions to test whether one belongs to those sets, and invoke those functions in assertions after validation returns in their respective contexts. That seems a bit more future-proof than just having comments of the form “CheckTxInputs may return MISSING_INPUTS but we can’t return that”. It would make me also a bit more comfortable with changes to checks from CorruptionPossible() to testing for a specific invalidity reason (assuming we know TX_WITNESS_MUTATED in the only tx-valid corruptionpossible one, and BLOCK_MUTATED the only block-valid corruptionpossible one).
  49. in src/net_processing.cpp:359 in 7682566acd outdated
    355+
    356+    //! Whether this peer is a manual connection
    357+    bool m_is_manual_connection;
    358+
    359+    CNodeState(CAddress addrIn, std::string addrNameIn, bool is_inbound, bool is_manual) :
    360+        address(addrIn), name(addrNameIn), m_is_inbound(is_inbound),
    


    sipa commented at 8:58 pm on February 8, 2019:
    Nit: you can use name(std::move(addrNameIn)) here to avoid a copy.

    sdaftuar commented at 4:58 pm on March 2, 2019:
    Fixed.
  50. in src/net_processing.cpp:1018 in 7682566acd outdated
    1018+            }
    1019+
    1020+            // Disconnect outbound (but not inbound) peers if on an invalid chain.
    1021+            // Exempt HB compact block peers and manual connections.
    1022+            if (!via_compact_block && !node_state->m_is_inbound && !node_state->m_is_manual_connection) {
    1023+                Misbehaving(nodeid, 100, message);
    


    sipa commented at 9:05 pm on February 8, 2019:
    The comment says “disconnect”, but the DoS score will also cause a ban here. Is that intentional? (it seems it’s retaining existing behavior, so I assume it is).

    sdaftuar commented at 3:28 pm on February 10, 2019:

    This is actually a behavior change from existing behavior, but hopefully a relatively harmless one. Here’s the relevant snippet from master:

    https://github.com/bitcoin/bitcoin/blob/2945492424934fa360f86b116184ee8e34f19d0a/src/net_processing.cpp#L1552-L1585

    It’s a bit hard to decipher because of the multiple layers going on here, but basically punish_duplicate_invalid is only set to true for outbound and non-manual peers relaying us headers outside of HB compact block mode, and nDoS is set to 0 for the cached-invalid case:

    https://github.com/bitcoin/bitcoin/blob/2945492424934fa360f86b116184ee8e34f19d0a/src/validation.cpp#L3341-L3357

    So this does indeed result in a ban rather than just a disconnect for an outbound peer that announces an invalid header. If there’s no reason that this is materially worse than other ban behaviors that exist, then I think I’d prefer to stick with this behavior change for now, and try to improve ban-behavior globally after this PR (which should become much easier, now that banning is contained to one place in the code in a more understandable way).

    I can update the comment though to make this clearer.


    ryanofsky commented at 10:02 pm on February 11, 2019:

    re: #15141 (review)

    Note: Thread pertains to commit “Fix handling of invalid headers” (9dd6fc18658b36b63b9f264676ac484879597b83)


    TheBlueMatt commented at 7:21 pm on February 21, 2019:
    This appears to be correct to me. Obviously should update the comment to note this.

    sdaftuar commented at 5:01 pm on March 2, 2019:
    Fixed comment in latest commit.
  51. MarcoFalke deleted a comment on Feb 10, 2019
  52. in src/consensus/tx_verify.cpp:163 in 96cedc8d0c outdated
    159@@ -160,9 +160,9 @@ bool CheckTransaction(const CTransaction& tx, CValidationState &state, bool fChe
    160 {
    161     // Basic checks that don't depend on any context
    162     if (tx.vin.empty())
    163-        return state.DoS(10, false, REJECT_INVALID, "bad-txns-vin-empty");
    164+        return state.DoS(100, false, REJECT_INVALID, "bad-txns-vin-empty");
    


    ryanofsky commented at 8:55 pm on February 11, 2019:

    In commit “Clean up banning levels” (96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5)

    Note: this is “Txn with empty vin/vout or null prevouts move from 10 DoS points to 100”

  53. in src/consensus/tx_verify.cpp:165 in 96cedc8d0c outdated
    159@@ -160,9 +160,9 @@ bool CheckTransaction(const CTransaction& tx, CValidationState &state, bool fChe
    160 {
    161     // Basic checks that don't depend on any context
    162     if (tx.vin.empty())
    163-        return state.DoS(10, false, REJECT_INVALID, "bad-txns-vin-empty");
    164+        return state.DoS(100, false, REJECT_INVALID, "bad-txns-vin-empty");
    165     if (tx.vout.empty())
    166-        return state.DoS(10, false, REJECT_INVALID, "bad-txns-vout-empty");
    167+        return state.DoS(100, false, REJECT_INVALID, "bad-txns-vout-empty");
    


    ryanofsky commented at 8:55 pm on February 11, 2019:

    In commit “Clean up banning levels” (96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5)

    Note: this is “Txn with empty vin/vout or null prevouts move from 10 DoS points to 100”

  54. in src/consensus/tx_verify.cpp:202 in 96cedc8d0c outdated
    198@@ -199,7 +199,7 @@ bool CheckTransaction(const CTransaction& tx, CValidationState &state, bool fChe
    199     {
    200         for (const auto& txin : tx.vin)
    201             if (txin.prevout.IsNull())
    202-                return state.DoS(10, false, REJECT_INVALID, "bad-txns-prevout-null");
    203+                return state.DoS(100, false, REJECT_INVALID, "bad-txns-prevout-null");
    


    ryanofsky commented at 8:56 pm on February 11, 2019:

    In commit “Clean up banning levels” (96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5)

    Note: this is “Txn with empty vin/vout or null prevouts move from 10 DoS points to 100”

  55. in src/consensus/tx_verify.cpp:225 in 96cedc8d0c outdated
    220@@ -221,8 +221,8 @@ bool Consensus::CheckTxInputs(const CTransaction& tx, CValidationState& state, c
    221 
    222         // If prev is coinbase, check that it's matured
    223         if (coin.IsCoinBase() && nSpendHeight - coin.nHeight < COINBASE_MATURITY) {
    224-            return state.Invalid(false,
    225-                REJECT_INVALID, "bad-txns-premature-spend-of-coinbase",
    226+            return state.DoS(100, false,
    227+                REJECT_INVALID, "bad-txns-premature-spend-of-coinbase", false,
    


    ryanofsky commented at 8:56 pm on February 11, 2019:

    In commit “Clean up banning levels” (96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5)

    Note: this is “Inclusion of a premature coinbase spend now results in a ban”

  56. in src/validation.cpp:766 in 96cedc8d0c outdated
    762@@ -763,7 +763,7 @@ static bool AcceptToMemoryPoolWorker(const CChainParams& chainparams, CTxMemPool
    763             const uint256 &hashAncestor = ancestorIt->GetTx().GetHash();
    764             if (setConflicts.count(hashAncestor))
    765             {
    766-                return state.DoS(10, false,
    767+                return state.DoS(100, false,
    


    ryanofsky commented at 8:56 pm on February 11, 2019:

    In commit “Clean up banning levels” (96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5)

    Note: this is “Loose transactions with a dependency loop now result in a ban instead of 10 DoS points”

  57. in src/validation.cpp:3064 in 96cedc8d0c outdated
    3060@@ -3061,7 +3061,7 @@ static bool CheckBlockHeader(const CBlockHeader& block, CValidationState& state,
    3061 {
    3062     // Check proof of work matches claimed amount
    3063     if (fCheckPOW && !CheckProofOfWork(block.GetHash(), block.nBits, consensusParams))
    3064-        return state.DoS(50, false, REJECT_INVALID, "high-hash", false, "proof of work failed");
    3065+        return state.DoS(100, false, REJECT_INVALID, "high-hash", false, "proof of work failed");
    


    ryanofsky commented at 8:57 pm on February 11, 2019:

    In commit “Clean up banning levels” (96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5)

    Note: this is “Proof of work failure moves from 50 DoS points to a ban”

  58. in src/validation.cpp:3233 in 96cedc8d0c outdated
    3229@@ -3230,7 +3230,7 @@ static bool ContextualCheckBlockHeader(const CBlockHeader& block, CValidationSta
    3230 
    3231     // Check timestamp against prev
    3232     if (block.GetBlockTime() <= pindexPrev->GetMedianTimePast())
    3233-        return state.Invalid(false, REJECT_INVALID, "time-too-old", "block's timestamp is too early");
    3234+        return state.DoS(100, false, REJECT_INVALID, "time-too-old", false, "block's timestamp is too early");
    


    ryanofsky commented at 8:57 pm on February 11, 2019:

    In commit “Clean up banning levels” (96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5)

    Note: this is “Blocks with timestamps under MTP now result in a ban”

  59. in src/validation.cpp:3244 in 96cedc8d0c outdated
    3240@@ -3241,7 +3241,7 @@ static bool ContextualCheckBlockHeader(const CBlockHeader& block, CValidationSta
    3241     if((block.nVersion < 2 && nHeight >= consensusParams.BIP34Height) ||
    3242        (block.nVersion < 3 && nHeight >= consensusParams.BIP66Height) ||
    3243        (block.nVersion < 4 && nHeight >= consensusParams.BIP65Height))
    3244-            return state.Invalid(false, REJECT_OBSOLETE, strprintf("bad-version(0x%08x)", block.nVersion),
    3245+            return state.DoS(100, false, REJECT_OBSOLETE, strprintf("bad-version(0x%08x)", block.nVersion), false,
    


    ryanofsky commented at 8:58 pm on February 11, 2019:

    In commit “Clean up banning levels” (96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5)

    Note: this is “Any pre-segwit soft-fork errors (ie all soft-fork errors) now result in a ban”

  60. in src/validation.cpp:3274 in 96cedc8d0c outdated
    3270@@ -3271,7 +3271,7 @@ static bool ContextualCheckBlock(const CBlock& block, CValidationState& state, c
    3271     // Check that all transactions are finalized
    3272     for (const auto& tx : block.vtx) {
    3273         if (!IsFinalTx(*tx, nHeight, nLockTimeCutoff)) {
    3274-            return state.DoS(10, false, REJECT_INVALID, "bad-txns-nonfinal", false, "non-final transaction");
    3275+            return state.DoS(100, false, REJECT_INVALID, "bad-txns-nonfinal", false, "non-final transaction");
    


    ryanofsky commented at 8:59 pm on February 11, 2019:

    In commit “Clean up banning levels” (96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5)

    Note: this is “Inclusion of non-final transactions in a block now results in a ban”

  61. in src/validation.cpp:904 in ac3873e2a9 outdated
    900@@ -901,7 +901,8 @@ static bool AcceptToMemoryPoolWorker(const CChainParams& chainparams, CTxMemPool
    901             if (!tx.HasWitness() && CheckInputs(tx, stateDummy, view, true, scriptVerifyFlags & ~(SCRIPT_VERIFY_WITNESS | SCRIPT_VERIFY_CLEANSTACK), true, false, txdata) &&
    902                 !CheckInputs(tx, stateDummy, view, true, scriptVerifyFlags & ~SCRIPT_VERIFY_CLEANSTACK, true, false, txdata)) {
    903                 // Only the witness is missing, so the transaction itself may be fine.
    904-                state.SetCorruptionPossible();
    905+                state.DoS(0, ValidationInvalidReason::TX_WITNESS_MUTATED, false,
    


    ryanofsky commented at 9:17 pm on February 11, 2019:

    In commit “[refactor] Add useful-for-dos “reason” field to CValidationState” (ac3873e2a92457995f7e5a9e5fc24352af360c6b)

    Passing state.GetDoS() instead of 0 might make it clearer behavior isn’t changing.


    jnewbery commented at 9:53 pm on April 15, 2019:

    I agree that this is confusing. IMO, even better than passing in state.GetDoS() to state.DoS() would be:

    0state.Invalid(ValidationInvalidReason::TX_WITNESS_MUTATED, false, state.GetRejectCode(), state.GetRejectReason(), state.GetDebugMessage());
    1state.SetCorruptionPossible();
    
  62. in src/consensus/validation.h:147 in bee1d4f5e2 outdated
    143             return 10;
    144         case ValidationInvalidReason::CACHED_INVALID:
    145         case ValidationInvalidReason::RECENT_CONSENSUS_CHANGE:
    146         case ValidationInvalidReason::BLOCK_BAD_TIME:
    147         case ValidationInvalidReason::TX_NOT_STANDARD:
    148+        case ValidationInvalidReason::TX_MISSING_INPUTS:
    


    ryanofsky commented at 9:33 pm on February 11, 2019:

    In commit “TX_MISSING_INPUTS now has a DoS score of 0” (bee1d4f5e29c8c447ac47a608240b38216750072)

    Not sure, but it seems like it might be nice to have a comment here saying TX_MISSING_INPUTS reason will change to CONSENSUS if the transaction is included in a block, and lead to a higher dos score in that case.

  63. in src/net_processing.cpp:1561 in c558ebaa6d outdated
    1557@@ -1558,7 +1558,7 @@ bool static ProcessHeadersMessage(CNode *pfrom, CConnman *connman, const std::ve
    1558     CBlockHeader first_invalid_header;
    1559     if (!ProcessNewBlockHeaders(headers, state, chainparams, &pindexLast, &first_invalid_header)) {
    1560         if (state.IsInvalid()) {
    1561-            if (punish_duplicate_invalid && LookupBlockIndex(first_invalid_header.GetHash())) {
    1562+            if (punish_duplicate_invalid && state.GetReason() == ValidationInvalidReason::CACHED_INVALID) {
    


    ryanofsky commented at 9:52 pm on February 11, 2019:

    In commit “LookupBlockIndex -> CACHED_INVALID” (c558ebaa6d02154eaf762a28d2a7e954acee0661)

    Note: behavior should be unchanged here because ProcessNewBlockHeaders calls AcceptBlockHeader which sets CACHED_INVALID if a header is already known and has BLOCK_FAILED_MASK is set. If the first invalid header is invalid for a different reason the check shouldn’t trigger either before or after this change.

  64. ryanofsky commented at 10:04 pm on February 11, 2019: member
    Mostly finished reviewing this, just a few more commits left: #15141#pullrequestreview-196279133
  65. in src/net_processing.cpp:983 in 346699322c outdated
    986+        return false;
    987+    }
    988+    return false;
    989 }
    990 
    991+//! Returns true if the peer was punished (probably disconnected)
    


    ryanofsky commented at 5:02 pm on February 12, 2019:

    In commit “[refactor] Use Reasons directly instead of DoS codes” (346699322ca820c5d95c255386df3ce1fb1f3d11)

    Would be good to add a comment here saying that if this function is changed, then the MayResultInDisconnect function above should also be updated.


    sdaftuar commented at 5:27 pm on March 2, 2019:
    Is this still relevant after the latest changes?

    ryanofsky commented at 6:53 pm on March 5, 2019:

    re: #15141 (review)

    Is this still relevant after the latest changes?

    Not important. Maybe you could say a comment would be more helpful now that the functions don’t look related. But it’s not a big deal if there’s no reminder to check sending behavior when you change receiving behavior.


    sdaftuar commented at 7:21 pm on March 7, 2019:
    Done
  66. in src/net_processing.cpp:995 in 346699322c outdated
    993-    int nDoS = state.GetDoS();
    994-    if (nDoS > 0 && !via_compact_block) {
    995-         LOCK(cs_main);
    996-         Misbehaving(nodeid, nDoS, message);
    997-         return true;
    998+    switch (state.GetReason()) {
    


    ryanofsky commented at 5:07 pm on February 12, 2019:

    In commit “[refactor] Use Reasons directly instead of DoS codes” (346699322ca820c5d95c255386df3ce1fb1f3d11)

    This change seems like a step backwards to me. It seems cleaner and less error prone to keep MaybePunishNode and MayResultInDisconnect both implemented in terms of the same nice GetDoSForReason function added in ac3873e2a92457995f7e5a9e5fc24352af360c6b (renamed to GetDos in 6e3332a76f227b6dd6068513807934fd1b3d936b), than to duplicate the logic between these two functions and duplicate LOCK/Misbehaving stuff within this function.

    I would drop this commit entirely. The only parts of it that seem worth keeping are the added function comments, and these would make more sense as part of 4159f7ca7b449195f0e8a3f67a4045409c703e9d which adds the functions. Keeping GetDos would also reduce the churn in this PR and simplify the later 8e4590e522d4903d970cdaafb95e4fdfccf792fb commit.


    sdaftuar commented at 5:19 pm on March 2, 2019:

    The goal of this work was to make net_processing aware of the actual reasons for validation failures, rather than just deal with opaque numbers instructing it to do something.

    In the future, I’d like to make it so that we use more context to decide how to punish a peer. One example is to differentiate inbound and outbound peer misbehaviors. Another potential example is if we’d treat RECENT_CONSENSUS_CHANGE failures differently (ie after the next consensus change is implemented), and perhaps again we’d want to treat some peers differently than others.

    I do think that the MayResultInDisconnect logic was not very useful, and in the latest version I’ve vastly simplified things to reflect that we only use this for deciding whether to relay transactions from whitelisted peers. So there should be much less code duplication now.


    ryanofsky commented at 6:53 pm on March 5, 2019:

    re: #15141 (review)

    Thanks, this makes sense and the simplification fixes all the parts I didn’t like.


    jonatack commented at 11:37 am on March 6, 2019:

    re: #15141 (review)

    Possibly consider adding this helpful context information to the PR description.


    sdaftuar commented at 7:18 pm on March 7, 2019:
    Done
  67. in src/net_processing.cpp:951 in 346699322c outdated
    946@@ -947,17 +947,86 @@ void Misbehaving(NodeId pnode, int howmuch, const std::string& message) EXCLUSIV
    947         LogPrint(BCLog::NET, "%s: %s peer=%d (%d -> %d)%s\n", __func__, state->name, pnode, state->nMisbehavior-howmuch, state->nMisbehavior, message_prefixed);
    948 }
    949 
    950+/**
    951+ * Returns true if the given validation state result may result in us banning/disconnecting a peer
    


    ryanofsky commented at 5:24 pm on February 12, 2019:

    In commit “[refactor] Use Reasons directly instead of DoS codes” (346699322ca820c5d95c255386df3ce1fb1f3d11)

    It would be clearer to write “may result in a peer banning/disconnecting us” than “may result in us banning/disconnecting a peer.” Otherwise the next sentence makes less sense, and the overall comment is confusing about how the function is used.


    ryanofsky commented at 7:38 pm on March 5, 2019:

    re: #15141 (review)

    It would be clearer to write

    Still might be nice to make this change


    Sjors commented at 1:44 pm on March 6, 2019:
    Not sure about what to change here, but it’s helpful to clarify the role of whitelisting peers better.

    sdaftuar commented at 7:21 pm on March 7, 2019:
    Done in latest commit.
  68. in src/net_processing.cpp:1003 in 346699322c outdated
    1005+            LOCK(cs_main);
    1006+            Misbehaving(nodeid, 100, message);
    1007+            return true;
    1008+        }
    1009+        break;
    1010+    // Handled elsewhere for now
    


    ryanofsky commented at 6:14 pm on February 12, 2019:

    In commit “[refactor] Use Reasons directly instead of DoS codes” (346699322ca820c5d95c255386df3ce1fb1f3d11)

    Note: “elsewhere” refers to the CACHED_INVALID check in ProcessHeadersMessage added in earlier commit “LookupBlockIndex -> CACHED_INVALID” (c558ebaa6d02154eaf762a28d2a7e954acee0661).

    I would suggest clarifying this, but this is all replaced in the next commit “Fix handling of invalid headers” (9dd6fc18658b36b63b9f264676ac484879597b83), so it doesn’t matter much.


    sdaftuar commented at 7:24 pm on March 7, 2019:
    Yeah I was just going to skip this because it gets rewritten in a later commit.
  69. in src/net_processing.cpp:1563 in 9dd6fc1865 outdated
    1648@@ -1627,41 +1649,7 @@ bool static ProcessHeadersMessage(CNode *pfrom, CConnman *connman, const std::ve
    1649     CBlockHeader first_invalid_header;
    1650     if (!ProcessNewBlockHeaders(headers, state, chainparams, &pindexLast, &first_invalid_header)) {
    1651         if (state.IsInvalid()) {
    1652-            if (punish_duplicate_invalid && state.GetReason() == ValidationInvalidReason::CACHED_INVALID) {
    1653-                // Goal: don't allow outbound peers to use up our outbound
    


    ryanofsky commented at 6:35 pm on February 12, 2019:

    In commit “Fix handling of invalid headers” (9dd6fc18658b36b63b9f264676ac484879597b83)

    It would be nice to keep more of this comment, like the high-level goal of freeing outbound slots, and other parts if they are still accurate. It might also be good to keep a TODO about how we could “improve ban-behavior” here as mentioned in #15141 (review)


    sdaftuar commented at 5:24 pm on March 2, 2019:
    I think this PR is basically what that TODO is referring to!

    sdaftuar commented at 7:37 pm on March 7, 2019:

    Also I think most of the information in this comment is needed because the logic is much more complex before this PR than after. After this PR, it’s very easy to see how we handle different kinds of invalid blocks/headers, and so less verbose explanations are needed about the reasoning. I shrunk this comment down to:

    0            // Disconnect outbound (but not inbound) peers if on an invalid chain.
    

    which I think captures all that we’re trying to do.

  70. in src/net_processing.cpp:1028 in 346699322c outdated
    1013+    case ValidationInvalidReason::BLOCK_INVALID_HEADER:
    1014+    case ValidationInvalidReason::BLOCK_CHECKPOINT:
    1015+    case ValidationInvalidReason::BLOCK_INVALID_PREV:
    1016+        {
    1017+            LOCK(cs_main);
    1018+            Misbehaving(nodeid, 100, message);
    


    ryanofsky commented at 6:38 pm on February 12, 2019:

    In commit “[refactor] Use Reasons directly instead of DoS codes” (346699322ca820c5d95c255386df3ce1fb1f3d11)

    Is calling Misbehaving here when via_compact_block is true a change in behavior? Same question below for the BLOCK_MISSING_PREV case.


    sdaftuar commented at 8:28 pm on February 21, 2019:

    Regarding BLOCK_INVALID_PREV, see above:

    https://github.com/bitcoin/bitcoin/pull/15141/files/3048533275227e67ce22931c6360513bddbd1767..346699322ca820c5d95c255386df3ce1fb1f3d11#r248002976

    For the MISSING_PREV case, this is somewhat confusing but I don’t think it’s possible for via_compact_block to be true in that case, because we will send the peer a getheaders message if we receive a compact block announcement that doesn’t connect, and not process the compact block in that case.


    ryanofsky commented at 6:55 pm on March 5, 2019:

    re: #15141 (review)

    Thanks, looks like these cases are really not possible given assert(IsTransactionReason) in the new code.

  71. in src/net_processing.cpp:970 in 346699322c outdated
    967+        // knowable to a high-bandwidth compact block peer, prior to relay.
    968+        // Headers that are invalid for reasons that should be known prior to
    969+        // block validation -- such as bad proof of work, too-early-time, or
    970+        // building off an invalid or missing block -- are punished regardless
    971+        // (see below).
    972+        return !via_compact_block;
    


    ryanofsky commented at 6:43 pm on February 12, 2019:

    In commit “[refactor] Use Reasons directly instead of DoS codes” (346699322ca820c5d95c255386df3ce1fb1f3d11)

    Is returning true here a change in behavior? (via_compact_block is currently always false where this function is called)


    sdaftuar commented at 7:01 pm on February 21, 2019:
    So this is only used for transaction relay, but this function is written to be more generic… Perhaps I should limit the scope of this function to make it clear that transaction invalidity reasons are the only thing we would ask about?

    sdaftuar commented at 5:26 pm on March 2, 2019:
    Cleaned up in the latest version.
  72. ryanofsky approved
  73. ryanofsky commented at 7:06 pm on February 12, 2019: member
    utACK 7682566acd7b04aee9425772823e77f230268ad8. There is a lot here, so it’s hard for me to feel sure I didn’t miss something just from the volume of changes, but I think everything I’ve seen does look good. I left a few comments and some questions, but nothing critical.
  74. sdaftuar commented at 5:31 pm on March 2, 2019: member
    I believe I addressed all of @sipa’s comments. I think I’ve addressed all the substantive comments from @ryanofsky, with the exception of moving around code between commits. If reviewers would like to see the code moved to more accurately reflect each commit message, I can try, but I’d prefer to do something like that just before merge and have reviewers do a last verify that the overall diff is the same after the git history gets rewritten.
  75. in src/consensus/tx_verify.cpp:224 in beb6a373bd outdated
    220@@ -221,28 +221,27 @@ bool Consensus::CheckTxInputs(const CTransaction& tx, CValidationState& state, c
    221 
    222         // If prev is coinbase, check that it's matured
    223         if (coin.IsCoinBase() && nSpendHeight - coin.nHeight < COINBASE_MATURITY) {
    224-            return state.Invalid(false,
    225-                REJECT_INVALID, "bad-txns-premature-spend-of-coinbase",
    226+            return state.Invalid(ValidationInvalidReason::TX_MISSING_INPUTS, false, REJECT_INVALID, "bad-txns-premature-spend-of-coinbase",
    


    kallewoof commented at 5:21 am on March 5, 2019:
    Maybe this is covered already, but why is premature coinbase using TX_MISSING_INPUTS as the reason? It’s not really missing.

    ajtowns commented at 8:20 am on March 5, 2019:
    In either case, the transaction may be valid sometime later, but isn’t valid now and isn’t valid in a block, so they’re not very distinct.

    kallewoof commented at 8:29 am on March 5, 2019:
    TX_INPUTS_UNAVAILABLE?

    sdaftuar commented at 7:51 pm on March 7, 2019:
    I created a new TX_PREMATURE_COINBASE for this in the latest commit.
  76. in src/consensus/validation.h:32 in beb6a373bd outdated
    27+  * These are much more granular than the rejection codes, which may be more
    28+  * useful for some other use-cases.
    29+  */
    30+enum class ValidationInvalidReason {
    31+    // txn and blocks:
    32+    NONE,                    //!< not actually invalid
    


    kallewoof commented at 5:25 am on March 5, 2019:
    I understand if you don’t wanna bother this late in the process, but it really seems like ValidationResult or ValidationOutcome would have been easier on the eyes, where this would be SUCCESSFUL or something.

    sdaftuar commented at 2:52 pm on March 7, 2019:
    I’d prefer to rename this in a future PR, since that would be a simple change and I don’t want to hold up the structural improvements here.
  77. in src/net_processing.cpp:361 in beb6a373bd outdated
    355+
    356+    //! Whether this peer is a manual connection
    357+    bool m_is_manual_connection;
    358+
    359+    CNodeState(CAddress addrIn, std::string addrNameIn, bool is_inbound, bool is_manual) :
    360+        address(addrIn), name(std::move(addrNameIn)), m_is_inbound(is_inbound),
    


    kallewoof commented at 5:29 am on March 5, 2019:
    Any particular reason why this isn’t just CNodeState(.., const std::string& addrNameIn, ...) : ..., name(addrNameIn)?

    ryanofsky commented at 7:04 pm on March 5, 2019:

    re: #15141 (review)

    Any particular reason why this isn’t just CNodeState(.., const std::string& addrNameIn, …) : …, name(addrNameIn)?

    This would be worse because you can’t move from a const reference and there would be an unnecessary copy.

    If a function is inserting an argument into a data structure, it’s good practice for it to take the argument by value or by rvalue reference instead of by const reference, to avoid pointless copying.


    kallewoof commented at 3:02 am on March 6, 2019:
    I am pretty sure there will be no copying if the input is discardable, except for -O0, but I could be mistaken.

    ryanofsky commented at 4:32 am on March 6, 2019:

    Initializing the member either has to copy or move, and you can’t move from a const object, so the copy constructor will be called. If you are referring to -O0 it sounds like you are thinking about copy elision, but that doesn’t apply here for member initialization.

    Simple advice is to use const& for an argument that you want to copy from, && for an argument that you want to move from, and pass by value when you want to give the caller a choice of whether to copy or move.


    kallewoof commented at 4:44 am on March 6, 2019:
    I see. But we are doing const type& var all over the place elsewhere. This is in fact why I wondered about the “derivation from the standard” here. (A simple search for const std::string& gives 799 results.)

    sipa commented at 6:15 am on March 6, 2019:
    Passing a string by reference doesn’t need a copy. Using it to initialize a member does.

    kallewoof commented at 7:13 am on March 6, 2019:
    Interesting. Sorry for clogging up the PR.
  78. in src/net_processing.cpp:2504 in beb6a373bd outdated
    2527+                    LogPrintf("Not relaying invalid transaction %s from whitelisted peer=%d (%s)\n", tx.GetHash().ToString(), pfrom->GetId(), FormatStateMessage(state));
    2528+                } else {
    2529                     LogPrintf("Force relaying tx %s from whitelisted peer=%d\n", tx.GetHash().ToString(), pfrom->GetId());
    2530                     RelayTransaction(tx, connman);
    2531-                } else {
    2532-                    LogPrintf("Not relaying invalid transaction %s from whitelisted peer=%d (%s)\n", tx.GetHash().ToString(), pfrom->GetId(), FormatStateMessage(state));
    


    kallewoof commented at 5:39 am on March 5, 2019:

    You could shrink diff a few lines if you felt like it:

    0if (!state.IsInvalid() || !TxRelayMayResultInDisconnect(state)) {
    1    LogPrintf("Force relaying tx %s from whitelisted peer=%d\n", tx.GetHash().ToString(), pfrom->GetId());
    2    RelayTransaction(tx, connman);
    3} else {
    4    LogPrintf("Not relaying invalid transaction %s from whitelisted peer=%d (%s)\n", tx.GetHash().ToString(), pfrom->GetId(), FormatStateMessage(state));
    5}
    

    ajtowns commented at 6:26 am on March 5, 2019:
    Yeah, swapping the order here messes up the following fixup! commits too.
  79. in src/net_processing.cpp:2589 in beb6a373bd outdated
    2558@@ -2520,9 +2559,7 @@ bool static ProcessMessage(CNode* pfrom, const std::string& strCommand, CDataStr
    2559                 connman->PushMessage(pfrom, msgMaker.Make(NetMsgType::REJECT, strCommand, (unsigned char)state.GetRejectCode(),
    2560                                    state.GetRejectReason().substr(0, MAX_REJECT_MESSAGE_LENGTH), inv.hash));
    2561             }
    2562-            if (nDoS > 0) {
    2563-                Misbehaving(pfrom->GetId(), nDoS);
    2564-            }
    2565+            MaybePunishNode(pfrom->GetId(), state, /*via_compact_block*/ false);
    


    kallewoof commented at 5:42 am on March 5, 2019:
    Nit: you use /*paramname=*/ value above, but /*paramname*/ value here.

    sdaftuar commented at 7:26 pm on March 7, 2019:
    Not going to bother with these minor style nits, we can clean these up later if anyone cares to.
  80. in src/net_processing.cpp:2626 in beb6a373bd outdated
    2596-                    Misbehaving(pfrom->GetId(), nDoS, strprintf("Peer %d sent us invalid header via cmpctblock\n", pfrom->GetId()));
    2597-                } else {
    2598-                    LogPrint(BCLog::NET, "Peer %d sent us invalid header via cmpctblock\n", pfrom->GetId());
    2599-                }
    2600+            if (state.IsInvalid()) {
    2601+                MaybePunishNode(pfrom->GetId(), state, /*via_compact_block*/ true, "invalid header via cmpctblock");
    


    kallewoof commented at 5:43 am on March 5, 2019:
    And here (p= vs p).
  81. in src/validation.cpp:2018 in beb6a373bd outdated
    2034+                if (state.GetReason() == ValidationInvalidReason::TX_NOT_STANDARD) {
    2035+                    // CheckInputs may return NOT_STANDARD for extra flags we passed,
    2036+                    // but we can't return that, as it's not defined for a block, so
    2037+                    // we reset the reason flag to CONSENSUS here.
    2038+                    // (note that this may not be the case until we add additional
    2039+                    // soft-fork flags to our script flags, in which case we  need to
    


    kallewoof commented at 5:49 am on March 5, 2019:

    Still extra space as @ryanofsky pointed out.

    0                    // soft-fork flags to our script flags, in which case we need to
    

    TheBlueMatt commented at 7:21 pm on April 9, 2019:
    Still this. Also, the grammar here is confusing - what may not be the case until we add new flags? I think you meant “note that this may not be the case after we add additional flags”
  82. kallewoof commented at 5:52 am on March 5, 2019: member
    utACK with nits
  83. in src/net_processing.cpp:2523 in beb6a373bd outdated
    2520@@ -2480,12 +2521,11 @@ bool static ProcessMessage(CNode* pfrom, const std::string& strCommand, CDataStr
    2521                 // Never relay transactions that we would assign a non-zero DoS
    2522                 // score for, as we expect peers to do the same with us in that
    2523                 // case.
    


    ajtowns commented at 6:25 am on March 5, 2019:
    Should update the comment here too probably – s/assign a non-zero DoS score for/disconnect a peer/.

    sdaftuar commented at 7:25 pm on March 7, 2019:
    Done in latest commit.
  84. ajtowns commented at 8:17 am on March 5, 2019: member

    The TxRelayMayResultInDisconnect change is good; probably makes sense to call it that from the start. I’ve done that and reordered all the fixups to be as near as possible to their corresponding commit in https://github.com/ajtowns/bitcoin/tree/201901-dosreasons . I think it would make sense to pull out the first set of commits into an initial PR:

    09c2c8b37b Clean up banning levels 07f9ac6dad set nDoS rather than bumping it c803f52624 [refactor] drop IsInvalid(nDoSOut) 8ab3ffe458 Update comment to reference MaybePunishNode 04f76c87bd fixup! fixup! [refactor] Use maybepunish etc d57c7c124e fixup! [refactor] Use maybepunish etc 6cee43cd4f [refactor] Use maybepunish etc 73c4826d92 [refactor] stateDummy -> orphan_state d00c5904e1 drop obsolete comment e878e639df Remove redundant state.Invalid() call after CheckTransaction()

  85. ryanofsky approved
  86. ryanofsky commented at 7:51 pm on March 5, 2019: member

    utACK beb6a373bdfa91166c8872909ccce31c5b0d72a3, this resolves my earlier comment at #15141 (review) and seems to resolve sipa’s comment at #15141 (comment).

    The 201901-dosreasons branch which reorders commits also looks good, but I didn’t verify it as closely. Would be nice (I think) to squash fixup commits too.

  87. in src/consensus/validation.h:50 in 858bae6104 outdated
    44@@ -45,9 +45,9 @@ class CValidationState {
    45         strRejectReason = strRejectReasonIn;
    46         corruptionPossible = corruptionIn;
    47         strDebugMessage = strDebugMessageIn;
    48+        nDoS = level;
    49         if (mode == MODE_ERROR)
    50             return ret;
    51-        nDoS += level;
    


    Sjors commented at 2:27 pm on March 6, 2019:
    What was the original idea behind bumping this? Was it to continue validating and perhaps find additional errors before giving up, to see if the originating peer needs stronger punishment?

    sdaftuar commented at 7:32 pm on March 7, 2019:
    I don’t know the history here, but that seems like a reasonable guess.

    TheBlueMatt commented at 5:56 pm on April 9, 2019:
    Can you just drop this commit? Its really nontrivial to review and, by the end, is unused anyway.

    jnewbery commented at 2:25 pm on April 11, 2019:

    re: #15141 (review)

    This needs to be done in this early commit so that the second half of this PR is refactor-only. Once we move to invalid-reason based punishment, calling Invalid() on a CValidationState object sets the reason and doesn’t ‘increment’ it.

    I think this commit should stay, possible squashed with the following commit (Clean up banning levels) but with an expanded git commit log explaining why the change is being made and highlighting what reviewers should check to satisfy themselves that this isn’t a behaviour change (that DoS() is never called more than once on the same object).

  88. Sjors commented at 2:34 pm on March 6, 2019: member

    Concept ACK. Some questions based on reviewing beb6a373bd

    Where possible the changes in ban levels / policy deserve their own PR.

  89. sdaftuar force-pushed on Mar 7, 2019
  90. sdaftuar force-pushed on Mar 7, 2019
  91. sdaftuar commented at 8:15 pm on March 7, 2019: member
    Squashed the history down. Unsquashed version is 15141.4.
  92. sdaftuar force-pushed on Mar 8, 2019
  93. ryanofsky approved
  94. ryanofsky commented at 7:52 pm on March 8, 2019: member
    utACK f8240399ab2d03bc2095d85de2522c71507f228b. Only changes since last review: squashing and moving things between commits (thanks!), updated comments, introduction of TX_PREMATURE_SPEND reason.
  95. Sjors commented at 8:31 am on March 9, 2019: member
    @ryanofsky there’s still a few ==== commits that need to go.
  96. sdaftuar force-pushed on Mar 9, 2019
  97. sdaftuar commented at 4:32 pm on March 9, 2019: member
    @sjors Removed the empty commits from the PR. Old version is here: https://github.com/sdaftuar/bitcoin/commits/15141.6
  98. ryanofsky approved
  99. ryanofsky commented at 4:46 pm on March 11, 2019: member
    utACK daf23bf0b9791cb6a1515e3764d1f645e1181859, only change is removing the empty commits
  100. ryanofsky commented at 4:56 pm on March 28, 2019: member
    I wonder who else is still planning to review this PR. @sipa, @jnewbery, @kallewoof, @sjors, @ajtowns all reviewed previous versions. @TheBlueMatt wrote the original version and discussed it recently offline. @naumenkogs wrote that he intends to review. Is there anyone else? It seems like if some subset of the people who already reviewed this rereviewed it, it could be ready to merge.
  101. jamesob commented at 3:01 am on March 30, 2019: member
    I’m going to take a read through at some point (probably tomorrow), but that definitely shouldn’t hang up a merge if the reviewers mentioned above have signed off.
  102. ajtowns commented at 7:54 am on April 1, 2019: member

    utACK daf23bf0b9791cb6a1515e3764d1f645e1181859

    I think it would have made sense to split this PR after “Clean up banning levels”, so utACK 96cedc8d0c0e3ad279bc2223a7fc3185b17ebde5 as well, fwiw. I guess it’s probably better to just get reviews for the whole thing at this point though.

  103. DrahtBot added the label Needs rebase on Apr 1, 2019
  104. in src/consensus/validation.h:41 in ef099b059a outdated
    36+     * Currently unused as there are no such consensus rule changes, and any download
    37+     * sources realistically need to support SegWit in order to provide useful data,
    38+     * so differentiating between always-invalid and invalid-by-pre-SegWit-soft-fork
    39+     * is uninteresting.
    40+     */
    41+    RECENT_CONSENSUS_CHANGE,
    


    jamesob commented at 8:34 pm on April 1, 2019:
    This seems fluffy - I suspect we’d want to call out specific changes if we were to actually do something like this. I guess there’s value in having it here as a conceptual placeholder, but I’d be surprised if this symbol ever actually got used.

    sdaftuar commented at 9:28 am on April 3, 2019:
    Happy to remove if others agree since that was my preference as well, but I left this in because Matt preferred it when we originally discussed this on his PR.

    jamesob commented at 3:25 pm on April 3, 2019:
    It’s not hurting anything; I’d rather have this merged and remove it later.
  105. in src/consensus/validation.h:128 in ef099b059a outdated
    119@@ -72,12 +120,40 @@ class CValidationState {
    120         return mode == MODE_ERROR;
    121     }
    122     bool CorruptionPossible() const {
    123+        assert(corruptionPossible == (reason == ValidationInvalidReason::BLOCK_MUTATED || reason == ValidationInvalidReason::TX_WITNESS_MUTATED));
    124         return corruptionPossible;
    125     }
    126     void SetCorruptionPossible() {
    127         corruptionPossible = true;
    128+        assert(corruptionPossible == (reason == ValidationInvalidReason::BLOCK_MUTATED || reason == ValidationInvalidReason::TX_WITNESS_MUTATED));
    


    jamesob commented at 8:37 pm on April 1, 2019:
    Nice.
  106. in src/validation.cpp:3148 in ef099b059a outdated
    3144@@ -3123,7 +3145,7 @@ bool CheckBlock(const CBlock& block, CValidationState& state, const Consensus::P
    3145         nSigOps += GetLegacySigOpCount(*tx);
    3146     }
    3147     if (nSigOps * WITNESS_SCALE_FACTOR > MAX_BLOCK_SIGOPS_COST)
    3148-        return state.DoS(100, false, REJECT_INVALID, "bad-blk-sigops", false, "out-of-bounds SigOpCount");
    3149+        return state.DoS(100, ValidationInvalidReason::CONSENSUS, false, REJECT_INVALID, "bad-blk-sigops", false, "out-of-bounds SigOpCount");
    


    jamesob commented at 8:48 pm on April 1, 2019:
    I know this is behavior we’re inheriting, but I find it kind of weird that too many sigops in a transaction is considered a standardness issue (score 0), but too many sigops in a block is consensus violation (ban). I guess conceptually it’s not as bad to try to broadcast a transaction that has too many sigops vs. a PoW’d block, but I’m curious if there’s a specific reason for the inconsistency beyond that. Again, sort of an academic question outside the scope of this PR.
  107. jamesob approved
  108. jamesob commented at 3:19 pm on April 2, 2019: member

    Testedish ACK https://github.com/bitcoin/bitcoin/pull/15141/commits/daf23bf0b9791cb6a1515e3764d1f645e1181859

    This change makes a heck of a lot of sense in terms of separating responsibilities from validation, which now reports the nature of validation failures instead of dictating their implications, and net_processing, which now is the single source for how to treat a peer based upon the nature of the failure.

    The more we can restrict the conceptual scope of validation.cpp the better, and this change insulates it from a tricky and critical set of tasks, i.e. peer penalization.

    For kicks (mostly as an easy way to run IBD), I had bitcoinperf compare master and this branch on an IBD up to height 120,000.

    name iterations 2019-01-rewrite-validation-interface master
    build.make.7.clang 1 1.96 1.000
    build.make.7.clang.mem-usage 1 1.02 1.000
    ibd.local.100000.dbcache=2048 3 1.00 1.005
    ibd.local.100000.dbcache=2048.mem-usage 3 1.00 1.000
    ibd.local.120000.dbcache=2048 3 1.00 1.001
    ibd.local.120000.dbcache=2048.mem-usage 3 1.00 1.000
    ibd.local.5000.dbcache=2048 3 1.00 1.021
    ibd.local.5000.dbcache=2048.mem-usage 3 1.00 1.000
    ibd.local.80000.dbcache=2048 3 1.00 1.006
    ibd.local.80000.dbcache=2048.mem-usage 3 1.00 1.000
  109. sdaftuar force-pushed on Apr 3, 2019
  110. sdaftuar commented at 9:29 am on April 3, 2019: member
  111. DrahtBot removed the label Needs rebase on Apr 3, 2019
  112. jamesob commented at 3:16 pm on April 3, 2019: member
    reACK daf674e55a4efbf6031af8389de2537ed32b4bcc based on a review of the interdiff.
  113. ryanofsky approved
  114. ryanofsky commented at 9:19 pm on April 3, 2019: member
    utACK daf674e55a4efbf6031af8389de2537ed32b4bcc. No changes at all since last review other than rebase. (I rebased it myself and confirmed commits were identical.)
  115. in src/validation.cpp:728 in daf8a2e3d9 outdated
    728@@ -729,11 +729,6 @@ static bool AcceptToMemoryPoolWorker(const CChainParams& chainparams, CTxMemPool
    729                               fSpendsCoinbase, nSigOpsCost, lp);
    730         unsigned int nSize = entry.GetTxSize();
    731 
    732-        // Check that the transaction doesn't have an excessive number of
    733-        // sigops, making it impossible to mine. Since the coinbase transaction
    734-        // itself can contain sigops MAX_STANDARD_TX_SIGOPS is less than
    735-        // MAX_BLOCK_SIGOPS; we still consider this an invalid rather than
    736-        // merely non-standard transaction.
    


    TheBlueMatt commented at 9:24 pm on April 8, 2019:
    Can you amend the commit message to indicate why the “obsolete comment” is obsolete?

    sdaftuar commented at 6:38 pm on April 11, 2019:
    Done in latest version.
  116. in src/net_processing.cpp:994 in daf17f580f outdated
    959@@ -960,6 +960,23 @@ void Misbehaving(NodeId pnode, int howmuch, const std::string& message) EXCLUSIV
    960         LogPrint(BCLog::NET, "%s: %s peer=%d (%d -> %d)%s\n", __func__, state->name, pnode, state->nMisbehavior-howmuch, state->nMisbehavior, message_prefixed);
    961 }
    962 
    963+static bool MayResultInDisconnect(const CValidationState& state, bool via_compact_block) {
    964+    return (state.GetDoS() > 0);
    965+}
    966+
    967+static bool MaybePunishNode(NodeId nodeid, const CValidationState& state, bool via_compact_block, const std::string& message = "") {
    


    TheBlueMatt commented at 9:34 pm on April 8, 2019:
    “Use maybepunish” needs a real commitmessage. Wtf is a “maybepunish” and why are we using it?

    sdaftuar commented at 6:38 pm on April 11, 2019:
    Done in latest version.
  117. in src/net_processing.cpp:2625 in daf17f580f outdated
    2589-                    LOCK(cs_main);
    2590-                    Misbehaving(pfrom->GetId(), nDoS, strprintf("Peer %d sent us invalid header via cmpctblock\n", pfrom->GetId()));
    2591-                } else {
    2592-                    LogPrint(BCLog::NET, "Peer %d sent us invalid header via cmpctblock\n", pfrom->GetId());
    2593-                }
    2594+            if (state.IsInvalid()) {
    


    TheBlueMatt commented at 5:54 pm on April 9, 2019:
    In “use maybepunish” - this is a major behavior change. After this commit sending an invalid header via compact block no longer gets a ban when it previously did.

    sdaftuar commented at 6:38 pm on April 11, 2019:
    Fixed.
  118. in src/consensus/tx_verify.cpp:224 in daf35b19b3 outdated
    220@@ -221,8 +221,8 @@ bool Consensus::CheckTxInputs(const CTransaction& tx, CValidationState& state, c
    221 
    222         // If prev is coinbase, check that it's matured
    223         if (coin.IsCoinBase() && nSpendHeight - coin.nHeight < COINBASE_MATURITY) {
    224-            return state.Invalid(false,
    225-                REJECT_INVALID, "bad-txns-premature-spend-of-coinbase",
    226+            return state.DoS(100, false,
    


    TheBlueMatt commented at 6:14 pm on April 9, 2019:
    This now results in banning peers when they relay us mempool transactions when we aren’t yet fully synced. I dont think this is acceptable. Also, you add this here and then have a commit dedicated to removing it later, all to make intermediate assert()s pass, when you could just change the above DoS score…

    sdaftuar commented at 6:38 pm on April 11, 2019:
    Fixed the banning brokenness.
  119. in src/consensus/validation.h:96 in daf2ece95a outdated
    93         strRejectReason = strRejectReasonIn;
    94         corruptionPossible = corruptionIn;
    95         strDebugMessage = strDebugMessageIn;
    96         nDoS = level;
    97+        assert(nDoS == GetDoSForReason());
    98+        assert(corruptionPossible == (reason == ValidationInvalidReason::BLOCK_MUTATED || reason == ValidationInvalidReason::TX_WITNESS_MUTATED));
    


    TheBlueMatt commented at 6:53 pm on April 9, 2019:
    Both of these asserts are useless - we remove the value later in the same PR - please enforce this using the scripted-diff or just remove them - they only serve to confuse reviewers.
  120. in src/consensus/validation.h:131 in daf2ece95a outdated
    126     void SetCorruptionPossible() {
    127         corruptionPossible = true;
    128+        assert(corruptionPossible == (reason == ValidationInvalidReason::BLOCK_MUTATED || reason == ValidationInvalidReason::TX_WITNESS_MUTATED));
    129     }
    130     int GetDoS(void) const { return nDoS; }
    131+    int GetDoSForReason() const {
    


    TheBlueMatt commented at 6:56 pm on April 9, 2019:
    This is nonsense, why is it in consensus code, and why is it removed later? It just makes review harder.
  121. in src/consensus/validation.h:153 in daf2ece95a outdated
    148+        case ValidationInvalidReason::TX_WITNESS_MUTATED:
    149+        case ValidationInvalidReason::TX_CONFLICT:
    150+        case ValidationInvalidReason::TX_MEMPOOL_POLICY:
    151+            return 0;
    152+        default:
    153+            assert(false);
    


    TheBlueMatt commented at 6:59 pm on April 9, 2019:
    what? The compiler will warn you if you are missing an entry, instead of adding an assert which only warns at runtime?
  122. in src/net_processing.cpp:964 in daf2ece95a outdated
    960@@ -961,10 +961,13 @@ void Misbehaving(NodeId pnode, int howmuch, const std::string& message) EXCLUSIV
    961 }
    962 
    963 static bool MayResultInDisconnect(const CValidationState& state, bool via_compact_block) {
    964+    assert(!via_compact_block);
    


    TheBlueMatt commented at 6:59 pm on April 9, 2019:
    Huh?! Just remove the parameter, then.
  123. in src/consensus/validation.h:48 in daf2ece95a outdated
    43+    CACHED_INVALID,          //!< this object was cached as being invalid, but we don't know why
    44+    BLOCK_INVALID_HEADER,    //!< invalid proof of work or time too old
    45+    BLOCK_MUTATED,           //!< the block's data didn't match the data committed to by the PoW
    46+    BLOCK_MISSING_PREV,      //!< We don't have the previous block the checked one is built on
    47+    BLOCK_INVALID_PREV,      //!< A block this one builds on is invalid
    48+    BLOCK_BAD_TIME,          //!< block timestamp was > 2 hours in the future (or our clock is bad)
    


    TheBlueMatt commented at 7:23 pm on April 9, 2019:
    This name is somewhat more generic than it needs to be - maybe call it BLOCK_TIME_FUTURE.
  124. in src/validation.cpp:3260 in daf2ece95a outdated
    3259     // check for version 2, 3 and 4 upgrades
    3260     if((block.nVersion < 2 && nHeight >= consensusParams.BIP34Height) ||
    3261        (block.nVersion < 3 && nHeight >= consensusParams.BIP66Height) ||
    3262        (block.nVersion < 4 && nHeight >= consensusParams.BIP65Height))
    3263-            return state.DoS(100, false, REJECT_OBSOLETE, strprintf("bad-version(0x%08x)", block.nVersion), false,
    3264+            return state.DoS(100, ValidationInvalidReason::CONSENSUS, false, REJECT_OBSOLETE, strprintf("bad-version(0x%08x)", block.nVersion), false,
    


    TheBlueMatt commented at 7:24 pm on April 9, 2019:
    Why is this CONSENSUS and not BLOCK_INVALID_HEADER?
  125. in src/consensus/validation.h:105 in dafd6200ae outdated
    73@@ -74,16 +74,16 @@ class CValidationState {
    74         MODE_INVALID, //!< network rule violation (DoS value may be set)
    75         MODE_ERROR,   //!< run-time error
    76     } mode;
    77-    ValidationInvalidReason reason;
    78+    ValidationInvalidReason m_reason;
    


    TheBlueMatt commented at 7:34 pm on April 9, 2019:
    Can you squash “nit: reason -> m_reason”? Please don’t add full commits that just address nits. Ignoring the nit is also perfectly valid.

    sdaftuar commented at 6:38 pm on April 11, 2019:
    Fixed.
  126. in src/consensus/validation.h:83 in dafbf9e908 outdated
    76+           r == ValidationInvalidReason::TX_WITNESS_MUTATED ||
    77+           r == ValidationInvalidReason::TX_CONFLICT ||
    78+           r == ValidationInvalidReason::TX_MEMPOOL_POLICY;
    79+}
    80+
    81+inline bool IsBlockReason(ValidationInvalidReason r)
    


    TheBlueMatt commented at 7:35 pm on April 9, 2019:
    Oof this really sucks. To avoid delaying this PR I think this is fine, but we really need to just switch to two different ValidationState classes for Transactions and Blocks.

    ajtowns commented at 6:19 am on April 16, 2019:

    FWIW, I had a go at this. See https://github.com/ajtowns/bitcoin/commits/201904-dos-rework-sep

    I think it works okay, but splitting the type sure hits a lot of lines, see: https://github.com/ajtowns/bitcoin/commit/74bf5c12672d7eee877230d31b308c696fe83904 which is the commit that just splits the type. It’s followed by the rest of this PR rebased on top of it, but with ValidationInvalidReason renamed to [Block|Tx]ValidationResult (since I’m touching every reference to that class anyway), and TX_PREMATURE_SPEND brought in initially rather than as a later commit.


    jnewbery commented at 2:37 pm on April 16, 2019:
    Nice work @ajtowns, but I think @sdaftuar might have a meltdown if we try to rebase this PR on any additional changes at this point. It seems to me that splitting CValidationState into separate classes for Tx and Block can be done in a follow-up PR to this. I’ll commit to reviewing that follow-up PR if you open it after this is merged.

    JeremyRubin commented at 7:27 pm on April 25, 2019:

    Noting for later reference:

    Definitely not worth changing in this PR now given # of acks, but this can probably be cleaned up in a later PR to just give each category of error a unique prefix (e.g, make all TX_* have MSByte set to 3 and all BLOCK_* have MSB set to 2 and general set to 1) or something.

    0enum class ValidationInvalidReason : uint64_t {
    1  IS_GENERAL REASON =1 << 56,
    2  // ....
    3  IS_BLOCK_REASON = 2 << 56
    4 // ....
    5  IS_TX_REASON = 3 << 56
    6}
    

    Then these checks clean up to:

    0IS_TX_REASON & r || IS_GENERAL_REASON & r
    

    The downside is depending on the enum grouping order, but then this makes it easier to later modify the code to split these enums into different classes (while preserving offsets), and then make the general wrapper functions take the correct type of enum.

  127. TheBlueMatt commented at 7:35 pm on April 9, 2019: member
    I think its mostly fine at the end, but a bunch of intermediary commits are now broken.
  128. DrahtBot added the label Needs rebase on Apr 10, 2019
  129. jnewbery commented at 8:54 pm on April 10, 2019: member

    I’ve fully reviewed everything up to Clean up banning levels (daf35b19b3841fe93dfd717b7c233ff293b61503) and quickly looked at the rest of the commits. I agree with @TheBlueMatt that some of the intermediate commits introduce behaviour changes and should be changed or squashed.

    I have a branch here https://github.com/jnewbery/bitcoin/tree/2019_04_clean_up_banning that rebases everything up to Clean up banning levels on master, fixes a bunch of nits, expands comments and maintains existing p2p behaviour (except in the last commit which updates DoS scores). I think to make progress on this, we could open that as a separate PR as suggested by AJ here: #15141 (comment). @sdaftuar I’m happy to open the PR if that helps you.

  130. sdaftuar commented at 6:37 pm on April 11, 2019: member

    Old version of this pr is here: 15141.9

    I took that version and incorporated many of the comments and bugfixes Matt suggested, along with a few improvements from John, and a bunch of fixes to commit messages/authorship attribution/etc here: 15141.10

    Here’s a diff of 15141.9 and 15141.10, which may be of some value to prior reviewers:

     0diff --git a/src/net_processing.cpp b/src/net_processing.cpp
     1index d8cb564378e..b074cead4fd 100644
     2--- a/src/net_processing.cpp
     3+++ b/src/net_processing.cpp
     4@@ -980,8 +980,18 @@ static bool TxRelayMayResultInDisconnect(const CValidationState& state)
     5     return state.GetReason() == ValidationInvalidReason::CONSENSUS;
     6 }
     7 
     8-//! Returns true if the peer was punished (probably disconnected)
     9-//! Changes here may need to be reflected in TxRelayMayResultInDisconnect().
    10+/**
    11+ * Potentially ban a node based on the contents of a CValidationState object
    12+ *
    13+ * [@param](/bitcoin-bitcoin/contributor/param/)[in] via_compact_block: this bool is passed in because net_processing should
    14+ * punish peers differently depending on whether the data was provided in a compact
    15+ * block message or not. If the compact block had a valid header, but contained invalid
    16+ * txs, the peer should not be punished. See BIP 152.
    17+ *
    18+ * [@return](/bitcoin-bitcoin/contributor/return/) Returns true if the peer was punished (probably disconnected)
    19+ *
    20+ * Changes here may need to be reflected in TxRelayMayResultInDisconnect().
    21+ */
    22 static bool MaybePunishNode(NodeId nodeid, const CValidationState& state, bool via_compact_block, const std::string& message = "") {
    23     switch (state.GetReason()) {
    24     case ValidationInvalidReason::NONE:
    

    Finally, I rebased all that back onto master and have pushed that here. (I think the rebase is easiest to verify by doing a merge with master, rather than re-doing the rebase – it was pretty tedious since many commits touched the same conflicted files.)

  131. sdaftuar force-pushed on Apr 11, 2019
  132. DrahtBot removed the label Needs rebase on Apr 11, 2019
  133. in src/net_processing.cpp:2601 in 83fc55eb41 outdated
    2604+                // here.
    2605+                // TODO: when we switch from DoS scores to reasons that
    2606+                // tx/blocks are invalid, this call should set
    2607+                // via_compact_block to true, since MaybePunishNode will have
    2608+                // sufficient information to act correctly.
    2609+                MaybePunishNode(pfrom->GetId(), state, /*via_compact_block*/ false, "invalid header via cmpctblock");
    


    sdaftuar commented at 11:45 pm on April 13, 2019:
    @TheBlueMatt On further thought, I think this commit still introduces a bug in compact block headers handling, which is fixed later in this PR – a duplicate-invalid header delivered by a compact block peer would cause that peer to get disconnected. Will fix.
  134. sdaftuar force-pushed on Apr 14, 2019
  135. sdaftuar force-pushed on Apr 14, 2019
  136. sdaftuar commented at 12:15 pm on April 14, 2019: member
    Just pushed up a new version that fixes a bug in an intermediate commit. (Final code was unchanged as the bug was already fixed in a later commit.) Old version was 15141.11.
  137. sdaftuar force-pushed on Apr 14, 2019
  138. sdaftuar force-pushed on Apr 15, 2019
  139. sdaftuar commented at 3:07 pm on April 15, 2019: member

    Rewrote the history one more time to incorporate the rest of Matt’s feedback (other than squashing some of the intermediate commits). Old version is here: 15141.12

    Also, @jnewbery pointed out somewhere (I seem to have lost the link to the comment) that reviewers should take note that DoS scores would accumulate, while the switch to Reason enums eliminates that potential accumulation, so reviewers should carefully check that the behavior changes that result are only the ones claimed in the PR. I’ll repeat that here to make sure everyone is aware of this (but I don’t think it’s necessary to repeat this in a commit message or in a comment in an intermediate commit).

    Please re-review. While the commit history has been rewritten to fix several bugs, the overall diff of what has been previously ACK’ed by some reviewers to the current version is still pretty small (modulo the rebase to master that happened in between).

  140. ryanofsky approved
  141. ryanofsky commented at 5:22 pm on April 15, 2019: member
    utACK 77c1635d5162f3846300077ec6fd2b3a12de0ac8. This was rebased since last review and has some new comments and renames. Only non-obvious code change in overall diff was updating CONSENSUS -> BLOCK_INVALID_HEADER for #15141 (review). Internal commits have other changes. “Refactor misbehavior ban decisions” commit has a new workaround to keep banning invalid headers in CMPCTBLOCK messages until BLOCK_INVALID_HEADER reason is handled later in “Refactor misbehavior ban decisions to MaybePunishNode” commit. “Clean up banning levels” commit now does not add a ban on premature coinbase spend, but this happens in later ‘Add useful-for-dos “reason”’ commit. The “set nDoS rather than bumping it” commit is dropped. The “Refactor misbehavior ban decisions” commit has had some later cleanups squashed in.
  142. jnewbery commented at 10:03 pm on April 15, 2019: member

    I think it’s worth updating the git commit log in 088336aa3cab566a6418fb2e88824bdf668b8458 ([refactor] Add useful-for-dos “reason” field to CValidationState) to say that previously, blocks which had a transaction that failed a non-p2sh pre-segwit softfork would not result in a ban, but now will. I also think that you should remove the [refactor] tag for a commit which isn’t a pure refactor and changes p2p behaviour.

    Also, @jnewbery pointed out somewhere (I seem to have lost the link to the comment) that reviewers should take note that DoS scores would accumulate, while the switch to Reason enums eliminates that potential accumulation, so reviewers should carefully check that the behavior changes that result are only the ones claimed in the PR. I’ll repeat that here to make sure everyone is aware of this (but I don’t think it’s necessary to repeat this in a commit message or in a comment in an intermediate commit).

    git commit logs are basically free (there’s almost no downside to being more verbose in them) and I often look at old commit logs for motivation/justification for changes. If you do change your mind and decide to include a justification in the commit, then the commit log from my branch was:

    0This PR also introduces a subtle change in the way nDoS is set on
    1CValidationState objects. Instead of incrementing the nDoS score when
    2DoS() is called, the nDoS is set to the new value. This would
    3potentially be a behavior change if DoS() were called more than once on
    4the same CValidationState object. To verify that this is not a behavior
    5change, check that DoS() is never called more than once on the same
    6CValidationState object.
    

    If you wanted to make that change really easy to review, you could change the commit order so that all calls to state.DoS(0... are changed to state.Invalid(... before that commit. That means that the only call to state.DoS(... where the nDoS isn’t 100 is the ‘missing inputs’ failure. Reviewers would only need to verify that the CValidationState set there hasn’t already been set before, since calling state.DoS(100, ... has the same effect before and after that commit. (I’m not saying you should do that now, but it might aid other reviewers to see that that is equivalent to the final state of this PR).

  143. sdaftuar force-pushed on Apr 16, 2019
  144. sdaftuar commented at 5:08 pm on April 16, 2019: member

    I think it’s worth updating the git commit log in 088336a ([refactor] Add useful-for-dos “reason” field to CValidationState) to say that previously, blocks which had a transaction that failed a non-p2sh pre-segwit softfork would not result in a ban, but now will. I also think that you should remove the [refactor] tag for a commit which isn’t a pure refactor and changes p2p behaviour.

    This behavior change (where a node running with -par=1 would potentially not ban for a block with invalid scripts, to now always banning regardless of whether -par=1 or not) being present in that refactor commit was an oversight; I’ve moved it to its own commit.

    Prior version of the code is here: 15141.13. Once again there should be no diff in the final code.

    Regarding this:

    0CValidationState objects. Instead of incrementing the nDoS score when
    1DoS() is called, the nDoS is set to the new value.
    

    I actually dropped the commit (at Matt’s request) where nDoS would not be incremented, so that the assert()’s that @ajtowns had suggested (comparing the DoS score to the reason enum) should be an effective reminder to the reviewer that incrementing is not taking place.

  145. in src/consensus/validation.h:32 in fdd7683c55 outdated
    27+  * These are much more granular than the rejection codes, which may be more
    28+  * useful for some other use-cases.
    29+  */
    30+enum class ValidationInvalidReason {
    31+    // txn and blocks:
    32+    NONE,                    //!< not actually invalid
    


    jnewbery commented at 6:49 pm on April 16, 2019:
    nit: This should be “not invalid or not yet tested for validity”
  146. in src/consensus/validation.h:64 in fdd7683c55 outdated
    59+    TX_WITNESS_MUTATED,
    60+    /**
    61+     * Tx already in mempool or conflicts with a tx in the chain
    62+     * (if it conflicts with another tx in mempool, we use MEMPOOL_POLICY as it failed to reach the RBF threshold)
    63+     * TODO: Currently this is only used if the transaction already exists in the mempool or on chain,
    64+     * TODO: ATMP's fMissingInputs and a valid CValidationState being used to indicate missing inputs
    


    jnewbery commented at 6:51 pm on April 16, 2019:
    nit: I think this TODO should be next to the TX_MISSING_INPUTS line
  147. in src/validation.cpp:739 in fdd7683c55 outdated
    743+            return state.Invalid(ValidationInvalidReason::TX_MEMPOOL_POLICY, false, REJECT_INSUFFICIENTFEE, "min relay fee not met", strprintf("%d < %d", nModifiedFees, ::minRelayTxFee.GetFee(nSize)));
    744         }
    745 
    746         if (nAbsurdFee && nFees > nAbsurdFee)
    747-            return state.Invalid(false,
    748+            return state.Invalid(ValidationInvalidReason::TX_NOT_STANDARD, false,
    


    jnewbery commented at 7:00 pm on April 16, 2019:
    nit: this seems like the wrong reason (although nAbsurdFee is only used by the wallet so it doesn’t really matter. I also think that ATMP shouldn’t take an nAbsurdFee parameter - see #15810)
  148. in src/consensus/validation.h:43 in fdd7683c55 outdated
    38+     * so differentiating between always-invalid and invalid-by-pre-SegWit-soft-fork
    39+     * is uninteresting.
    40+     */
    41+    RECENT_CONSENSUS_CHANGE,
    42+    // Only blocks (or headers):
    43+    CACHED_INVALID,          //!< this object was cached as being invalid, but we don't know why
    


    jnewbery commented at 7:21 pm on April 16, 2019:
    nit: change “we don’t know why” to “we didn’t store the reason why”
  149. in src/net_processing.cpp:1644 in fdd7683c55 outdated
    1641@@ -1551,48 +1642,8 @@ bool static ProcessHeadersMessage(CNode *pfrom, CConnman *connman, const std::ve
    1642     CValidationState state;
    1643     CBlockHeader first_invalid_header;
    1644     if (!ProcessNewBlockHeaders(headers, state, chainparams, &pindexLast, &first_invalid_header)) {
    


    jnewbery commented at 7:33 pm on April 16, 2019:

    nit: drop the first_invalid_header argument. It defaults to nullptr and you’re no longer using the returned value.

    Follow-up: remove the first_invalid argument from ProcessNewBlockHeaders() since none of the callers now use it.

  150. jnewbery commented at 7:41 pm on April 16, 2019: member

    utACK fdd7683c558984a96dd556e2c93dde156b85a75f

    I’ve left a few nits inline, which I think should be addressed in a follow-up PR.

  151. ryanofsky approved
  152. ryanofsky commented at 8:50 pm on April 16, 2019: member
    utACK fdd7683c558984a96dd556e2c93dde156b85a75f. No changes in overall diff from last review. There is a new note added to commit “Clean up banning levels” message. There is also a new intermediate commit “Ban all peers for all block script failures” pulled out of the following commit “Add useful-for-dos “reason” field to CValidationState”.
  153. jnewbery commented at 10:07 pm on April 16, 2019: member
    @ajtowns @TheBlueMatt - you’ve both more-or-less ACKed a branch that reaches basically the same final state as this current branch. Are you able to reACK?
  154. JeremyRubin commented at 7:37 pm on April 25, 2019: contributor

    Concept ACK that this seems ok – haven’t had a chance to re-go through commit by commit and make sure the behavioral changes are solid, but they seem sensible.

    Noting that merging this would make it possible to reboot the work in #11523 if there’s positive sentiment for it.

  155. TheBlueMatt commented at 7:55 pm on April 26, 2019: member
    utACK fdd7683c558984a96dd556e2c93dde156b85a75f
  156. in src/validation.cpp:1974 in fdd7683c55 outdated
    1974+                    // defined for a block, so we reset the reason flag to
    1975+                    // CONSENSUS here.
    1976                     state.Invalid(ValidationInvalidReason::CONSENSUS, false,
    1977                             state.GetRejectCode(), state.GetRejectReason(), state.GetDebugMessage());
    1978                 }
    1979-                assert(IsBlockReason(state.GetReason()));
    


    promag commented at 0:22 am on April 30, 2019:
    nit, this could have stayed.
  157. promag commented at 0:47 am on April 30, 2019: member
    utACK fdd7683. Change looks good despite being a bit overwhelming. Not sure what the best time to merge this but it seems we shouldn’t hold this much more considering the effort to rebase and that 0.18 was branched recently. Also, @jnewbery and @ajtowns followup improve this change further IMO.
  158. laanwj commented at 6:35 am on April 30, 2019: member

    Getting a local failure in mining_getblocktemplate_longpoll.py—might be unrelated to this change

     01/119 - mining_getblocktemplate_longpoll.py failed, Duration: 12 s
     1
     2stdout:
     32019-04-30T06:29:27.597000Z TestFramework (INFO): Initializing test directory /tmp/test_runner_₿_🏃_20190430_082919/mining_getblocktemplate_longpoll_115
     42019-04-30T06:29:28.464000Z TestFramework (INFO): Warning: this test will take about 70 seconds in the best case. Be patient.
     52019-04-30T06:29:38.491000Z TestFramework (ERROR): Assertion failed
     6Traceback (most recent call last):
     7  File ".../bitcoin/test/functional/test_framework/test_framework.py", line 175, in main
     8    self.run_test()
     9  File ".../bitcoin/test/functional/mining_getblocktemplate_longpoll.py", line 54, in run_test
    10    assert not thr.is_alive()
    11AssertionError
    122019-04-30T06:29:38.544000Z TestFramework (INFO): Stopping nodes
    132019-04-30T06:29:39.303000Z TestFramework (WARNING): Not cleaning up dir /tmp/test_runner_₿_🏃_20190430_082919/mining_getblocktemplate_longpoll_115
    142019-04-30T06:29:39.304000Z TestFramework (ERROR): Test failed. Test logging available at /tmp/test_runner_₿_🏃_20190430_082919/mining_getblocktemplate_longpoll_115/test_framework.log
    152019-04-30T06:29:39.305000Z TestFramework (ERROR): Hint: Call .../bitcoin/test/functional/combine_logs.py '/tmp/test_runner_₿_🏃_20190430_082919/mining_getblocktemplate_long
    16poll_115' to consolidate all logs
    17
    18
    19stderr:
    20Exception in thread Thread-1:
    21Traceback (most recent call last):
    22  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    23    self.run()
    24  File ".../bitcoin/test/functional/mining_getblocktemplate_longpoll.py", line 25, in run
    25    self.node.getblocktemplate({'longpollid': self.longpollid, 'rules': ['segwit']})
    26  File ".../bitcoin/test/functional/test_framework/coverage.py", line 47, in __call__
    27    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
    28  File ".../bitcoin/test/functional/test_framework/authproxy.py", line 141, in __call__
    29    raise JSONRPCException(response['error'], status)
    30test_framework.authproxy.JSONRPCException: Shutting down (-9)
    

    Edit: this only happened the first time, cannot reproduce it :(

  159. promag commented at 7:13 am on April 30, 2019: member
    @laanwj so it took getblocktemplate more than 5 seconds? I don’t remember seeing this failure.
  160. in src/validation.cpp:3106 in f3883a321b outdated
    3105-        if (!CheckTransaction(*tx, state, true))
    3106-            return state.Invalid(false, state.GetRejectCode(), state.GetRejectReason(),
    3107-                                 strprintf("Transaction check failed (tx hash %s) %s", tx->GetHash().ToString(), state.GetDebugMessage()));
    3108+    for (const auto& tx : block.vtx) {
    3109+        if (!CheckTransaction(*tx, state, true)) {
    3110+            LogPrintf("Transaction check failed (tx hash %s) %s\n", tx->GetHash().ToString(), state.GetDebugMessage());
    


    laanwj commented at 7:14 am on April 30, 2019:

    Two things here:

    • I think the point here was to add the specific failed tx hash as context information
    • Not sure it’s a good idea to add LogPrintf in validation. We’ve spent a lot of time some years ago to remove all direct logging in validation to reduce the spamminess for users (don’t want ERRORs in the log for things that are not the users’s fault), replacing it with messages in the validation state, so that one summary message can be given at most (and returned over say, RPC)

    sdaftuar commented at 7:11 pm on May 2, 2019:
    Thanks, I see your point (and good catch on the unconditional log message, I agree that we should not do that). I’ll revert this change.
  161. in src/validation.cpp:3123 in fdd7683c55 outdated
    3125-        if (!CheckTransaction(*tx, state, true))
    3126-            return state.Invalid(false, state.GetRejectCode(), state.GetRejectReason(),
    3127-                                 strprintf("Transaction check failed (tx hash %s) %s", tx->GetHash().ToString(), state.GetDebugMessage()));
    3128+    for (const auto& tx : block.vtx) {
    3129+        if (!CheckTransaction(*tx, state, true)) {
    3130+            LogPrintf("Transaction check failed (tx hash %s) %s\n", tx->GetHash().ToString(), state.GetDebugMessage());
    


    practicalswift commented at 7:25 am on April 30, 2019:
    Use LogPrint(BCLog, …) instead to log to a specific log category and thus reduce the default log noise?
  162. in src/net_processing.cpp:2552 in fdd7683c55 outdated
    2552-                int nDoS = 0;
    2553-                if (!state.IsInvalid(nDoS) || nDoS == 0) {
    2554+                // Never relay transactions that might result in being
    2555+                // disconnected (or banned).
    2556+                if (state.IsInvalid() && TxRelayMayResultInDisconnect(state)) {
    2557+                    LogPrintf("Not relaying invalid transaction %s from whitelisted peer=%d (%s)\n", tx.GetHash().ToString(), pfrom->GetId(), FormatStateMessage(state));
    


    practicalswift commented at 7:25 am on April 30, 2019:
    Use LogPrint(BCLog, …) instead to log to a specific log category and thus reduce the default log noise?
  163. in src/consensus/tx_check.cpp:14 in dd6429e50f outdated
    10@@ -11,24 +11,24 @@ bool CheckTransaction(const CTransaction& tx, CValidationState &state, bool fChe
    11 {
    12     // Basic checks that don't depend on any context
    13     if (tx.vin.empty())
    14-        return state.DoS(100, false, REJECT_INVALID, "bad-txns-vin-empty");
    15+        return state.DoS(100, ValidationInvalidReason::CONSENSUS, false, REJECT_INVALID, "bad-txns-vin-empty");
    


    laanwj commented at 7:25 am on April 30, 2019:

    There’s a very large correlation with the reject codes here e.g. ValidationInvalidReason::CONSENSUS<->REJECT_INVALID. Not really a problem, though adding another way to classify every instance of Invalid/DoS seems a bit overkill and the number of parameters keeps growing.

    Edit: couldn’t we replace the reject code with a mapping from ValidationInvalidReason?


    jnewbery commented at 2:00 pm on April 30, 2019:
    See commit https://github.com/bitcoin/bitcoin/pull/15921/commits/903d68f21da8eaa9f397232f54d35b8c58de8851 in follow-up PR #15921 which removes the reject code from ValidationState.
  164. laanwj commented at 7:57 am on April 30, 2019: member

    Edit: couldn’t we replace the reject code with a mapping from ValidationInvalidReason?

    So to check this I’ve analyzed all occurences of Invalid (with this PR applied):

    Occurences

     0src/consensus/tx_check.cpp:14       ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
     1src/consensus/tx_check.cpp:16       ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
     2src/consensus/tx_check.cpp:19       ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
     3src/consensus/tx_check.cpp:26       ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
     4src/consensus/tx_check.cpp:28       ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
     5src/consensus/tx_check.cpp:31       ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
     6src/consensus/tx_check.cpp:40       ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
     7src/consensus/tx_check.cpp:47       ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
     8src/consensus/tx_check.cpp:53       ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
     9src/consensus/tx_verify.cpp:163     ValidationInvalidReason::TX_MISSING_INPUTS            REJECT_INVALID
    10src/consensus/tx_verify.cpp:175     ValidationInvalidReason::TX_PREMATURE_SPEND           REJECT_INVALID
    11src/consensus/tx_verify.cpp:182     ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    12src/consensus/tx_verify.cpp:188     ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    13src/consensus/tx_verify.cpp:195     ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    14src/validation.cpp:588              ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    15src/validation.cpp:593              ValidationInvalidReason::TX_NOT_STANDARD              REJECT_NONSTANDARD
    16src/validation.cpp:599              ValidationInvalidReason::TX_NOT_STANDARD              REJECT_NONSTANDARD
    17src/validation.cpp:605              ValidationInvalidReason::TX_PREMATURE_SPEND           REJECT_NONSTANDARD
    18src/validation.cpp:609              ValidationInvalidReason::TX_CONFLICT                  REJECT_DUPLICATE
    19src/validation.cpp:645              ValidationInvalidReason::TX_MEMPOOL_POLICY            REJECT_DUPLICATE
    20src/validation.cpp:675              ValidationInvalidReason::TX_CONFLICT                  REJECT_DUPLICATE
    21src/validation.cpp:698              ValidationInvalidReason::TX_PREMATURE_SPEND           REJECT_NONSTANDARD
    22src/validation.cpp:707              ValidationInvalidReason::TX_NOT_STANDARD              REJECT_NONSTANDARD
    23src/validation.cpp:711              ValidationInvalidReason::TX_WITNESS_MUTATED           REJECT_NONSTANDARD
    24src/validation.cpp:735              ValidationInvalidReason::TX_NOT_STANDARD              REJECT_NONSTANDARD
    25src/validation.cpp:740              ValidationInvalidReason::TX_MEMPOOL_POLICY            REJECT_INSUFFICIENTFEE
    26src/validation.cpp:745              ValidationInvalidReason::TX_MEMPOOL_POLICY            REJECT_INSUFFICIENTFEE
    27src/validation.cpp:749              ValidationInvalidReason::TX_NOT_STANDARD              REJECT_HIGHFEE
    28src/validation.cpp:761              ValidationInvalidReason::TX_MEMPOOL_POLICY            REJECT_NONSTANDARD
    29src/validation.cpp:773              ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    30src/validation.cpp:815              ValidationInvalidReason::TX_MEMPOOL_POLICY            REJECT_INSUFFICIENTFEE
    31src/validation.cpp:843              ValidationInvalidReason::TX_MEMPOOL_POLICY            REJECT_NONSTANDARD
    32src/validation.cpp:862              ValidationInvalidReason::TX_MEMPOOL_POLICY            REJECT_NONSTANDARD
    33src/validation.cpp:874              ValidationInvalidReason::TX_MEMPOOL_POLICY            REJECT_INSUFFICIENTFEE
    34src/validation.cpp:884              ValidationInvalidReason::TX_MEMPOOL_POLICY            REJECT_INSUFFICIENTFEE
    35src/validation.cpp:905              ValidationInvalidReason::TX_WITNESS_MUTATED           REJECT_passthrough
    36src/validation.cpp:965              ValidationInvalidReason::TX_MEMPOOL_POLICY            REJECT_INSUFFICIENTFEE
    37src/validation.cpp:1437             ValidationInvalidReason::TX_NOT_STANDARD              REJECT_NONSTANDARD
    38src/validation.cpp:1448             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    39src/validation.cpp:1943             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    40src/validation.cpp:1988             ValidationInvalidReason::CONSENSUS                    REJECT_passthrough
    41src/validation.cpp:1995             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    42src/validation.cpp:2008             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    43src/validation.cpp:2019             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    44src/validation.cpp:2035             ValidationInvalidReason::CONSENSUS                    REJECT_passthrough
    45src/validation.cpp:2055             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    46src/validation.cpp:2061             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    47src/validation.cpp:3087             ValidationInvalidReason::BLOCK_INVALID_HEADER         REJECT_INVALID
    48src/validation.cpp:3109             ValidationInvalidReason::BLOCK_MUTATED                REJECT_INVALID
    49src/validation.cpp:3115             ValidationInvalidReason::BLOCK_MUTATED                REJECT_INVALID
    50src/validation.cpp:3126             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    51src/validation.cpp:3130             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    52src/validation.cpp:3133             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    53src/validation.cpp:3149             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    54src/validation.cpp:3258             ValidationInvalidReason::BLOCK_INVALID_HEADER         REJECT_INVALID
    55src/validation.cpp:3267             ValidationInvalidReason::BLOCK_CHECKPOINT             REJECT_CHECKPOINT
    56src/validation.cpp:3272             ValidationInvalidReason::BLOCK_INVALID_HEADER         REJECT_INVALID
    57src/validation.cpp:3276             ValidationInvalidReason::BLOCK_TIME_FUTURE            REJECT_INVALID
    58src/validation.cpp:3283             ValidationInvalidReason::BLOCK_INVALID_HEADER         REJECT_OBSOLETE
    59src/validation.cpp:3313             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    60src/validation.cpp:3323             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    61src/validation.cpp:3345             ValidationInvalidReason::BLOCK_MUTATED                REJECT_INVALID
    62src/validation.cpp:3349             ValidationInvalidReason::BLOCK_MUTATED                REJECT_INVALID
    63src/validation.cpp:3359             ValidationInvalidReason::BLOCK_MUTATED                REJECT_INVALID
    64src/validation.cpp:3371             ValidationInvalidReason::CONSENSUS                    REJECT_INVALID
    65src/validation.cpp:3391             ValidationInvalidReason::CACHED_INVALID               REJECT_0
    66src/validation.cpp:3402             ValidationInvalidReason::BLOCK_MISSING_PREV           REJECT_0
    67src/validation.cpp:3405             ValidationInvalidReason::BLOCK_INVALID_PREV           REJECT_INVALID
    68src/validation.cpp:3442             ValidationInvalidReason::BLOCK_INVALID_PREV           REJECT_INVALID
    

    Mapping

     0ValidationInvalidReason::BLOCK_CHECKPOINT -> REJECT_CHECKPOINT
     1ValidationInvalidReason::BLOCK_INVALID_HEADER -> REJECT_INVALID
     2ValidationInvalidReason::BLOCK_INVALID_HEADER -> REJECT_OBSOLETE
     3ValidationInvalidReason::BLOCK_INVALID_PREV -> REJECT_INVALID
     4ValidationInvalidReason::BLOCK_MISSING_PREV -> REJECT_0
     5ValidationInvalidReason::BLOCK_MUTATED -> REJECT_INVALID
     6ValidationInvalidReason::BLOCK_TIME_FUTURE -> REJECT_INVALID
     7ValidationInvalidReason::CACHED_INVALID -> REJECT_0
     8ValidationInvalidReason::CONSENSUS -> REJECT_INVALID
     9ValidationInvalidReason::TX_CONFLICT -> REJECT_DUPLICATE
    10ValidationInvalidReason::TX_MEMPOOL_POLICY -> REJECT_DUPLICATE
    11ValidationInvalidReason::TX_MEMPOOL_POLICY -> REJECT_INSUFFICIENTFEE
    12ValidationInvalidReason::TX_MEMPOOL_POLICY -> REJECT_NONSTANDARD
    13ValidationInvalidReason::TX_MISSING_INPUTS -> REJECT_INVALID
    14ValidationInvalidReason::TX_NOT_STANDARD -> REJECT_HIGHFEE
    15ValidationInvalidReason::TX_NOT_STANDARD -> REJECT_NONSTANDARD
    16ValidationInvalidReason::TX_PREMATURE_SPEND -> REJECT_INVALID
    17ValidationInvalidReason::TX_PREMATURE_SPEND -> REJECT_NONSTANDARD
    18ValidationInvalidReason::TX_WITNESS_MUTATED -> REJECT_NONSTANDARD
    

    Doesn’t look like a direct mapping is possible with the current reasons¸ unfortunately, though it’s close—there’s only a few exceptions where the reject code provides more elaboration:

    • ValidationInvalidReason::TX_PREMATURE_SPEND is non-standard in AcceptToMemoryPoolWorker (two occurences) but invalid in Consensus::CheckTxInputs
    • ValidationInvalidReason::BLOCK_INVALID_HEADER maps to REJECT_OBSOLETE (when concerning block versions) or REJECT_INVALID(everything else)
    • ValidationInvalidReason::TX_MEMPOOL_POLICY maps to various different REJECT reasons
    • ValidationInvalidReason::TX_NOT_STANDARD maps to either high fee or nonstandard
  165. laanwj commented at 8:09 am on April 30, 2019: member
    • apart from f3883a321bf4ab289edcd9754b12cae3a648b175 whose rationale I disagree with (I don’t think it’s redundant, as commented), I think this is overall a move in the right direction
    • my last comment could be addressed (if at all) in a follow-up PR

    utACK fdd7683c558984a96dd556e2c93dde156b85a75f

  166. Drop obsolete sigops comment
    This comment was confusing and incorrect when first added ("invalid rather than
    merely non-standard" has the opposite meaning of what is actually the case),
    and was also not updated after segwit with the correct variable names.
    
    Delete it since the code reads just fine on its own.
    
    Co-authored by: Anthony Towns <aj@erisian.com.au>
                    Suhas Daftuar <sdaftuar@gmail.com>
    f34fa719cf
  167. [refactor] rename stateDummy -> orphan_state
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
                    Suhas Daftuar <sdaftuar@gmail.com>
    00e11e61c0
  168. [refactor] Refactor misbehavior ban decisions to MaybePunishNode()
    Isolate the decision of whether to ban a peer to one place in the
    code, rather than having it sprinkled throughout net_processing.
    
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
                    Suhas Daftuar <sdaftuar@gmail.com>
                    John Newbery <john@johnnewbery.com>
    8818729013
  169. [refactor] drop IsInvalid(nDoSOut)
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    b8b4c80146
  170. Clean up banning levels
    Compared with previous bans, the following changes are made:
     * Txn with empty vin/vout or null prevouts move from 10 DoS
       points to 100.
     * Loose transactions with a dependency loop now result in a ban
       instead of 10 DoS points.
     * Many pre-segwit soft-fork errors now result in a ban.
       Note: Transactions that violate soft-fork script flags since P2SH do not generally
       result in a ban. Also, banning behavior for invalid blocks is dependent on
       whether the node is validating with multiple script check threads, due to a long-
       standing bug. That inconsistency is still present after this commit.
     * Proof of work failure moves from 50 DoS points to a ban.
     * Blocks with timestamps under MTP now result in a ban, blocks
       too far in the future continue to *not* result in a ban.
     * Inclusion of non-final transactions in a block now results in a
       ban instead of 10 DoS points.
    
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    7b999103e2
  171. Ban all peers for all block script failures
    This eliminates a discrepancy between block validation with multiple
    script check threads, versus a single script check thread.
    6a7f8777a0
  172. [refactor] Add useful-for-dos "reason" field to CValidationState
    This is a first step towards cleaning up our DoS interface - make
    validation return *why* something is invalid, and let net_processing
    figure out what that implies in terms of banning/disconnection/etc.
    
    Behavior change: peers will now be banned for providing blocks
    with premature coinbase spends.
    
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
                    Suhas Daftuar <sdaftuar@gmail.com>
    34477ccd39
  173. [refactor] Drop redundant nDoS, corruptionPossible, SetCorruptionPossible
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    c8b0d22698
  174. LookupBlockIndex -> CACHED_INVALID
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    7df16e70e6
  175. CorruptionPossible -> TX_WITNESS_MUTATED
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    6e55b292b0
  176. CorruptionPossible -> BLOCK_MUTATED
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    9ab2a0412e
  177. [refactor] Use Reasons directly instead of DoS codes ef54b486d5
  178. Fix handling of invalid headers
    We only disconnect outbound peers (excluding HB compact block peers and manual
    connections) when receiving a CACHED_INVALID header.
    6b34bc6b6f
  179. Allow use of state.Invalid() for all reasons
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    5e78c5734b
  180. [refactor] Prep for scripted-diff by removing some \ns which annoy sed. 7721ad64f4
  181. scripted-diff: Remove DoS calls to CValidationState
    -BEGIN VERIFY SCRIPT-
    sed -i 's/\.DoS(\(.*\), REJECT_\(.*\), \(true\|false\)/.DoS(\1, REJECT_\2/' src/validation.cpp src/consensus/tx_verify.cpp src/consensus/tx_check.cpp
    sed -i 's/state.GetRejectCode(), state.GetRejectReason(), [^,]\+, state.GetDebugMessage())/state.GetRejectCode(), state.GetRejectReason(), state.GetDebugMessage())/' src/validation.cpp
    sed -i 's/\.DoS([^,]*, /.Invalid\(/' src/validation.cpp src/consensus/tx_verify.cpp src/consensus/tx_check.cpp
    -END VERIFY SCRIPT-
    
    Co-authored-by: Suhas Daftuar <sdaftuar@gmail.com>
    aa502b88d1
  182. [refactor] Drop unused state.DoS(), state.GetDoS(), state.CorruptionPossible()
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    12dbdd7a41
  183. [refactor] Update some comments in validation.cpp as we arent doing DoS there 2120c31521
  184. Assert validation reasons are contextually correct 54470e767b
  185. Separate reason for premature spends (coinbase/locktime) 0ff1c2a838
  186. sdaftuar force-pushed on May 3, 2019
  187. sdaftuar commented at 1:40 pm on May 3, 2019: member

    @laanwj Thanks for the review. I updated the PR to drop that commit, so that we once again use a CValidationState to return the debug information if CheckTransaction() fails for a transaction in a block.

    Previous version of the code is at 15141.14.

  188. jnewbery commented at 2:10 pm on May 3, 2019: member

    utACK 0ff1c2a838da9e8dc7f77609adc89124bbea3e2b

    ~Only change is dropping the Drop obsolete sigops comment commit.~

    EDIT: Copy-paste error above. The only change was dropping the Remove redundant state.Invalid() call after CheckTransaction() commit.

  189. ryanofsky approved
  190. ryanofsky commented at 2:16 pm on May 3, 2019: member
    utACK 0ff1c2a838da9e8dc7f77609adc89124bbea3e2b. Only change is dropping the first commit (f3883a321bf4ab289edcd9754b12cae3a648b175), and dropping the temporary assert(level == GetDoS()) that was in 35ee77f2832eaffce30042e00785c310c5540cdc (now c8b0d22698385f91215ce8145631e3d5826dc977)
  191. laanwj commented at 9:59 am on May 4, 2019: member
    thanks for addressing my comment, and sorry for holding this up last-minute utACK 0ff1c2a838da9e8dc7f77609adc89124bbea3e2b
  192. laanwj removed this from the "Blockers" column in a project

  193. laanwj merged this on May 4, 2019
  194. laanwj closed this on May 4, 2019

  195. laanwj referenced this in commit d7d7d31506 on May 4, 2019
  196. domob1812 referenced this in commit 7c2a5515c5 on May 6, 2019
  197. in src/validation.cpp:701 in 0ff1c2a838
    698+            return state.Invalid(ValidationInvalidReason::TX_NOT_STANDARD, false, REJECT_NONSTANDARD, "bad-txns-nonstandard-inputs");
    699 
    700         // Check for non-standard witness in P2WSH
    701         if (tx.HasWitness() && fRequireStandard && !IsWitnessStandard(tx, view))
    702-            return state.DoS(0, false, REJECT_NONSTANDARD, "bad-witness-nonstandard", true);
    703+            return state.Invalid(ValidationInvalidReason::TX_WITNESS_MUTATED, false, REJECT_NONSTANDARD, "bad-witness-nonstandard");
    


    sdaftuar commented at 3:09 pm on February 5, 2020:
    @TheBlueMatt I was tracking down our use of TX_WITNESS_MUTATED and the introduction of it here seems like a bug – any idea why we did this? TX_NOT_STANDARD seems more correct, and actually I think now that we should rename TX_WITNESS_MUTATED to TX_WITNESS_STRIPPED to make it clear how we use it.

    TheBlueMatt commented at 8:10 pm on February 5, 2020:
    It was my understanding that this could trigger in case the witness was malleated by a third part (making it nonstandard), which was the original definition for WITNESS_MUTATED.
  198. deadalnix referenced this in commit b4c7688e45 on Jun 23, 2020
  199. deadalnix referenced this in commit c02c6c9966 on Jun 23, 2020
  200. deadalnix referenced this in commit 6523d29f7c on Jun 23, 2020
  201. deadalnix referenced this in commit 980b086d18 on Jun 23, 2020
  202. deadalnix referenced this in commit 4ee2470e29 on Jun 24, 2020
  203. deadalnix referenced this in commit f74ee657f0 on Jun 24, 2020
  204. deadalnix referenced this in commit 4b7d6a093c on Jun 24, 2020
  205. deadalnix referenced this in commit 6c56712051 on Jun 24, 2020
  206. deadalnix referenced this in commit f0b9e1055c on Jun 24, 2020
  207. deadalnix referenced this in commit 599693863b on Jun 25, 2020
  208. deadalnix referenced this in commit 46872d36ce on Jun 25, 2020
  209. deadalnix referenced this in commit 70ddc17473 on Jun 25, 2020
  210. deadalnix referenced this in commit 3534a7e7ef on Jun 25, 2020
  211. deadalnix referenced this in commit 5f2783164f on Jun 25, 2020
  212. deadalnix referenced this in commit 28819304c2 on Jun 26, 2020
  213. deadalnix referenced this in commit 06c3f0a88f on Jun 26, 2020
  214. jasonbcox referenced this in commit 0704a943d9 on Jul 14, 2020
  215. DrahtBot locked this on Feb 15, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-11-17 06:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me