fix uninitialized variable nMinerConfirmationWindow #17449

It is used for the computation of BIP9WarningHeight, and by that time it isn't initialized.

I find it scary, that a bug like this can be introduced. I am wondering if there are not any tools that check the code for references to uninitialized variables?

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

No conflicts as of last run.

Fixes what? Both are already initialized..

Fixes what? Both are already initialized..

It's used for the computation of BIP9WarningHeight, and by that time it isn't initialized (I missed it first, too!).

I find it scary, that a bug like this can be introduced. I am wondering if there are not any tools that check the code for references to uninitialized variables?

The tests are being run in various sanitizers, and valgrind can do this IIRC. I'm surprised it wasn't caught.

ACK 6fcd798b42569130f85328133e2124476fb2a42a

Should fix #16697.

@promag It looks like consensus.nMinerConfirmationWindow was indeed being read uninitialised on L74 prior to the change suggested by this PR?

consensus.MinBIP9WarningHeight = consensus.SegwitHeight + consensus.nMinerConfirmationWindow;

This code was introduced in fdb3e8f8b27e3b0b2f88c32915975c6e4c299b1e in PR #16713 which was merged in to master on September 27, 2019.

Really fascinating that this 1.) wasn't caught automatically by our static and dynamic analysis, 2.) wasn't caught by our review process and 3.) was unnoticed in master for over a month. Room for improvement! :)

@bitcoinVBR

Thanks a lot for reporting this! What an excellent first time contribution! Hope to see more great contributions from you.

Sorry about the initial misunderstanding in your original PR #17433 :)

Really fascinating that this 1.) wasn't caught automatically by our static and dynamic analysis,

Port to rust when? :smile:

Oh!

Worth noting is that the read of nMinerConfirmationWindow takes place on mainnet and testnet but not on regtest:

$ grep -E '(nMinerConfirmationWindow|CBaseChainParams|MinBIP9WarningHeight)' \
      src/chainparams.cpp | grep -A2 " = *CBaseChainParams"
        strNetworkID = CBaseChainParams::MAIN;
        consensus.MinBIP9WarningHeight = consensus.SegwitHeight + consensus.nMinerConfirmationWindow;
        consensus.nMinerConfirmationWindow = 2016; // nPowTargetTimespan / nPowTargetSpacing
        strNetworkID = CBaseChainParams::TESTNET;
        consensus.MinBIP9WarningHeight = consensus.SegwitHeight + consensus.nMinerConfirmationWindow;
        consensus.nMinerConfirmationWindow = 2016; // nPowTargetTimespan / nPowTargetSpacing
        strNetworkID =  CBaseChainParams::REGTEST;
        consensus.MinBIP9WarningHeight = 0;
        consensus.nMinerConfirmationWindow = 144; // Faster than normal for regtest (144 instead of 2016)

ACK https://github.com/bitcoin/bitcoin/pull/17449/commits/6fcd798b42569130f85328133e2124476fb2a42a Really surprising that apparently no compiler spit out a warning about this uninitialized access...

Worth noting is that the read of nMinerConfirmationWindow takes place on mainnet and testnet but not on regtest:

Good catch. If there is no unit test that instantiates the main / testnet chain, that'd explain why sanitizer CI checks don't notice it. But it seems that quite a few do, for example:

pow_tests.cpp:    const auto consensus = CreateChainParams(CBaseChainParams::MAIN)->GetConsensus();

@laanwj I think there might be a more subtle issue here too - note that MinBIPWarningHeight appears to have the correct value (481824 + 2016 == 483840) when reaching Condition(…) on mainnet: #16713 (comment).

I think we might be chasing two issues here where the uninitialised read is the first one :)

Good catch. If there is no unit test that instantiates the main / testnet chain, that'd explain why sanitizer CI checks don't notice it. But it seems that quite a few do, for example:

We have functional tests that run on the testnet chain. So this should have been caught by a sanitizer (assuming one was enabled)

I don't think we have the memory sanitizer enabled anywhere

@bitcoinVBR

Thanks a lot for reporting this! What an excellent first time contribution! Hope to see more great contributions from you.

Sorry about the initial misunderstanding in your original PR #17433 :)

Thanks! I'm happy to help. I actually only found this bug because of my work on BitcoinV. I'll continue to keep an eye out for anything fishy.

Compiler writers get a lot of heat for exploiting UB to the fullest for optimisation purposes, but in this case it seems like we got really lucky -- they "fixed" our code (assuming Clang -O2) 😉

Look at the Clang -O2 case where we get the 483840 (481824 + 2016 == 483840) we intended (!) :)

$ clang++-8 -O0 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 481824
consensus.MinBIP9WarningHeight[2] == 4678000
consensus.MinBIP9WarningHeight[3] == 481824
$ clang++-8 -O1 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 586474128
consensus.MinBIP9WarningHeight[2] == 4678896
consensus.MinBIP9WarningHeight[3] == -1100057664
$ clang++-8 -O2 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 483840
consensus.MinBIP9WarningHeight[2] == 483840
consensus.MinBIP9WarningHeight[3] == 483840
$ g++-8 -O0 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 503884
consensus.MinBIP9WarningHeight[2] == 481824
consensus.MinBIP9WarningHeight[3] == 503884
$ g++-8 -O1 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 481824
consensus.MinBIP9WarningHeight[2] == 481824
consensus.MinBIP9WarningHeight[3] == 481824
$ g++-8 -O2 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 481824
consensus.MinBIP9WarningHeight[2] == 481824
consensus.MinBIP9WarningHeight[3] == 481824
$ cat repro.cpp
#include <cstdint>
#include <iostream>

namespace Consensus {
struct Params {
  int SegwitHeight;
  int MinBIP9WarningHeight;
  uint32_t nMinerConfirmationWindow;
};
} // namespace Consensus

class CChainParams {
public:
  const Consensus::Params &GetConsensus() const { return consensus; }

protected:
  CChainParams() {}

  Consensus::Params consensus;
};

class CMainParams : public CChainParams {
public:
  CMainParams() {
    consensus.SegwitHeight = 481824;
    consensus.MinBIP9WarningHeight =
        consensus.SegwitHeight + consensus.nMinerConfirmationWindow;
    consensus.nMinerConfirmationWindow = 2016;
  }
};

int main() {
  CMainParams main_params_1;
  std::cout << "consensus.MinBIP9WarningHeight[1] == "
            << main_params_1.GetConsensus().MinBIP9WarningHeight << std::endl;

  CMainParams main_params_2;
  std::cout << "consensus.MinBIP9WarningHeight[2] == "
            << main_params_2.GetConsensus().MinBIP9WarningHeight << std::endl;

  CMainParams main_params_3;
  std::cout << "consensus.MinBIP9WarningHeight[3] == "
            << main_params_3.GetConsensus().MinBIP9WarningHeight << std::endl;
}

:)

@jonatack The output you posted from the testing of PR #16713 in #16713 (comment) ...

2019-09-20T11:26:36Z SegwitHeight 481824
2019-09-20T11:26:36Z nMinerConfirmationWindow 2016
2019-09-20T11:26:36Z MinBIPWarningHeight 483840
2019-09-20T11:26:36Z pindex->nHeight 164836
2019-09-20T11:26:36Z pindex->nVersion 1
2019-09-20T11:26:36Z ComputeBlockVersion 536870912

... could it be the case that you happened to test with Clang with the project default optimisation level -O2? :) If so, could you retry using GCC (any optimisation level), or Clang with -O0 or -O1 and see if you get the same results?

@practicalswift I tried gcc 9.2.1 and it happened to optimize out the bug for me as well. Though running the same binary in valgrind still jells at me, so I can't make any sense out of that.

@MarcoFalke What is the output of ./repro for binaries compiled at the different optimisation levels?

could it be the case that you happened to test with Clang with the project default optimisation level -O2? :) If so, could you retry using GCC (any optimisation level), or Clang with -O0 or -O1 and see if you get the same results?

It was with gcc version 8.3.0 (Debian 8.3.0-6) with no optimisation flags.

It was with gcc version 8.3.0 (Debian 8.3.0-6) with no optimisation flags.

The default is O2 for Bitcoin Core

I'd prefer to just hard code the MinBIP9WarningHeight, eg:

        consensus.MinBIP9WarningHeight = 483840; // segwit activation height + miner confirmation window

Hard-coding the value means that even if the member initializations are moved around again, nMinerConfirmationWindow will not be accessed before initialization.

MinBIP9WarningHeight is currently the only member variable that is set from the value of a different member, except the hashGenesisBlock which is checked with an assert against a hard-coded value.

I was about to suggest that the members should be marked const, so that initialization and initialization order is enforced by the compiler. Though, I believe that makes the code less readable as C++ does not have named parameters that can be passed into a constructor.

@bitcoinVBR Could you please adjust the patch as suggested by @jnewbery and then squash your commits according to https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits

Has been fixed.

utACK edb6b768a

ACK edb6b768a4185a4aaa6281ee50a6538f7426cb1e, commit description could be cleaned up though.

ACK edb6b768a4185a4aaa6281ee50a6538f7426cb1e, used python3 to do the addition locally 📍

<details><summary>Show signature and timestamp</summary>

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ACK edb6b768a4185a4aaa6281ee50a6538f7426cb1e, used python3 to do the addition locally 📍
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUgIZAwAmxU36N8KM439sYBxtHQ+TI9Qy+RaHDVhm6ccEcQfGXIh0O/AhPyqwRmE
kq/t2Ukk2tp6tya/qGNTYqGXBvgMIcH7w7QcWSt+/kcGz/t5isrMG3I9A6IB4ZVn
DGC57cd3A+kge+DZkp5pQcTqHZvSKVFD0jCF0yF5ui6+2u7yyCCY3YaWJfaqvZt1
w6ZYv3VBLPBgls3PLzzU+iho/GKivNlGSaReFdynlZ57gzrWt9E4eQMIifkf5tNo
DBiJh5Jeodf8C/j6IDiL6l9gzPXUFzxLCoOctJu9uvHpnT5GXNO5GfeUmNDjLJYp
72gYa7/UZkg+kl1a0mF32UNhzm4PObfeQn9UqH0MwTuJzdOgGqaNIwz6oISPdNd2
1pwXaBsbfgrusHKPd8GXlFFhfm7P+ICNwDUJA5aJWvxPkbQn4ZM2l+OXqUryP9xW
76tR8c9G0xP7sYDNwfEKp5x9BaSSeR/f9dauOwndRMYazqLYRrb3p8vKDa8yHcs5
pjdRxFk8
=te4h
-----END PGP SIGNATURE-----

Timestamp of file with hash 346fefa7ddee2dc8d6ced4fd005240f5e06d18da505f8b0edfea1d7208fcec73 -

</details>

ACK edb6b768a4185a4aaa6281ee50a6538f7426cb1e, used clang++ -O2 on the previous version^W^W^W^W^W^Wbc to verify the addition locally 🏓

Code review ACK edb6b76. Nit: commit description has duplicate text.

FWIW, Valgrind warns about this when running a bitcoind binary compiled from current master using GCC with -O0:

==19043== Thread 25 b-msghand:
==19043== Conditional jump or move depends on uninitialised value(s)
==19043==    at 0x49E12D: WarningBitsConditionChecker::Condition(CBlockIndex const*, Consensus::Params const&) const (validation.cpp:1815)
==19043==    by 0x541AED: AbstractThresholdConditionChecker::GetStateFor(CBlockIndex const*, Consensus::Params const&, std::map<CBlockIndex const*, ThresholdState, std::less<CBlockIndex const*>, std::allocator<std::pair<CBlockIndex const* const, ThresholdState> > >&) const (versionbits.cpp:70)
==19043==    by 0x484248: UpdateTip(CBlockIndex const*, CChainParams const&) (validation.cpp:2379)
==19043==    by 0x4854A1: CChainState::ConnectTip(BlockValidationState&, CChainParams const&, CBlockIndex*, std::shared_ptr<CBlock const> const&, ConnectTrace&, DisconnectedBlockTransactions&) (validation.cpp:2584)
==19043==    by 0x485FD8: CChainState::ActivateBestChainStep(BlockValidationState&, CChainParams const&, CBlockIndex*, std::shared_ptr<CBlock const> const&, bool&, ConnectTrace&) (validation.cpp:2715)
==19043==    by 0x4866B2: CChainState::ActivateBestChain(BlockValidationState&, CChainParams const&, std::shared_ptr<CBlock const>) (validation.cpp:2836)
==19043==    by 0x48CFAC: ProcessNewBlock(CChainParams const&, std::shared_ptr<CBlock const>, bool, bool*) (validation.cpp:3792)
==19043==    by 0x21AA69: ProcessMessage(CNode*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CDataStream&, long, CChainParams const&, CConnman*, BanMan*, std::atomic<bool> const&) (net_processing.cpp:2974)
==19043==    by 0x21D80F: PeerLogicValidation::ProcessMessages(CNode*, std::atomic<bool>&) (net_processing.cpp:3331)
==19043==    by 0x1BDD83: CConnman::ThreadMessageHandler() (net.cpp:2013)
==19043==    by 0x202E2F: void std::__invoke_impl<void, void (CConnman::*&)(), CConnman*&>(std::__invoke_memfun_deref, void (CConnman::*&)(), CConnman*&) (invoke.h:73)
==19043==    by 0x1FE55C: std::__invoke_result<void (CConnman::*&)(), CConnman*&>::type std::__invoke<void (CConnman::*&)(), CConnman*&>(void (CConnman::*&)(), CConnman*&) (invoke.h:95)

@bitcoinVBR Thanks again for reporting! May I ask how you noticed this issue? :)

@practicalswift he said how above #17449 (comment)

@promag Yes, I understood that it was found in the context of @bitcoinVBR's work on a fork, but I didn't get how it was found. More specifically if it was caught by manual source code review, static analysis or dynamic analysis :)

I was updating BitcoinV to add the variable block reward feature based on Bitcoin v0.19.0 and my eye noticed the variable used before being initialized.

Are there any bounty rewards for these finds?

17yVfEbGZqX1Ce4qTmv4QD29DfeohPq2zi

Thanks, BitcoinV

Sent from my iPhone

On Nov 14, 2019, at 7:13 AM, practicalswift notifications@github.com wrote:

 FWIW, Valgrind warns about this when running a bitcoind binary compiled from current master using GCC with -O0:

==19043== Thread 25 b-msghand: ==19043== Conditional jump or move depends on uninitialised value(s) ==19043== at 0x49E12D: WarningBitsConditionChecker::Condition(CBlockIndex const, Consensus::Params const&) const (validation.cpp:1815) ==19043== by 0x541AED: AbstractThresholdConditionChecker::GetStateFor(CBlockIndex const, Consensus::Params const&, std::map<CBlockIndex const, ThresholdState, std::less<CBlockIndex const>, std::allocator<std::pair<CBlockIndex const* const, ThresholdState> > >&) const (versionbits.cpp:70) ==19043== by 0x484248: UpdateTip(CBlockIndex const, CChainParams const&) (validation.cpp:2379) ==19043== by 0x4854A1: CChainState::ConnectTip(BlockValidationState&, CChainParams const&, CBlockIndex, std::shared_ptr<CBlock const> const&, ConnectTrace&, DisconnectedBlockTransactions&) (validation.cpp:2584) ==19043== by 0x485FD8: CChainState::ActivateBestChainStep(BlockValidationState&, CChainParams const&, CBlockIndex, std::shared_ptr<CBlock const> const&, bool&, ConnectTrace&) (validation.cpp:2715) ==19043== by 0x4866B2: CChainState::ActivateBestChain(BlockValidationState&, CChainParams const&, std::shared_ptr<CBlock const>) (validation.cpp:2836) ==19043== by 0x48CFAC: ProcessNewBlock(CChainParams const&, std::shared_ptr<CBlock const>, bool, bool) (validation.cpp:3792) ==19043== by 0x21AA69: ProcessMessage(CNode, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CDataStream&, long, CChainParams const&, CConnman, BanMan, std::atomic<bool> const&) (net_processing.cpp:2974) ==19043== by 0x21D80F: PeerLogicValidation::ProcessMessages(CNode, std::atomic<bool>&) (net_processing.cpp:3331) ==19043== by 0x1BDD83: CConnman::ThreadMessageHandler() (net.cpp:2013) ==19043== by 0x202E2F: void std::__invoke_impl<void, void (CConnman::&)(), CConnman&>(std::__invoke_memfun_deref, void (CConnman::&)(), CConnman&) (invoke.h:73) ==19043== by 0x1FE55C: std::__invoke_result<void (CConnman::&)(), CConnman&>::type std::__invoke<void (CConnman::&)(), CConnman&>(void (CConnman::&)(), CConnman&) (invoke.h:95) @bitcoinVBR Thanks again for reporting! May I ask how you noticed this issue? :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Are there any bounty rewards for these finds?

The Bitcoin Core project does not have any official bounty program AFAIK, but the is nothing stopping individuals from donating to other individuals who they think have done valuable security research.

(Aside: My personal view is that we could do a much better job recognising good security research: especially the non-glamorous kind of low-level security research. As an example: if we had a proper incentive structure in place I guess someone would be running bitcoind binaries compiled from master with -O{0,1,2} running under valgrind around the clock just waiting to report an issue like this. Incentives matter. Unfortunately I don't know the optimal incentive structure (the mix between public recognition vs official bug bounty vs donations vs just being polite and thanking researchers vs long-term funding, etc.), but I'm fairly certain we're not at the global optimum right now :))

Shameless plug: People interested in running bitcoind under valgrind might want to help out by reviewing #17455 :)

Maybe you can get funding for such a thing from one of the exchanges or companies in the space, that's always how developer funding has worked, but the 'bitcoin core' project is a loose name for people contributing to an open source project, and does not manage funds, and should not manage funds, and will never have an "official incentive structure".

[…] the 'bitcoin core' project is a loose name for people contributing to an open source project, and does not manage funds, and should not manage funds, and will never have an "official incentive structure".

Agree 100% if we are talking monetary reward.

Note that incentive structure does not necessarily imply monetary reward: giving proper credit and treating researchers nicely goes a surprisingly long way when it comes to attracting security research :)

To bring this on-topic again (sorry, my fault!): I hope we can make this a good example by making sure @bitcoinVBR is properly credited for allowing us to ship 0.19.0 without an "optimisation unstable" UpdateTip(…) (due to the uninitialised read) :)

As all contributors to the release, they've been added to the authors list in 0.19.0.1 (as NullFunctor, as that's the name on their git mail address, @bitcoinVBR: if you want to be credited under a different name let me know) and this PR has been added to the changelog.

This is not a security issue so I'm not sure why you bring that up. It doesn't matter either, bugs are bugs. All people that contribute deserve credit.

Backported to 0.19 in https://github.com/bitcoin/bitcoin/commit/6ec0dc195dc6db2c949acdf3c54a2c3b22c23f65.

ACK on the rebase 6ec0dc1.

consensus.MinBIP9WarningHeight = consensus.SegwitHeight + consensus.nMinerConfirmationWindow;

Really fascinating that this 1.) wasn't caught automatically by our static and dynamic analysis,

Looks like -Werror=uninitialized is able to catch this, but only if the initialisation is done in a Consensus::Params constructor via a member initializer list rather than a code block (and -O3 is specified if using g++). (I thought I was getting g++ to catch it in code blocks as well, but can't duplicate that now)

@ajtowns Worth mentioning from a dynamic analysis perspective is that both Valgrind and MemorySanitizer (-fsanitize=memory) catch this too :)