Potential crash (assert) rescanning wallet #31474

issue psgreco openend this issue on December 12, 2024
  1. psgreco commented at 0:17 am on December 12, 2024: contributor

    Is there an existing issue for this?

    • I have searched the existing issues

    Current behaviour

    Preface: This whole analysis was made using Elements Core, not Bitcoin Core, but the code and the conditions should line up in a way that makes this likely in Bitcoin too.

    There’s an assert crash in assert(chain_depth >= 0); // coinbase tx should not be conflicted calling CWallet::GetTxBlocksToMaturity because CWallet::GetTxDepthInMainChain returns a negative value for TxStateConfirmed txs. The reason for this is that m_last_block_processed is set at the beginning of AttachChain, but the rescan loop keeps going beyond that point, so when GetTxDepthInMainChain is called right after the scan is done, m_last_block_processed is lower than the actual block in the latest txs, hence returning a negative value and crashing.

    In discussions with @achow101 , it’s possible that #30221 fixes this issue by calling SetBestBlock after the rescan is done. Another approach could be to stop the scan when m_last_block_processed is reached with something like this

     0@@ -1689,6 +1693,11 @@ CWallet::ScanResult CWallet::ScanForWalletTransactions(const uint256& start_bloc
     1         if (max_height && block_height >= *max_height) {
     2             break;
     3         }
     4+        if (block_height >= GetLastBlockHeight()) {
     5+            WalletLogPrintf("Stopping scan. At block %d. Progress=%f\n", block_height, progress_current);
     6+            break;
     7+        }
     8+
     9         {
    10             if (!next_block) {
    11                 // break successfully when rescan has reached the tip, or
    

    Expected behaviour

    The wallet data should be consistent to avoid the crash

    Steps to reproduce

    This is reproduced using QT

    1. Create a chain with many coinbase txs.
    2. Attach a wallet (could be read only) with the address of the coinbase txs
    3. While the wallet is still being loaded, create many more blocks with coinbase txs to that address
    4. Assert happens when the wallet is done loading

    Relevant log output

    No response

    How did you obtain Bitcoin Core

    Compiled from source

    What version of Bitcoin Core are you using?

    master@62bd61de110b057cbfd6e31e4d0b727d93119c72

    Operating system and version

    Fedora 41

    Machine specifications

    No response

  2. mzumsande commented at 8:38 pm on January 7, 2025: contributor

    I tried to reproduce the crash locally (putting a sleep into AttachChain) but didn’t manage - Not the biggest GUI or wallet expert so I may well be missing something, but here’s what I tried:

    • With qt, how do I create more transactions while the wallet is loading? The GUI blocks for me until the wallet is loaded.
    • With bitcoind, I was able create more blocks during the rescan - but those will create blockConnected() notifications, and several RPCs that call GetTxBlocksToMaturity (such as getbalance) use BlockUntilSyncedToCurrentChain, waiting for the notifications to be processed and causing m_last_block_processed to be set to the correct value when GetTxBlocksToMaturity is finally called.

    Even if I couldn’t reproduce the crash, rescanning blocks higher than m_last_block_processed seems wrong, so the suggested fix makes sense to me.

  3. maflcko commented at 9:54 pm on January 7, 2025: member

    With qt, how do I create more transactions while the wallet is loading? The GUI blocks for me until the wallet is loaded.

    Would it be possible to start the GUI with -server=1 and then use one of the RPC threads to create more blocks while the GUI is blocked?

  4. mzumsande commented at 11:44 pm on January 7, 2025: contributor

    Would it be possible to start the GUI with -server=1 and then use one of the RPC threads to create more blocks while the GUI is blocked?

    Good point, that worked! I could reproduce the crash in the GUI that way (with sleeps in AttachChain() and blockConnected() to avoid races) - I can confirm that it’s fixed by the patch in the OP, whereas I could still reproduce it with #30221, so that PR apparently doesn’t fix it.

  5. maflcko commented at 8:21 am on January 8, 2025: member
    Thanks for reproducing! It would be good to share the backtrace. I haven’t tried reproducing myself and I am still wondering if this is a GUI issue or a wallet issue.
  6. psgreco commented at 1:08 pm on January 8, 2025: contributor
    From my tests, it’s a wallet issue (inconsistent state), that only the GUI triggers, The wallet has an invalid state, but the “happy path” in the CLI version doesn’t notice it
  7. mzumsande commented at 4:34 pm on January 8, 2025: contributor

    From my tests, it’s a wallet issue (inconsistent state), that only the GUI triggers, The wallet has an invalid state, but the “happy path” in the CLI version doesn’t notice it

    I agree, my understanding is this: There is a temporary inconsistent state in the wallet when during a rescan the node receives additional blocks (could be via RPC or p2p), that are prematurely processed without updating m_last_block_processed. This will self-correct when the blockConnected signals for the added blocks are processed and m_last_block_processed is set to its correct value. But if CWallet::GetTxBlocksToMaturity is called after the rescan finished and before the signals are processed, the node crashes if the coinbase is a wallet tx. The reason why it only crashes in the GUI is that many RPC that end up calling GetTxBlocksToMaturity, such as getbalance, will first call BlockUntilSyncedToCurrentChain that would correct the issue, so I didn’t find a way to trigger. However, the GUI calls the function as part of refreshWallet / getWalletTxs, see the backtrace below.

    It would be good to share the backtrace.

     0[#0](/bitcoin-bitcoin/0/)  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
     1[#1](/bitcoin-bitcoin/1/)  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
     2[#2](/bitcoin-bitcoin/2/)  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
     3[#3](/bitcoin-bitcoin/3/)  0x00007ffff5e4526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
     4[#4](/bitcoin-bitcoin/4/)  0x00007ffff5e288ff in __GI_abort () at ./stdlib/abort.c:79
     5[#5](/bitcoin-bitcoin/5/)  0x00007ffff5e2881b in __assert_fail_base (fmt=0x7ffff5fd01e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x555556007eba "chain_depth >= 0", 
     6    file=file@entry=0x555556007da1 "./wallet/wallet.cpp", line=line@entry=3482, function=function@entry=0x555555fcf578 "int wallet::CWallet::GetTxBlocksToMaturity(const wallet::CWalletTx&) const")
     7    at ./assert/assert.c:94
     8[#6](/bitcoin-bitcoin/6/)  0x00007ffff5e3b507 in __assert_fail (assertion=0x555556007eba "chain_depth >= 0", file=0x555556007da1 "./wallet/wallet.cpp", line=3482, 
     9    function=0x555555fcf578 "int wallet::CWallet::GetTxBlocksToMaturity(const wallet::CWalletTx&) const") at ./assert/assert.c:103
    10[#7](/bitcoin-bitcoin/7/)  0x0000555555c2ac94 in wallet::CWallet::GetTxBlocksToMaturity (this=this@entry=0x7fff1802a650, wtx=...) at ./wallet/wallet.cpp:3482
    11[#8](/bitcoin-bitcoin/8/)  0x0000555555c2accd in wallet::CWallet::IsTxImmatureCoinBase (this=this@entry=0x7fff1802a650, wtx=...) at ./wallet/wallet.cpp:3491
    12[#9](/bitcoin-bitcoin/9/)  0x0000555555bb4b6a in wallet::CachedTxGetCredit (wallet=..., wtx=..., filter=filter@entry=@0x7fff7cbfedd8: 3) at ./wallet/receive.cpp:114
    13[#10](/bitcoin-bitcoin/10/) 0x0000555555b9cddd in wallet::(anonymous namespace)::MakeWalletTx (wallet=..., wtx=...) at ./wallet/interfaces.cpp:79
    14[#11](/bitcoin-bitcoin/11/) 0x0000555555b9de40 in wallet::(anonymous namespace)::WalletImpl::getWalletTxs (this=0x7fff18222630) at ./wallet/interfaces.cpp:354
    15[#12](/bitcoin-bitcoin/12/) 0x000055555581f832 in TransactionTablePriv::refreshWallet (this=0x7fff182927e0, wallet=...) at ./qt/transactiontablemodel.cpp:115
    16[#13](/bitcoin-bitcoin/13/) 0x000055555581d168 in TransactionTableModel::TransactionTableModel (this=this@entry=0x7fff181247f0, _platformStyle=_platformStyle@entry=0x555556b4c9a0, parent=parent@entry=0x7fff18001450)
    17    at ./qt/transactiontablemodel.cpp:259
    18[#14](/bitcoin-bitcoin/14/) 0x0000555555792bc0 in WalletModel::WalletModel (this=this@entry=0x7fff18001450, wallet=std::unique_ptr<interfaces::Wallet> = {...}, client_model=..., platformStyle=0x555556b4c9a0, 
    19    parent=parent@entry=0x0) at ./qt/walletmodel.cpp:50
    20[#15](/bitcoin-bitcoin/15/) 0x000055555577efaa in WalletController::getOrCreateWallet (this=0x5555575987c0, wallet=std::unique_ptr<interfaces::Wallet> = {...}) at /usr/include/c++/13/bits/unique_ptr.h:197
    21[#16](/bitcoin-bitcoin/16/) 0x000055555577f510 in operator() (__closure=<optimized out>, wallet=std::unique_ptr<interfaces::Wallet> = {...}) at /usr/include/c++/13/bits/unique_ptr.h:197
    22[#17](/bitcoin-bitcoin/17/) std::__invoke_impl<void, WalletController::WalletController(ClientModel&, const PlatformStyle*, QObject*)::<lambda(std::unique_ptr<interfaces::Wallet>)>&, std::unique_ptr<interfaces::Wallet, std::default_delete<interfaces::Wallet> > > (__f=...) at /usr/include/c++/13/bits/invoke.h:61
    23[#18](/bitcoin-bitcoin/18/) std::__invoke_r<void, WalletController::WalletController(ClientModel&, const PlatformStyle*, QObject*)::<lambda(std::unique_ptr<interfaces::Wallet>)>&, std::unique_ptr<interfaces::Wallet, std::default_delete<interfaces::Wallet> > > (__fn=...) at /usr/include/c++/13/bits/invoke.h:111
    24[#19](/bitcoin-bitcoin/19/) std::_Function_handler<void(std::unique_ptr<interfaces::Wallet, std::default_delete<interfaces::Wallet> >), WalletController::WalletController(ClientModel&, const PlatformStyle*, QObject*)::<lambda(std::unique_ptr<interfaces::Wallet, std::default_delete<interfaces::Wallet> >)> >::_M_invoke(const std::_Any_data &, std::unique_ptr<interfaces::Wallet, std::default_delete<interfaces::Wallet> > &&) (
    25    __functor=..., __args#0=...) at /usr/include/c++/13/bits/std_function.h:290
    26[#20](/bitcoin-bitcoin/20/) 0x0000555555c28b2f in std::function<void (std::unique_ptr<interfaces::Wallet, std::default_delete<interfaces::Wallet> >)>::operator()(std::unique_ptr<interfaces::Wallet, std::default_delete<interfaces::Wallet> >) const (__args#0=std::unique_ptr<interfaces::Wallet> = {...}, this=0x555556a12440) at /usr/include/c++/13/bits/std_function.h:591
    27[#21](/bitcoin-bitcoin/21/) wallet::NotifyWalletLoaded (context=..., wallet=std::shared_ptr<wallet::CWallet> (use count 3, weak count 0) = {...}) at ./wallet/wallet.cpp:224
    28[#22](/bitcoin-bitcoin/22/) 0x0000555555c490ed in wallet::(anonymous namespace)::LoadWalletInternal (warnings=std::vector of length 0, capacity 0, error=..., status=@0x7fff7cbff694: wallet::DatabaseStatus::SUCCESS, 
    29    options=..., load_on_start=..., name="", context=...) at ./wallet/wallet.cpp:296
    30[#23](/bitcoin-bitcoin/23/) wallet::LoadWallet (context=..., name="", load_on_start=..., load_on_start@entry=std::optional = {...}, options=..., status=@0x7fff7cbff694: wallet::DatabaseStatus::SUCCESS, error=..., 
    31    warnings=std::vector of length 0, capacity 0) at ./wallet/wallet.cpp:378
    32[#24](/bitcoin-bitcoin/24/) 0x0000555555b9873d in wallet::(anonymous namespace)::WalletLoaderImpl::loadWallet (this=0x7fff78002070, name="", warnings=std::vector of length 0, capacity 0) at /usr/include/c++/13/optional:213
    33[#25](/bitcoin-bitcoin/25/) 0x000055555577fd4c in operator() (__closure=0x55555723b0f0) at ./qt/walletcontroller.cpp:359
    34[#26](/bitcoin-bitcoin/26/) QtPrivate::FunctorCall<QtPrivate::IndexesList<>, QtPrivate::List<>, void, OpenWalletActivity::open(const std::string&)::<lambda()> >::call (arg=<optimized out>, f=...)
    35    at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:146
    36[#27](/bitcoin-bitcoin/27/) QtPrivate::Functor<OpenWalletActivity::open(const std::string&)::<lambda()>, 0>::call<QtPrivate::List<>, void> (arg=<optimized out>, f=...)
    37    at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:256
    38[#28](/bitcoin-bitcoin/28/) QtPrivate::QFunctorSlotObject<OpenWalletActivity::open(const std::string&)::<lambda()>, 0, QtPrivate::List<>, void>::impl(int, QtPrivate::QSlotObjectBase *, QObject *, void **, bool *) (
    39    which=<optimized out>, this_=0x55555723b0e0, r=<optimized out>, a=<optimized out>, ret=<optimized out>) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:443
    40[#29](/bitcoin-bitcoin/29/) 0x00007ffff6906343 in QObject::event(QEvent*) () from /lib/x86_64-linux-gnu/libQt5Core.so.5
    41[#30](/bitcoin-bitcoin/30/) 0x00007ffff796bd45 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /lib/x86_64-linux-gnu/libQt5Widgets.so.5
    42--Type <RET> for more, q to quit, c to continue without paging--
    43[#31](/bitcoin-bitcoin/31/) 0x00007ffff68d8118 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /lib/x86_64-linux-gnu/libQt5Core.so.5
    44[#32](/bitcoin-bitcoin/32/) 0x00007ffff68db94b in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /lib/x86_64-linux-gnu/libQt5Core.so.5
    45[#33](/bitcoin-bitcoin/33/) 0x00007ffff6935c0f in ?? () from /lib/x86_64-linux-gnu/libQt5Core.so.5
    46[#34](/bitcoin-bitcoin/34/) 0x00007ffff59145b5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    47[#35](/bitcoin-bitcoin/35/) 0x00007ffff5973717 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    48[#36](/bitcoin-bitcoin/36/) 0x00007ffff5913a53 in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    49[#37](/bitcoin-bitcoin/37/) 0x00007ffff6935279 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /lib/x86_64-linux-gnu/libQt5Core.so.5
    50[#38](/bitcoin-bitcoin/38/) 0x00007ffff68d6a7b in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /lib/x86_64-linux-gnu/libQt5Core.so.5
    51[#39](/bitcoin-bitcoin/39/) 0x00007ffff66da36b in QThread::exec() () from /lib/x86_64-linux-gnu/libQt5Core.so.5
    52[#40](/bitcoin-bitcoin/40/) 0x00007ffff66db674 in ?? () from /lib/x86_64-linux-gnu/libQt5Core.so.5
    53[#41](/bitcoin-bitcoin/41/) 0x00007ffff5e9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
    54[#42](/bitcoin-bitcoin/42/) 0x00007ffff5f29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
    

    I’ll open a fix (probably just @psgreco’s suggested fix, but I want to look into it a bit more).

  8. psgreco commented at 6:06 pm on January 8, 2025: contributor
    @mzumsande if you end up using my version, please look at the final version here https://github.com/ElementsProject/elements/pull/1376/files, which addresses the locking too
  9. mzumsande commented at 6:02 pm on January 9, 2025: contributor
    See #31629, I took your version, just added some doc changes.
  10. psgreco commented at 10:30 pm on January 9, 2025: contributor
    Looks good, thank you! Do you want me to ack or do anything in #31629 ?
  11. mzumsande commented at 4:06 pm on January 10, 2025: contributor
    Up to you of course, but yes, if this fixed the problem for you it’d help to say it there - thanks!
  12. laanwj added the label Wallet on Jan 14, 2025

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-01-21 06:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me