intermittent issue in wallet_upgradewallet.py: AssertionError: bdb magic does not match bdb btree magic #31210

1. 
   fanquake commented at 12:29 pm on November 4, 2024: member

   https://cirrus-ci.com/task/5232872305459200:

    0067]  node0 2024-11-04T11:58:48.064759Z [httpworker.0] [src/wallet/wallet.h:936] [WalletLogPrintf] [default wallet] m_address_book.size() = 2 
    1[06:58:49.067]  test  2024-11-04T11:58:48.065000Z TestFramework (INFO): Can upgrade to HD 
    2[06:58:49.067]  test  2024-11-04T11:58:48.066000Z TestFramework (ERROR): Assertion failed 
    3[06:58:49.067]                                    Traceback (most recent call last):
    4[06:58:49.067]                                      File "/ci_container_base/test/functional/test_framework/test_framework.py", line 132, in main
    5[06:58:49.067]                                        self.run_test()
    6[06:58:49.067]                                      File "/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/test/functional/wallet_upgradewallet.py", line 226, in run_test
    7[06:58:49.067]                                        orig_kvs = dump_bdb_kv(node_master_wallet)
    8[06:58:49.067]                                      File "/ci_container_base/test/functional/test_framework/bdb.py", line 142, in dump_bdb_kv
    9[06:58:49.067]                                        dump_meta_page(pages[INNER_META_PAGE])
   10[06:58:49.067]                                      File "/ci_container_base/test/functional/test_framework/bdb.py", line 100, in dump_meta_page
   11[06:58:49.067]                                        assert magic == BTREE_MAGIC, 'bdb magic does not match bdb btree magic'
   12[06:58:49.067]                                    AssertionError: bdb magic does not match bdb btree magic
   13[06:58:49.067]  test  2024-11-04T11:58:48.067000Z TestFramework (DEBUG): Closing down network thread 
   

   Looking at all the logs in #30798, this seems to be a different issue.

2. 
   maflcko commented at 12:33 pm on November 4, 2024: member

   This only reproduces on a specific machine. I’ll try to take a closer look.
3. maflcko added the label CI failed on Nov 4, 2024
4. 
   maflcko commented at 3:49 pm on November 4, 2024: member

   Steps to reproduce on a fresh Ubuntu 24.04 podman container on that machine:

   export DEBIAN_FRONTEND=noninteractive && apt update && apt install curl wget htop git vim ccache -y && git clone https://github.com/bitcoin/bitcoin.git --depth=1 ./b-c && cd b-c && apt install build-essential cmake pkg-config libevent-dev libboost-dev libsqlite3-dev libdb++-dev python3 -y && cmake -B ./bld-cmake -DBUILD_TESTS=OFF -DWITH_BDB=ON -DENABLE_WALLET=ON -DBUILD_GUI=OFF -DBUILD_FUZZ_BINARY=OFF -DBUILD_BENCH=OFF -DWITH_ZMQ=OFF && cmake --build ./bld-cmake --parallel $(nproc) && test/get_previous_releases.py -b && ./bld-cmake/test/functional/wallet_upgradewallet.py --legacy-wallet

5. 
   maflcko commented at 5:09 pm on November 4, 2024: member

   Possibly related: https://github.com/containers/podman/issues/23808

   I’ll try tomorrow without a container.

6. 
   maflcko commented at 11:08 am on November 5, 2024: member

   Not sure how to fix this. IIUC neither podman (https://github.com/containers/podman/issues/11415#issuecomment-912015581), nor docker (https://docs.docker.com/engine/security/rootless/#known-limitations) support rootless zfs.
7. 
   achow101 commented at 5:36 pm on November 5, 2024: member

   I don’t think this issue has to do with ZFS. I’m debugging the test on the problematic system and it seems that BDB decided to use a page size of 16384 for some reason, rather than our expected and hardcoded 4096. Changing it to 16384 resolves this, but we should instead be using the pagesize given by the wallet file rather than a fixed pagesize. This is a test only issue as the C++ parser does not use a fixed pagesize.
8. 
   maflcko commented at 1:56 pm on November 6, 2024: member

   Thanks, tested that #31222 (74ff8467d107b609d699729399b83a1219add78a) or #30125 (d45eb3964f693eddcf96f1e4083cf19d327be989) fix the issue.

   I reproduced in a fresh install of Ubuntu 24.10 VM with zfs (experimental) selected, then ran the test in podman.

Contributors
fanquake maflcko achow101

Labels
CI failed