Consensus failure while upgrading bitcoind 0.8.3 > 0.22.0 #23913

During testing, a completely synchronized 0.8.3 node was updated to 0.22.0 using static release binaries. As expected, this faulted with an error describing that to continue a -reindex would need to be used to account for the missing witness data post-segwit

Opened LevelDB successfully
Using obfuscation key for /root/.bitcoin/chainstate: 0000000000000000
Upgrading utxo-set database...
[0%]...[10%]...[20%]...[30%]...[40%]...[50%]...[60%]...[70%]...[80%]...[90%]...[100%]...[DONE].
Loaded best chain: hashBestChain=00000000000000000009c8231207a2e13e246db331dc8c536454c37683d3ccdb height=715165 date=2021-12-21T23:08:50Z progress=0.997369
: Witness data for blocks after height 481824 requires validation. Please restart with -reindex..
Please restart with -reindex or -reindex-chainstate to recover.
: Witness data for blocks after height 481824 requires validation. Please restart with -reindex..
Please restart with -reindex or -reindex-chainstate to recover.
Aborted block database rebuild. Exiting.

Restarting 0.22 with bitcoind -reindex resulted in apparent consensus failure, with the node being unable to make forward progress and failing each block on disk.

ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock : invalid witness reserved value size

Restarting the node again has the software seemingly making forward progress with reindexing past height 310k without further issue or AcceptBlock() failure. There is no viable snapshot of my 0.8.3 data pre-upgrade so this can't be easily replicated, and is such an absurd situation as to almost certainly be unique, however it seemed worthy of reporting in case this is indicative of some other concerns.

The 0.22 node continued operating normally and was able to fully synchronize with no further issues, so this is truly transient and somehow related to the upgrade rather than any of the state on disk.

Shutting down the node once it had finished sync resulted in database corruption.

A clean shutdown? What kind of hardware/OS/disk? Anything in debug.log at shutdown?

2022-01-08T18:13:01.787093Z UpdateTip: new best=0000000000000000000ae927d6f1ae3a23590e445f6cfeff70b9825f8c1356db height=717757 version=0x2000e004 log2_work=93.275074 tx=701213772 date='2022-01-08T18:10:48Z' progress=1.000000 cache=58.4MiB(360942txo)
2022-01-08T18:13:58.051037Z UpdateTip: new best=00000000000000000003fa6d4ca08f0b3ebd3e447b337112d820105fc85bd74a height=717758 version=0x27ffe004 log2_work=93.275086 tx=701214324 date='2022-01-08T18:13:50Z' progress=1.000000 cache=58.4MiB(360647txo)
2022-01-08T18:15:05.166257Z New outbound peer connected: version: 70016, blocks=717758, peer=69137, peeraddr=x:8333 (block-relay-only)
2022-01-08T18:17:27.233906Z New outbound peer connected: version: 70015, blocks=717756, peer=69159, peeraddr=x:8333 (block-relay-only)
2022-01-08T18:19:18.327346Z New outbound peer connected: version: 70015, blocks=717758, peer=69179, peeraddr=x:8333 (block-relay-only)
2022-01-08T18:23:13.156116Z New outbound peer connected: version: 70015, blocks=717756, peer=69221, peeraddr=x:8333 (block-relay-only)
2022-01-08T18:24:53.589685Z New outbound peer connected: version: 70016, blocks=717758, peer=69236, peeraddr=x:8333 (block-relay-only)
2022-01-08T18:31:31.463866Z New outbound peer connected: version: 70015, blocks=717758, peer=69302, peeraddr=x:8333 (block-relay-only)
2022-01-08T18:41:56.223101Z tor: Thread interrupt
2022-01-08T18:41:56.223160Z addcon thread exit
2022-01-08T18:41:56.224073Z opencon thread exit
2022-01-08T18:41:56.224339Z torcontrol thread exit
2022-01-08T18:41:56.224938Z Shutdown: In progress...
2022-01-08T18:41:56.260124Z net thread exit
2022-01-08T18:41:56.310667Z msghand thread exit
2022-01-08T18:41:56.350784Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat started
2022-01-08T18:41:56.351804Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat completed (0.00s)
2022-01-08T18:41:56.353204Z scheduler thread exit
2022-01-08T18:41:56.407914Z Writing 0 unbroadcast transactions to disk.
2022-01-08T18:41:56.416664Z Dumped mempool: 0.009024s to copy, 0.042569s to dump
2022-01-08T18:41:56.422955Z FlushStateToDisk: write coins cache to disk (367217 coins, 62066kB) started
2022-01-08T18:41:56.821496Z FlushStateToDisk: write coins cache to disk (367217 coins, 62066kB) completed (0.40s)
2022-01-08T18:41:56.827039Z FlushStateToDisk: write coins cache to disk (0 coins, 14660kB) started
2022-01-08T18:41:56.828439Z FlushStateToDisk: write coins cache to disk (0 coins, 14660kB) completed (0.00s)
2022-01-08T18:41:56.976586Z [default wallet] Releasing wallet
2022-01-08T18:41:57.142466Z Shutdown: done

Shutdown was clean.

init message: Loading block index...
Opening LevelDB in /root/.bitcoin/blocks/index
: Error opening block database.
Do you want to rebuild the block database now?
: Error opening block database.
Do you want to rebuild the block database now?
Shutdown : In progress...
Flush(false)
DBFlush(false) ended               0ms
StopNode()
Flushed 0 addresses to peers.dat  2ms
Flush(true)
DBFlush(true) ended               0ms
Shutdown : done

Debug.log doesn't give any further detail.

Linux amd 5.10.0-9-amd64 [#1](/bitcoin-bitcoin/1/) SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux

model name	: AMD Ryzen 7 2700X Eight-Core Processor

Model Number:                       INTEL SSDPEKNW020T8
SMART overall-health self-assessment test result: PASSED

That's very strange.

There hasn't been forward progress on this for over a year. Appears to be a won't fix.