Fixes #24400.
See #24400 (comment) to reproduce the failure.
As MarcoFalke suggested in #24400 (comment), it can happen that the wallet is not fully flushed before being copied over to the other node. Fixed by unloading the wallet before copying the file.