Huge disk fragmentation #29296

Is there an existing issue for this?

• I have searched the existing issues

Current behaviour

very many fragmented files

Expected behaviour

files are not fragmented

Steps to reproduce

An extra description of the behaviour from #26063 Prerequisite: disc defragmented (I used Full optimization from UltraDefrag), Windows Security(including real-time protection) are disabled.

1. Start Bitcoin Core
2. Wait for half an hour
3. Stop Bitcoin Core
4. Run UltraDefrag report

Relevant log output

fragmentation_and_speed_degradation.zip

How did you obtain Bitcoin Core

Pre-built binaries

What version of Bitcoin Core are you using?

26.0

Operating system and version

Windows 10 22H2

Machine specifications

HDD SATA III (1 TB, 64 MB cache, NTFS) Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz 2.30 GHz 16 GB RAM

Disk fragmentation is a property of your hardware and your operating system. Software such as Bitcoin Core do not have low level access to the filesystem and hardware to control where the data will be physically placed. This is a problem with your operating system and how it chooses where files are actually physically stored.

In my test environment fragmentation is only caused by Bitcoin Core, and affects Bitcoin Core. I think to eliminate fragmentation, all you need to do is increase buffers(e.g. to 2MB for .ldb files), limit simultaneous writes for different files, or preallocate disc space. I don't see your answer as a solution, as it boils down to: 1)buying an SSD(which imposes limitations, and doesn't completely solve the problem) 2)using defragmentation(requires time, user involvement and restarting Bitcoin Core) 3)using non-NTFS(requires reformatting the volume, effectively prohibiting Windows) I think fixing the issue would greatly improve the UX of IBD. Please consider it specifically as a user experience improvement.

According to your debug log 5 blocks are advanced per second, which seems fast, even for spinning storage. Not sure if much can be done to improve upon that. In any case, if you have a software solution, no one is holding you back from testing and benchmarking it. If it does really does improve performance significantly, it can be considered.