Blocks directory on macOS uses more disk space than expected #17827

It seems the Initial Block Download (IBD) on macOS Catalina uses a lot more disk space than expected, and will result in an unsuccesful download even when 480 GB of free disk space is available at the beginning.

Expected behavior

That the Initial Block Download finishes after using about 284 GB of disk space, as indicated by the Welcome window.

Actual behavior

When the download reaches 52% completion, over 284 GB of disk space has already been consumed. If the download is left to run, the machine will run out of disk space and Bitcoin Core will stop/quit before completing the download.

Looking a bit into the Bitcoin directory and what could be causing it, the files prefixed rev in the blocks folder are a lot larger than they are when compared to a previous blockchain downloaded on Linux.

Comparing the rev*.dat files with a previous successful IBD on Linux (below mounted with ext4fuse), the rev files use at most 23M, and the total is 32G:

blocks # pwd
/Users/erik/linux-ext4/bitcoin/.bitcoin/blocks

blocks # du -ch rev*.dat | sort -n
6.0M	rev01866.dat
 12M	rev00311.dat
 12M	rev00328.dat
 13M	rev00314.dat
 14M	rev00295.dat
 14M	rev00309.dat
 ...
 21M	rev01206.dat
 21M	rev01291.dat
 23M	rev01036.dat
 32G	total

On Mac, the rev files will be significantly larger than 23M. You can see that the files can get as large as 184M, and in this case where 22.8% of the blockchain had been downloaded, we were already using 48G for just the rev files, 107G for the whole data directory.

$ tail -n 1 /Users/erik/Library/Application\ Support/Bitcoin/debug.log
2019-12-29T07:30:29Z UpdateTip: new best=0000000000000000025930ce69bc9fcfae882de7231457f09534da493e114c52 height=399555 version=0x00000004 log2_work=84.162383 tx=111703322 date='2016-02-22T08:57:07Z' progress=0.228149 cache=466.9MiB(3406855txo)

$ du -ch /Users/erik/Library/Application\ Support/Bitcoin/blocks/rev*.dat | sort -n
 16M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00003.dat
 17M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00056.dat
 18M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00006.dat
 20M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00000.dat
 21M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00450.dat
 24M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00078.dat
 27M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00002.dat
 ...
171M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00232.dat
172M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00266.dat
172M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00334.dat
174M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00256.dat
175M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00143.dat
175M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00275.dat
177M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00132.dat
177M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00204.dat
178M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00337.dat
184M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00208.dat
 48G	total

$ du -ch /Users/erik/Library/Application\ Support/Bitcoin
 91M	/Users/erik/Library/Application Support/Bitcoin/blocks/index
105G	/Users/erik/Library/Application Support/Bitcoin/blocks
1.8G	/Users/erik/Library/Application Support/Bitcoin/chainstate
2.0M	/Users/erik/Library/Application Support/Bitcoin/wallets
107G	/Users/erik/Library/Application Support/Bitcoin
107G	total

Here you can see the Linux-Mac diff side by side:

To reproduce

• Download and install Bitcoin Core 0.19.0.1 on a Mac with a fresh install of macOS Catalina 10.15.2
• Start Bitcoin Core and select the default data directory and leave the prune checkbox unchecked (see picture at the start of the report)
• After downloading ~55% of the blockchain, check the disk space use of the ~/Library/Application Support/Bitcoin folder, it should now already exceed 284 GB
• If you had 480 GB of free space when starting the download, you should not be able to finish the download before running out of disk space

Possible workaround

Based on https://bitcoin.stackexchange.com/questions/11104/what-is-the-database-for/11108#11108 it seems the rev*.dat files can be removed to free up disk space. By deleting the earlier ones created I was able to free up enough space continue and complete the IBD on the Mac mini. However, doing so may lead to side effects that I am unaware of.

System information

bitcoin-0.19.0.1-osx.dmg downloaded from https://bitcoin.org/en/download

Machine 1:

• Mac mini (Late 2012)
• 2,5 GHz Dual-Core Intel Core i5
• 16 GB 1600 MHz DDR3
• macOS Catalina 10.15.2 (fresh install on a 500 GB SSD)

Machine 2:

• MacBook Pro (15-inch, 2018)
• 2,6 GHz 6-Core Intel Core i7
• 32 GB 2400 MHz DDR4
• macOS Catalina 10.15.2 (upgraded from Mojave)

I tried the workaround I suggested in the bug report, by removing the earlier of the rev.dat files to free up the required space to download all blocks, and then removing all rev.dat files forcing a reindex but the result is the same. On the Mac Mini I get to 83% when the disk is full. So achieving a full node on the Mac Mini with 480 GB free space to begin with seems impossible.

blocks % pwd

/Users/erik/Library/Application Support/Bitcoin/blocks

blocks % du -h .

 90M	./index
391G	.

% df -h

Filesystem          Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk1s5       466Gi   10Gi   52Gi    17%  484073 4881333367    0%   /
devfs              233Ki  233Ki    0Bi   100%     808          0  100%   /dev
/dev/disk1s1       466Gi  401Gi   52Gi    89%  142604 4881674836    0%   /System/Volumes/Data
/dev/disk1s4       466Gi  2.0Gi   52Gi     4%       2 4881817438    0%   /private/var/vm
map auto_home        0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home

blocks % du -ch rev*.dat | sort -n

 17M	rev01912.dat
 19M	rev00000.dat
 32M	rev00002.dat
 ...
 210M	rev01291.dat
 250M	rev01036.dat
152G	total

blocks % du -ch blk*.dat

128M	blk00000.dat
128M	blk00001.dat
128M	blk00002.dat
...
128M	blk01912.dat
125M	blk01913.dat
238G	total

blocks % rm rev*.dat

% df -h

Filesystem          Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk1s5       466Gi   10Gi  204Gi     5%  484073 4881333367    0%   /
devfs              233Ki  233Ki    0Bi   100%     808          0  100%   /dev
/dev/disk1s1       466Gi  248Gi  204Gi    55%  141579 4881675861    0%   /System/Volumes/Data
/dev/disk1s4       466Gi  2.0Gi  204Gi     1%       2 4881817438    0%   /private/var/vm
map auto_home        0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home

Started Bitcoin Core and clicked OK to rebuild the block database.

% tail debug.log

2019-12-31T04:32:26Z UpdateTip: new best=00000000000000000016319a033ba5a534d962bf17077825634a0bbcce5dbf48 height=573963 version=0x20000000 log2_work=90.59244 tx=408110896 date='2019-04-30T19:18:40Z' progress=0.832507 cache=726.0MiB(5526596txo)
2019-12-31T04:32:26Z *** Disk space is too low!
2019-12-31T04:32:26Z Error: Disk space is too low!
2019-12-31T04:52:10Z socket sending timeout: 1201s
2019-12-31T04:52:10Z socket sending timeout: 1201s
2019-12-31T04:52:10Z socket sending timeout: 1201s
2019-12-31T04:52:10Z socket sending timeout: 1201s
2019-12-31T04:52:10Z socket sending timeout: 1201s
2019-12-31T04:52:10Z socket sending timeout: 1201s
2019-12-31T04:52:10Z socket sending timeout: 1201s
2019-12-31T04:52:10Z socket sending timeout: 1201s
2019-12-31T04:52:10Z socket sending timeout: 1201s
2019-12-31T04:52:13Z socket sending timeout: 1201s
2019-12-31T06:02:11Z Failed to connect best block (Disk space is too low! (code 0))
2019-12-31T06:02:11Z tor: Thread interrupt
2019-12-31T06:02:11Z Shutdown: In progress...
2019-12-31T06:02:11Z torcontrol thread exit
2019-12-31T06:02:11Z addcon thread exit
2019-12-31T06:02:11Z opencon thread exit
2019-12-31T06:02:11Z msghand thread exit
2019-12-31T06:02:11Z net thread exit
2019-12-31T06:02:11Z scheduler thread interrupt
2019-12-31T06:02:11Z *** Disk space is too low!
2019-12-31T06:02:11Z Error: Disk space is too low!
2019-12-31T06:02:11Z ForceFlushStateToDisk: failed to flush state (Disk space is too low! (code 0))
2019-12-31T06:02:11Z *** Disk space is too low!
2019-12-31T06:02:11Z Error: Disk space is too low!
2019-12-31T06:02:11Z ForceFlushStateToDisk: failed to flush state (Disk space is too low! (code 0))
2019-12-31T06:02:12Z [default wallet] Releasing wallet
2019-12-31T06:02:12Z Shutdown: done

Once again the rev*.dat files uses 205G of disk space, the sizes range from 0B to 231M.

blocks % du -ch rev*.dat | sort -n

  0B	rev01623.dat
  0B	rev01624.dat
  0B	rev01625.dat
  0B	rev01626.dat
  0B	rev01627.dat
  0B	rev01628.dat
...
210M	rev00349.dat
210M	rev00350.dat
210M	rev00352.dat
210M	rev00353.dat
210M	rev00432.dat
229M	rev00337.dat
231M	rev00333.dat
231M	rev00336.dat
205G	total

% df -h
                     
Filesystem          Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk1s5       466Gi   10Gi  247Mi    98%  484073 4881333367    0%   /
devfs              233Ki  233Ki    0Bi   100%     808          0  100%   /dev
/dev/disk1s1       466Gi  452Gi  247Mi   100%  143019 4881674421    0%   /System/Volumes/Data
/dev/disk1s4       466Gi  2.0Gi  247Mi    90%       2 4881817438    0%   /private/var/vm
map auto_home        0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home

It says "at least". We can't accurately predict exactly how large the blockchain and associated data will be by the time synchronization finishes.

With the help of people on IRC #bitcoin we've managed to rule out some potential causes:

1. Time Machine is not configured on the Mac mini:

It says "at least". We can't accurately predict exactly how large the blockchain and associated data will be by the time synchronization finishes.

You're absolutely correct. However, I would argue that 480+ GB is way more than 284 GB. I actually don't know how much the whole download would require, as 480 GB only gets me to 83%.

My apologies, I missed that the difference was so large between two systems.

Your rev files are way larger than they should be. However you can't just delete them, as that will interfere with reorgs and a few other operations.

As an immediate solution, you can enable pruning (which will delete blocks and rev files that are sufficiently old automatically and safely), but that is only addressing the symptom.

My apologies, I missed that the difference was so large between two systems.

Your rev files are way larger than they should be. However you can't just delete them, as that will interfere with reorgs and a few other operations.

As an immediate solution, you can enable pruning (which will delete blocks and rev files that are sufficiently old automatically and safely), but that is only addressing the symptom.

Thanks for clarifying why deleting them is not a working solution (or even workaround as I concluded in my later tests). In my case, as I understand it, I cannot enable pruning since I will be running Lightning towards this full node.

Furthermore, if this is the intended behaviour on macOS, then perhaps it would be a good idea to revise the minimum requirements on https://bitcoin.org/en/full-node#special-cases to make sure that Mac users avoid running into this problem.

No, this is not intended behavior at all. The rev*.dat files shouldn't go above 15-20 MB usually.

FWIW, some lightning implementations (e.g. c-lightning) work fine with a pruned bitcoind.

No, this is not intended behavior at all. The rev*.dat files shouldn't go above 15-20 MB usually.

FWIW, some lightning implementations (e.g. c-lightning) work fine with a pruned bitcoind.

Thanks for the hint, I have only tried the lnd implementation so it could also be interesting to try that out!

Digging a bit more into it with the help of #bitcoin we see that the files are mostly empty. Copying the file will result in a file size in the expected range.

blocks % du -h rev00336.dat 
231M	rev00336.dat

blocks % hexdump -C rev00336.dat | tail -4
0145c010  ea 7b e8 e9 86 7e c0 bc  fb 13 00 00 00 00 00 00  |.{...~..........|
0145c020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01500000

% grep "rev00336.dat" debug.log          
2019-12-30T20:48:10Z Pre-allocating up to position 0x100000 in rev00336.dat
2019-12-30T20:48:39Z Pre-allocating up to position 0x200000 in rev00336.dat
2019-12-30T20:48:43Z Pre-allocating up to position 0x300000 in rev00336.dat
2019-12-30T20:48:45Z Pre-allocating up to position 0x400000 in rev00336.dat
2019-12-30T20:48:49Z Pre-allocating up to position 0x500000 in rev00336.dat
2019-12-30T20:48:50Z Pre-allocating up to position 0x600000 in rev00336.dat
2019-12-30T20:48:51Z Pre-allocating up to position 0x700000 in rev00336.dat
2019-12-30T20:48:52Z Pre-allocating up to position 0x800000 in rev00336.dat
2019-12-30T20:48:54Z Pre-allocating up to position 0x900000 in rev00336.dat
2019-12-30T20:48:55Z Pre-allocating up to position 0xa00000 in rev00336.dat
2019-12-30T20:48:57Z Pre-allocating up to position 0xb00000 in rev00336.dat
2019-12-30T20:48:58Z Pre-allocating up to position 0xc00000 in rev00336.dat
2019-12-30T20:49:00Z Pre-allocating up to position 0xd00000 in rev00336.dat
2019-12-30T20:49:01Z Pre-allocating up to position 0xe00000 in rev00336.dat
2019-12-30T20:49:02Z Pre-allocating up to position 0xf00000 in rev00336.dat
2019-12-30T20:49:03Z Pre-allocating up to position 0x1000000 in rev00336.dat
2019-12-30T20:49:05Z Pre-allocating up to position 0x1100000 in rev00336.dat
2019-12-30T20:49:06Z Pre-allocating up to position 0x1200000 in rev00336.dat
2019-12-30T20:49:08Z Pre-allocating up to position 0x1300000 in rev00336.dat
2019-12-30T20:49:10Z Pre-allocating up to position 0x1400000 in rev00336.dat
2019-12-30T20:49:12Z Pre-allocating up to position 0x1500000 in rev00336.dat

blocks % ls -lh rev00336.dat 
-rw-------  1 erik  staff    21M Dec 30 22:49 rev00336.dat

blocks % cp rev00336.dat rev00336.copy

blocks % du -h rev00336.*
 21M	rev00336.copy
231M	rev00336.dat

So a workaround might be to replace all the rev files with copies.

No, this is not intended behavior at all. The rev*.dat files shouldn't go above 15-20 MB usually.

I currently have a fully synced node running on macOS (10.14) that uses nearly 500GB on disk:

475G	total
422G	blocks
 40G	indexes
8.4G	testnet3
4.1G	chainstate
356M	regtest
9.8M	debug.log
4.0M	wallet.dat
4.0M	peers.dat
4.0M	mempool.dat
896K	some_wallet
256K	fee_estimates.dat
4.0K	bitcoin.conf
4.0K	banlist.dat
  0B	db.log
  0B	.walletlock
  0B	.lock

Largest rev*.dat files:

206M	rev00352.dat
193M	rev00331.dat
190M	rev00461.dat
190M	rev00447.dat
190M	rev00390.dat
190M	rev00389.dat
190M	rev00386.dat
190M	rev00374.dat

No, this is not intended behavior at all. The rev*.dat files shouldn't go above 15-20 MB usually.

I currently have a fully synced node running on macOS (10.14) that uses nearly 500GB on disk:

475G	total
422G	blocks
 40G	indexes
8.4G	testnet3
4.1G	chainstate
356M	regtest
9.8M	debug.log
4.0M	wallet.dat
4.0M	peers.dat
4.0M	mempool.dat
896K	some_wallet
256K	fee_estimates.dat
4.0K	bitcoin.conf
4.0K	banlist.dat
  0B	db.log
  0B	.walletlock
  0B	.lock

Largest rev*.dat files:

206M	rev00352.dat
193M	rev00331.dat
190M	rev00461.dat
190M	rev00447.dat
190M	rev00390.dat
190M	rev00389.dat
190M	rev00386.dat
190M	rev00374.dat

And if you copy one of the larger files, will it reduce in size to the expected 15-20 MB range?

What is the file system used on the drive that keeps blocks folder? Also, can you post your bitcoin.conf file?

The code that does the OS-specific file allocation hasn't been touched in 7 years (#2229), so I wonder what suddenly causes this.

What is the file system used on the drive that keeps blocks folder? Also, can you post your bitcoin.conf file?

Both machines use APFS.

I have no bitcoin.conf file in use, so only defaults should be used. This is a clean install using the dmg, no custom settings have been done.

The code that does the OS-specific file allocation hasn't been touched in 7 years (#2229), so I wonder what suddenly causes this.

It may not be so sudden, considering @fanquake 's 10.14 (macOS Mojave) machine that shows the same behavior.

I am struggling to reproduce this issue. I am now around 43% downloading blocks progress and upon investigating the blocks folder, I see all my rev* files are between 17-22MB and blk* files are around 134MB each, so I guess all looks good.

Tested on mac mini (2018) with macos 10.15.2 (19C57), 16GB RAM, i5. The drive selected for data is 500GB encrypted APFS partition on sata3 ssd connected via usb-c adapter. I haven't customized anything except unchecking pruning option and selecting the external drive as a data folder.

What about your wallet file, is it just created from scratch or did you put an existing, old wallet into the bitcoin folder?

I am struggling to reproduce this issue. I am now around 43% downloading blocks progress and upon investigating the blocks folder, I see all my rev* files are between 17-22MB and blk* files are around 134MB each, so I guess all looks good.

Tested on mac mini (2018) with macos 10.15.2 (19C57), 16GB RAM, i5. The drive selected for data is 500GB encrypted APFS partition on sata3 ssd connected via usb-c adapter. I haven't customized anything except unchecking pruning option and selecting the external drive as a data folder.

What about your wallet file, is it just created from scratch or did you put an existing, old wallet into the bitcoin folder?

Interesting! Everything is completely from scratch, even the wallet. On the second machine my APFS is encrypted, and it's also showing the same issue, the largest rev files is 184M.

$ du -ch rev*.dat | sort -n
 16M	rev00003.dat
 17M	rev00056.dat
 18M	rev00006.dat
 20M	rev00000.dat
...
174M	rev00256.dat
175M	rev00143.dat
175M	rev00275.dat
177M	rev00132.dat
177M	rev00204.dat
178M	rev00337.dat
184M	rev00208.dat

I've located a spare 500 GB external SSD so I will try to reproduce with an external drive to see what happens.

As a workaround based on a tip from the IRC discussions, I did a small bash script to slim down the rev files one by one, by copying them once, allowing me to continue from 83% and finish the download on the mac mini. These are the steps I took, below reproduced on my second machine that was on 23% when I stopped downloading, hence the total size is lower, but you can see there is already an extra 40G of space being consumed:

$ cd Library/Application\ Support/Bitcoin/blocks
$ du -h
 91M	./index
105G	.

$ cat slim.sh 
for file in ./rev*.dat
do
    if [[ -f $file ]]; then
	size_before=$(du -h $file | cut -f1)
        cp $file $file.tmp
        rm $file
	mv $file.tmp $file
	size_after=$(du -h $file | cut -f1)
	echo $file $size_before " -> " $size_after
    fi
done

$ bash slim.sh 
./rev00000.dat 20M  ->  19M
./rev00001.dat 33M  ->  17M
./rev00002.dat 27M  ->  16M
./rev00003.dat 16M  ->  16M
...
./rev00443.dat 108M  ->  19M
./rev00444.dat 114M  ->  19M
./rev00445.dat 52M  ->  19M
./rev00446.dat 77M  ->  19M
./rev00447.dat 54M  ->  19M
./rev00448.dat 62M  ->  19M
./rev00449.dat 79M  ->  17M
./rev00450.dat 21M  ->  6.0M

$ du -h
 91M	./index
 64G	.

I kinda need to retract what I said before. Today I woke up to a bitcoin core message "Error: Disk space is too low!". This was at 91.20% of IBD. I checked the drive and About this Mac, Storage says only 80MB is left on the 500GB dedicated partition. Du and df commands and disk utility report the whole partition space used. Finder also reports whole disk space used when checking the whole drive but when checking the bitcoin folder only it says it takes only 273GB (there is really nothing else on this partition). Upon investigating further, I can see that Get Info option on a folder returns: 273 165 546 724 bytes (499,43 GB on disk) for 5 369 items similarly, all rev files in the blocks folder are visible in Finder as around 19MB files but when using Get info option the true size is revealed, e.g. 19 922 944 bytes (199,3 MB on disk) Same goes for ncdu, it reports true size of files on the disk and it seems that a lot of rev files actually take much more space than they should (e.g. 200MB).

I checked the debug.log and noticed that the rev files that are the biggest have the most Pre-allocating up to position, messages, and the smallest ones have the least:

• rev01037, 350.2 MiB, pre-allocating goes up to 0x1a00000
• rev01767, 8.6MiB, pre-allocating goes up to 0x500000 only

FYI, the Time Machine is disabled for this drive, later I disabled it completely, restarted the machine, this did not free up any space.

I guess that some dev needs to investigate closer how this pre-allocating mechanism behaves on APFS drives. Perhaps it reserves more space than it needs and later on it does not release it properly?

I kinda need to retract what I said before. Today I woke up to a bitcoin core message "Error: Disk space is too low!". This was at 91.20% of IBD. I checked the drive and About this Mac, Storage says only 80MB is left on the 500GB dedicated partition. Du and df commands and disk utility report the whole partition space used. Finder also reports whole disk space used when checking the whole drive but when checking the bitcoin folder only it says it takes only 273GB (there is really nothing else on this partition). Upon investigating further, I can see that Get Info option on a folder returns: 273 165 546 724 bytes (499,43 GB on disk) for 5 369 items similarly, all rev files in the blocks folder are visible in Finder as around 19MB files but when using Get info option the true size is revealed, e.g. 19 922 944 bytes (199,3 MB on disk) Same goes for ncdu, it reports true size of files on the disk and it seems that a lot of rev files actually take much more space than they should (e.g. 200MB).

I checked the debug.log and noticed that the rev files that are the biggest have the most Pre-allocating up to position, messages, and the smallest ones have the least:

* rev01037, 350.2 MiB, pre-allocating goes up to 0x1a00000

* rev01767, 8.6MiB, pre-allocating goes up to 0x500000 only

FYI, the Time Machine is disabled for this drive, later I disabled it completely, restarted the machine, this did not free up any space.

I guess that some dev needs to investigate closer how this pre-allocating mechanism behaves on APFS drives. Perhaps it reserves more space than it needs and later on it does not release it properly?

Thanks for letting me know. Then I won't spend time on testing with an external drive right now, but I'll be prepared to do it later if needed.

To further pinpoint the problem I created two disk images using Disk Utility, one with Mac OS Extended (Journaled) and the other with APFS. I then symlinked the volumes to the default data directory:

ln -s /Volumes/MacOSExtended /Users/erik/Library/Application Support/Bitcoin

After that I started bitcoind to download the first five blocks and stopped. Then I symlinked the other:

rm /Users/erik/Library/Application Support/Bitcoin
ln -s /Volumes/APFS /Users/erik/Library/Application Support/Bitcoin

and ran bitcoind again to sync the first five blocks.

The problem only seems to exist on APFS:

$ cd /Volumes/APFS/blocks/
blocks erik$ du -ch *.dat
128M	blk00000.dat
128M	blk00001.dat
128M	blk00002.dat
128M	blk00003.dat
128M	blk00004.dat
 19M	rev00000.dat
 34M	rev00001.dat
 31M	rev00002.dat
 32M	rev00003.dat
 49M	rev00004.dat
805M	total
blocks erik$ ls -lh *.dat
-rw-------  1 erik  staff   128M  5 Tam 00:25 blk00000.dat
-rw-------  1 erik  staff   128M  5 Tam 00:26 blk00001.dat
-rw-------  1 erik  staff   128M  5 Tam 00:27 blk00002.dat
-rw-------  1 erik  staff   128M  5 Tam 00:28 blk00003.dat
-rw-------  1 erik  staff   128M  5 Tam 00:30 blk00004.dat
-rw-------  1 erik  staff    19M  5 Tam 00:25 rev00000.dat
-rw-------  1 erik  staff    17M  5 Tam 00:26 rev00001.dat
-rw-------  1 erik  staff    16M  5 Tam 00:28 rev00002.dat
-rw-------  1 erik  staff    16M  5 Tam 00:29 rev00003.dat
-rw-------  1 erik  staff    17M  5 Tam 00:30 rev00004.dat

Mac OS Extended (HFS+):

$ cd /Volumes/MacOSExtended/blocks/
blocks erik$ du -ch *.dat
128M	blk00000.dat
128M	blk00001.dat
128M	blk00002.dat
128M	blk00003.dat
128M	blk00004.dat
 19M	rev00000.dat
 17M	rev00001.dat
 16M	rev00002.dat
 16M	rev00003.dat
 17M	rev00004.dat
724M	total
blocks erik$ ls -lh *.dat
-rw-------  1 erik  staff   128M  5 Tam 00:02 blk00000.dat
-rw-------  1 erik  staff   128M  5 Tam 00:03 blk00001.dat
-rw-------  1 erik  staff   128M  5 Tam 00:05 blk00002.dat
-rw-------  1 erik  staff   128M  5 Tam 00:06 blk00003.dat
-rw-------  1 erik  staff   128M  5 Tam 00:07 blk00004.dat
-rw-------  1 erik  staff    19M  5 Tam 00:02 rev00000.dat
-rw-------  1 erik  staff    17M  5 Tam 00:04 rev00001.dat
-rw-------  1 erik  staff    16M  5 Tam 00:05 rev00002.dat
-rw-------  1 erik  staff    16M  5 Tam 00:06 rev00003.dat
-rw-------  1 erik  staff    17M  5 Tam 00:07 rev00004.dat

The diff:

What are the exact file sizes you're seeing for the rev*.dat files? Are they all multiples of some power of two? That could give a hint about where the problem is.

Checking the file content

The file rev00337.dat has a reported size by du of 212MB before copying and 21MB after. However, the content of the files is exactly the same.

cd ~/Library/Application\ Support/Bitcoin/blocks

du -ch ./rev00337.dat
 212M	./rev00337.dat
 212M	total

hexdump -C rev00337.dat | tail -4
0141a770  e0 92 b0 c3 00 00 00 00  00 00 00 00 00 00 00 00  |................|
0141a780  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01500000

cp rev00337.dat ~/Desktop

cd ~/Desktop

hexdump -C rev00337.dat | tail -4
0141a770  e0 92 b0 c3 00 00 00 00  00 00 00 00 00 00 00 00  |................|
0141a780  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01500000

Checking that the files content is the same after copy

I have modified the previously posted script to show the files hashes before and after copy.

for file in ./rev*.dat
do
    if [[ -f $file ]]; then
	size_before=$(du -h $file | cut -f1)
	hash_before=$(shasum -a 256 $file | cut -d" " -f1)
        cp $file $file.tmp
        rm $file
	mv $file.tmp $file
	size_after=$(du -h $file | cut -f1)
        hash_after=$(shasum -a 256 $file | cut -d" " -f1)
	echo $file $size_before " -> " $size_after " hash_before: " $hash_before "hash_after: " $hash_after
    fi
done

Here is a sample output:

./rev01590.dat 3.9M  ->  4.0M  hash_before:  9fa0f20cfc1d9593b873399130cfce323aa574473f3a6b7caa58a970e41dc45b hash_after:  9fa0f20cfc1d9593b873399130cfce323aa574473f3a6b7caa58a970e41dc45b
./rev01591.dat 844K  ->  1.0M  hash_before:  ab8811a65806921492c63c85c3d05643a0a7cba9fb5b3a677cb16fabb1167918 hash_after:  ab8811a65806921492c63c85c3d05643a0a7cba9fb5b3a677cb16fabb1167918

All the hashes are the same before and after copying the files.

Difference in file size reporting

The file size reported by du is different from the one reported by ls and stat.

cd ~/Library/Application\ Support/Bitcoin/blocks

du -ch ./rev01668.dat
 90M	./rev01668.dat
 90M	total

ls -lah | grep rev01668.dat 
-rw-------     1 something  something    17M Jan  6 11:18 rev01668.dat

stat -x rev01668.dat          
  File: "rev01668.dat"
  Size: 17825792     FileType: Regular File
  Mode: (0600/-rw-------)         Uid: (  502/  something)  Gid: (   20/   something)
Device: 1,4   Inode: 65189970    Links: 1
Access: Mon Jan  6 11:53:20 2020
Modify: Mon Jan  6 11:18:58 2020
Change: Mon Jan  6 11:18:58 2020

Diagnosis

du is reporting the size in used blocks, ls the size of the files content.

diskutil info / | grep "Block Size"
   Device Block Size:         4096 Bytes
   Allocation Block Size:     4096 Bytes

cd ~/Library/Application\ Support/Bitcoin/blocks

du -ch ./rev01668.dat
 90M	./rev01668.dat
 90M	total

BLOCKSIZE=4096 du -c ./rev01668.dat
22916	./rev01668.dat
22916	total

BLOCKSIZE=4096 ls -lahs | grep rev01668.dat
 22916 -rw-------     1 something  something    17M Jan  6 11:18 rev01668.dat

cp rev01668.dat ~/Desktop

cd ~/Desktop

du -ch ./rev01668.dat
 17M	./rev01668.dat
 17M	total

BLOCKSIZE=4096 du -c ./rev01668.dat
4352	./rev01668.dat
4352	total

BLOCKSIZE=4096 ls -lahs | grep rev01668.dat
  4352 -rw-------   1 something  something    17M Jan  6 12:08 rev01668.dat

It can be seen that, after copying the file, the number of blocks used is reduced drastically and the file size is as expected.

Hypothesis and troubleshooting

ftruncate function is not working as expected when used in an APFS formated volume.

I have tried to reproduce the problem in an isolated environment wirhout any success. The files sizes were always correct.

Possible fix

If the function AllocateFileRange is modified to use the fallback method in macOS the file sizes are as expected and the problem dissapear.

Possible fix

If the function AllocateFileRange is modified to use the fallback method in macOS the file sizes are as expected and the problem dissapear.

I can confirm that I've tried this fix and using the fallback works. I also lean towards something with the ftruncate being broken, but wouldn't you also see the same problem in the blk*.dat files then?

What are the exact file sizes you're seeing for the rev*.dat files? Are they all multiples of some power of two? That could give a hint about where the problem is.

I cannot see any indication that they would be multiples of power of two, unless I'm missing something.

@sipa

What are the exact file sizes you're seeing for the rev*.dat files? Are they all multiples of some power of two? That could give a hint about where the problem is.

This is not how it ends up, actually. Every pre-allocation will append "expected size + 1 MB" to the actual size for each allocation time. I.e.

(once)
a = 1024*1024
b = a
(for each increase)
a = a + 1024*1024; b = b+a; b/1024/1024

This gives the sizes 6, 10, 15, 21, 28, 36, 45, 55, 66, 78, 91, 105, 120, 136, 153, 171, ... which exactly matches the sizes I see.

As a workaround based on a tip from the IRC discussions, I did a small bash script to slim down the rev files one by one, by copying them once, allowing me to continue from 83% and finish the download on the mac mini. These are the steps I took, below reproduced on my second machine that was on 23% when I stopped downloading, hence the total size is lower, but you can see there is already an extra 40G of space being consumed:

Thanks for the workaround of copying them out. I was able to reduce disk usage by 237G. My before & after:

$ du -h
 92M	./index
538G	.

$ du -h
 92M	./index
301G