Blocks directory on macOS uses more disk space than expected #17827

issue eriknylund openend this issue on December 29, 2019
  1. eriknylund commented at 8:37 pm on December 29, 2019: contributor

    It seems the Initial Block Download (IBD) on macOS Catalina uses a lot more disk space than expected, and will result in an unsuccesful download even when 480 GB of free disk space is available at the beginning.

    Expected behavior

    That the Initial Block Download finishes after using about 284 GB of disk space, as indicated by the Welcome window.

    ibd-480-GB-available-at-least-284-GB-required

    Actual behavior

    When the download reaches 52% completion, over 284 GB of disk space has already been consumed. If the download is left to run, the machine will run out of disk space and Bitcoin Core will stop/quit before completing the download.

    Looking a bit into the Bitcoin directory and what could be causing it, the files prefixed rev in the blocks folder are a lot larger than they are when compared to a previous blockchain downloaded on Linux.

    Comparing the rev*.dat files with a previous successful IBD on Linux (below mounted with ext4fuse), the rev files use at most 23M, and the total is 32G:

     0blocks # pwd
 1/Users/erik/linux-ext4/bitcoin/.bitcoin/blocks
 2
 3blocks # du -ch rev*.dat | sort -n
 46.0M	rev01866.dat
 5 12M	rev00311.dat
 6 12M	rev00328.dat
 7 13M	rev00314.dat
 8 14M	rev00295.dat
 9 14M	rev00309.dat
10 ...
11 21M	rev01206.dat
12 21M	rev01291.dat
13 23M	rev01036.dat
14 32G	total

    On Mac, the rev files will be significantly larger than 23M. You can see that the files can get as large as 184M, and in this case where 22.8% of the blockchain had been downloaded, we were already using 48G for just the rev files, 107G for the whole data directory.

     0$ tail -n 1 /Users/erik/Library/Application\ Support/Bitcoin/debug.log
 12019-12-29T07:30:29Z UpdateTip: new best=0000000000000000025930ce69bc9fcfae882de7231457f09534da493e114c52 height=399555 version=0x00000004 log2_work=84.162383 tx=111703322 date='2016-02-22T08:57:07Z' progress=0.228149 cache=466.9MiB(3406855txo)
 2
 3$ du -ch /Users/erik/Library/Application\ Support/Bitcoin/blocks/rev*.dat | sort -n
 4 16M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00003.dat
 5 17M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00056.dat
 6 18M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00006.dat
 7 20M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00000.dat
 8 21M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00450.dat
 9 24M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00078.dat
10 27M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00002.dat
11 ...
12171M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00232.dat
13172M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00266.dat
14172M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00334.dat
15174M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00256.dat
16175M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00143.dat
17175M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00275.dat
18177M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00132.dat
19177M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00204.dat
20178M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00337.dat
21184M	/Users/erik/Library/Application Support/Bitcoin/blocks/rev00208.dat
22 48G	total
23
24$ du -ch /Users/erik/Library/Application\ Support/Bitcoin
25 91M	/Users/erik/Library/Application Support/Bitcoin/blocks/index
26105G	/Users/erik/Library/Application Support/Bitcoin/blocks
271.8G	/Users/erik/Library/Application Support/Bitcoin/chainstate
282.0M	/Users/erik/Library/Application Support/Bitcoin/wallets
29107G	/Users/erik/Library/Application Support/Bitcoin
30107G	total

    Here you can see the Linux-Mac diff side by side: linux-vs-mac

    To reproduce

    • Download and install Bitcoin Core 0.19.0.1 on a Mac with a fresh install of macOS Catalina 10.15.2
    • Start Bitcoin Core and select the default data directory and leave the prune checkbox unchecked (see picture at the start of the report)
    • After downloading ~55% of the blockchain, check the disk space use of the ~/Library/Application Support/Bitcoin folder, it should now already exceed 284 GB
    • If you had 480 GB of free space when starting the download, you should not be able to finish the download before running out of disk space

    Possible workaround

    Based on https://bitcoin.stackexchange.com/questions/11104/what-is-the-database-for/11108#11108 it seems the rev*.dat files can be removed to free up disk space. By deleting the earlier ones created I was able to free up enough space continue and complete the IBD on the Mac mini. However, doing so may lead to side effects that I am unaware of.

    System information

    bitcoin-0.19.0.1-osx.dmg downloaded from https://bitcoin.org/en/download

    Machine 1:

    • Mac mini (Late 2012)
    • 2,5 GHz Dual-Core Intel Core i5
    • 16 GB 1600 MHz DDR3
    • macOS Catalina 10.15.2 (fresh install on a 500 GB SSD)

    Machine 2:

    • MacBook Pro (15-inch, 2018)
    • 2,6 GHz 6-Core Intel Core i7
    • 32 GB 2400 MHz DDR4
    • macOS Catalina 10.15.2 (upgraded from Mojave)
  2. eriknylund added the label Bug on Dec 29, 2019
  3. fanquake added the label macOS on Dec 29, 2019
  4. eriknylund commented at 6:23 am on December 31, 2019: contributor

    I tried the workaround I suggested in the bug report, by removing the earlier of the rev*.dat files to free up the required space to download all blocks, and then removing all rev*.dat files forcing a reindex but the result is the same. On the Mac Mini I get to 83% when the disk is full. So achieving a full node on the Mac Mini with 480 GB free space to begin with seems impossible.

     0blocks % pwd
 1
 2/Users/erik/Library/Application Support/Bitcoin/blocks
 3
 4blocks % du -h .
 5
 6 90M	./index
 7391G	.
 8
 9% df -h
10
11Filesystem          Size   Used  Avail Capacity iused      ifree %iused  Mounted on
12/dev/disk1s5       466Gi   10Gi   52Gi    17%  484073 4881333367    0%   /
13devfs              233Ki  233Ki    0Bi   100%     808          0  100%   /dev
14/dev/disk1s1       466Gi  401Gi   52Gi    89%  142604 4881674836    0%   /System/Volumes/Data
15/dev/disk1s4       466Gi  2.0Gi   52Gi     4%       2 4881817438    0%   /private/var/vm
16map auto_home        0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home
17
18blocks % du -ch rev*.dat | sort -n
19
20 17M	rev01912.dat
21 19M	rev00000.dat
22 32M	rev00002.dat
23 ...
24 210M	rev01291.dat
25 250M	rev01036.dat
26152G	total
27
28blocks % du -ch blk*.dat
29
30128M	blk00000.dat
31128M	blk00001.dat
32128M	blk00002.dat
33...
34128M	blk01912.dat
35125M	blk01913.dat
36238G	total
37
38blocks % rm rev*.dat
39
40% df -h
41
42Filesystem          Size   Used  Avail Capacity iused      ifree %iused  Mounted on
43/dev/disk1s5       466Gi   10Gi  204Gi     5%  484073 4881333367    0%   /
44devfs              233Ki  233Ki    0Bi   100%     808          0  100%   /dev
45/dev/disk1s1       466Gi  248Gi  204Gi    55%  141579 4881675861    0%   /System/Volumes/Data
46/dev/disk1s4       466Gi  2.0Gi  204Gi     1%       2 4881817438    0%   /private/var/vm
47map auto_home        0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home

    Started Bitcoin Core and clicked OK to rebuild the block database.

    corrupted-block-database-detected

    reindexing-blocks-on-disk

    processing-blocks-on-disk

     0% tail debug.log
 1
 22019-12-31T04:32:26Z UpdateTip: new best=00000000000000000016319a033ba5a534d962bf17077825634a0bbcce5dbf48 height=573963 version=0x20000000 log2_work=90.59244 tx=408110896 date='2019-04-30T19:18:40Z' progress=0.832507 cache=726.0MiB(5526596txo)
 32019-12-31T04:32:26Z *** Disk space is too low!
 42019-12-31T04:32:26Z Error: Disk space is too low!
 52019-12-31T04:52:10Z socket sending timeout: 1201s
 62019-12-31T04:52:10Z socket sending timeout: 1201s
 72019-12-31T04:52:10Z socket sending timeout: 1201s
 82019-12-31T04:52:10Z socket sending timeout: 1201s
 92019-12-31T04:52:10Z socket sending timeout: 1201s
102019-12-31T04:52:10Z socket sending timeout: 1201s
112019-12-31T04:52:10Z socket sending timeout: 1201s
122019-12-31T04:52:10Z socket sending timeout: 1201s
132019-12-31T04:52:10Z socket sending timeout: 1201s
142019-12-31T04:52:13Z socket sending timeout: 1201s
152019-12-31T06:02:11Z Failed to connect best block (Disk space is too low! (code 0))
162019-12-31T06:02:11Z tor: Thread interrupt
172019-12-31T06:02:11Z Shutdown: In progress...
182019-12-31T06:02:11Z torcontrol thread exit
192019-12-31T06:02:11Z addcon thread exit
202019-12-31T06:02:11Z opencon thread exit
212019-12-31T06:02:11Z msghand thread exit
222019-12-31T06:02:11Z net thread exit
232019-12-31T06:02:11Z scheduler thread interrupt
242019-12-31T06:02:11Z *** Disk space is too low!
252019-12-31T06:02:11Z Error: Disk space is too low!
262019-12-31T06:02:11Z ForceFlushStateToDisk: failed to flush state (Disk space is too low! (code 0))
272019-12-31T06:02:11Z *** Disk space is too low!
282019-12-31T06:02:11Z Error: Disk space is too low!
292019-12-31T06:02:11Z ForceFlushStateToDisk: failed to flush state (Disk space is too low! (code 0))
302019-12-31T06:02:12Z [default wallet] Releasing wallet
312019-12-31T06:02:12Z Shutdown: done

    Once again the rev*.dat files uses 205G of disk space, the sizes range from 0B to 231M.

     0blocks % du -ch rev*.dat | sort -n
 1
 2  0B	rev01623.dat
 3  0B	rev01624.dat
 4  0B	rev01625.dat
 5  0B	rev01626.dat
 6  0B	rev01627.dat
 7  0B	rev01628.dat
 8...
 9210M	rev00349.dat
10210M	rev00350.dat
11210M	rev00352.dat
12210M	rev00353.dat
13210M	rev00432.dat
14229M	rev00337.dat
15231M	rev00333.dat
16231M	rev00336.dat
17205G	total
18
19% df -h
20                     
21Filesystem          Size   Used  Avail Capacity iused      ifree %iused  Mounted on
22/dev/disk1s5       466Gi   10Gi  247Mi    98%  484073 4881333367    0%   /
23devfs              233Ki  233Ki    0Bi   100%     808          0  100%   /dev
24/dev/disk1s1       466Gi  452Gi  247Mi   100%  143019 4881674421    0%   /System/Volumes/Data
25/dev/disk1s4       466Gi  2.0Gi  247Mi    90%       2 4881817438    0%   /private/var/vm
26map auto_home        0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home
  5. sipa commented at 7:33 am on December 31, 2019: member
    It says “at least”. We can’t accurately predict exactly how large the blockchain and associated data will be by the time synchronization finishes.
  6. eriknylund commented at 7:44 am on December 31, 2019: contributor

    With the help of people on IRC #bitcoin we’ve managed to rule out some potential causes:

    1. Time Machine is not configured on the Mac mini: no-time-machine-on-mac-mini
  7. eriknylund commented at 7:47 am on December 31, 2019: contributor

    It says “at least”. We can’t accurately predict exactly how large the blockchain and associated data will be by the time synchronization finishes.

    You’re absolutely correct. However, I would argue that 480+ GB is way more than 284 GB. I actually don’t know how much the whole download would require, as 480 GB only gets me to 83%.

  8. sipa commented at 7:55 am on December 31, 2019: member

    My apologies, I missed that the difference was so large between two systems.

    Your rev files are way larger than they should be. However you can’t just delete them, as that will interfere with reorgs and a few other operations.

    As an immediate solution, you can enable pruning (which will delete blocks and rev files that are sufficiently old automatically and safely), but that is only addressing the symptom.

  9. eriknylund commented at 8:08 am on December 31, 2019: contributor

    My apologies, I missed that the difference was so large between two systems.

    Your rev files are way larger than they should be. However you can’t just delete them, as that will interfere with reorgs and a few other operations.

    As an immediate solution, you can enable pruning (which will delete blocks and rev files that are sufficiently old automatically and safely), but that is only addressing the symptom.

    Thanks for clarifying why deleting them is not a working solution (or even workaround as I concluded in my later tests). In my case, as I understand it, I cannot enable pruning since I will be running Lightning towards this full node.

    Furthermore, if this is the intended behaviour on macOS, then perhaps it would be a good idea to revise the minimum requirements on https://bitcoin.org/en/full-node#special-cases to make sure that Mac users avoid running into this problem.

  10. sipa commented at 8:57 am on December 31, 2019: member

    No, this is not intended behavior at all. The rev*.dat files shouldn’t go above 15-20 MB usually.

    FWIW, some lightning implementations (e.g. c-lightning) work fine with a pruned bitcoind.

  11. eriknylund commented at 9:10 am on December 31, 2019: contributor

    No, this is not intended behavior at all. The rev*.dat files shouldn’t go above 15-20 MB usually.

    FWIW, some lightning implementations (e.g. c-lightning) work fine with a pruned bitcoind.

    Thanks for the hint, I have only tried the lnd implementation so it could also be interesting to try that out!

  12. eriknylund commented at 11:34 am on December 31, 2019: contributor

    Digging a bit more into it with the help of #bitcoin we see that the files are mostly empty. Copying the file will result in a file size in the expected range.

     0blocks % du -h rev00336.dat 
 1231M	rev00336.dat
 2
 3blocks % hexdump -C rev00336.dat | tail -4
 40145c010  ea 7b e8 e9 86 7e c0 bc  fb 13 00 00 00 00 00 00  |.{...~..........|
 50145c020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 6*
 701500000
 8
 9% grep "rev00336.dat" debug.log          
102019-12-30T20:48:10Z Pre-allocating up to position 0x100000 in rev00336.dat
112019-12-30T20:48:39Z Pre-allocating up to position 0x200000 in rev00336.dat
122019-12-30T20:48:43Z Pre-allocating up to position 0x300000 in rev00336.dat
132019-12-30T20:48:45Z Pre-allocating up to position 0x400000 in rev00336.dat
142019-12-30T20:48:49Z Pre-allocating up to position 0x500000 in rev00336.dat
152019-12-30T20:48:50Z Pre-allocating up to position 0x600000 in rev00336.dat
162019-12-30T20:48:51Z Pre-allocating up to position 0x700000 in rev00336.dat
172019-12-30T20:48:52Z Pre-allocating up to position 0x800000 in rev00336.dat
182019-12-30T20:48:54Z Pre-allocating up to position 0x900000 in rev00336.dat
192019-12-30T20:48:55Z Pre-allocating up to position 0xa00000 in rev00336.dat
202019-12-30T20:48:57Z Pre-allocating up to position 0xb00000 in rev00336.dat
212019-12-30T20:48:58Z Pre-allocating up to position 0xc00000 in rev00336.dat
222019-12-30T20:49:00Z Pre-allocating up to position 0xd00000 in rev00336.dat
232019-12-30T20:49:01Z Pre-allocating up to position 0xe00000 in rev00336.dat
242019-12-30T20:49:02Z Pre-allocating up to position 0xf00000 in rev00336.dat
252019-12-30T20:49:03Z Pre-allocating up to position 0x1000000 in rev00336.dat
262019-12-30T20:49:05Z Pre-allocating up to position 0x1100000 in rev00336.dat
272019-12-30T20:49:06Z Pre-allocating up to position 0x1200000 in rev00336.dat
282019-12-30T20:49:08Z Pre-allocating up to position 0x1300000 in rev00336.dat
292019-12-30T20:49:10Z Pre-allocating up to position 0x1400000 in rev00336.dat
302019-12-30T20:49:12Z Pre-allocating up to position 0x1500000 in rev00336.dat
31
32blocks % ls -lh rev00336.dat 
33-rw-------  1 erik  staff    21M Dec 30 22:49 rev00336.dat
34
35blocks % cp rev00336.dat rev00336.copy
36
37blocks % du -h rev00336.*
38 21M	rev00336.copy
39231M	rev00336.dat

    So a workaround might be to replace all the rev files with copies.

  13. fanquake renamed this:
    The Initial Block Download (IBD) on macOS Catalina uses a lot more disk space than the expected 284GB
    Blocks directory on macOS uses more disk space than expected
    on Dec 31, 2019
  14. fanquake commented at 6:21 pm on December 31, 2019: member

    No, this is not intended behavior at all. The rev*.dat files shouldn’t go above 15-20 MB usually.

    I currently have a fully synced node running on macOS (10.14) that uses nearly 500GB on disk:

     0475G	total
 1422G	blocks
 2 40G	indexes
 38.4G	testnet3
 44.1G	chainstate
 5356M	regtest
 69.8M	debug.log
 74.0M	wallet.dat
 84.0M	peers.dat
 94.0M	mempool.dat
10896K	some_wallet
11256K	fee_estimates.dat
124.0K	bitcoin.conf
134.0K	banlist.dat
14  0B	db.log
15  0B	.walletlock
16  0B	.lock

    Largest rev*.dat files:

    0206M	rev00352.dat
1193M	rev00331.dat
2190M	rev00461.dat
3190M	rev00447.dat
4190M	rev00390.dat
5190M	rev00389.dat
6190M	rev00386.dat
7190M	rev00374.dat
  15. eriknylund commented at 7:55 am on January 1, 2020: contributor

    No, this is not intended behavior at all. The rev*.dat files shouldn’t go above 15-20 MB usually.

    I currently have a fully synced node running on macOS (10.14) that uses nearly 500GB on disk:

     0475G	total
 1422G	blocks
 2 40G	indexes
 38.4G	testnet3
 44.1G	chainstate
 5356M	regtest
 69.8M	debug.log
 74.0M	wallet.dat
 84.0M	peers.dat
 94.0M	mempool.dat
10896K	some_wallet
11256K	fee_estimates.dat
124.0K	bitcoin.conf
134.0K	banlist.dat
14  0B	db.log
15  0B	.walletlock
16  0B	.lock

    Largest rev*.dat files:

    0206M	rev00352.dat
1193M	rev00331.dat
2190M	rev00461.dat
3190M	rev00447.dat
4190M	rev00390.dat
5190M	rev00389.dat
6190M	rev00386.dat
7190M	rev00374.dat

    And if you copy one of the larger files, will it reduce in size to the expected 15-20 MB range?

  16. streamofstars commented at 3:12 pm on January 1, 2020: none
    What is the file system used on the drive that keeps blocks folder? Also, can you post your bitcoin.conf file?
  17. sipa commented at 4:48 pm on January 1, 2020: member
    The code that does the OS-specific file allocation hasn’t been touched in 7 years (#2229), so I wonder what suddenly causes this.
  18. eriknylund commented at 5:07 pm on January 1, 2020: contributor

    What is the file system used on the drive that keeps blocks folder? Also, can you post your bitcoin.conf file?

    Both machines use APFS.

    mac-mini-disk-utility

    I have no bitcoin.conf file in use, so only defaults should be used. This is a clean install using the dmg, no custom settings have been done.

  19. eriknylund commented at 5:10 pm on January 1, 2020: contributor

    The code that does the OS-specific file allocation hasn’t been touched in 7 years (#2229), so I wonder what suddenly causes this.

    It may not be so sudden, considering @fanquake ’s 10.14 (macOS Mojave) machine that shows the same behavior.

  20. streamofstars commented at 11:09 pm on January 1, 2020: none

    I am struggling to reproduce this issue. I am now around 43% downloading blocks progress and upon investigating the blocks folder, I see all my rev* files are between 17-22MB and blk* files are around 134MB each, so I guess all looks good.

    Tested on mac mini (2018) with macos 10.15.2 (19C57), 16GB RAM, i5. The drive selected for data is 500GB encrypted APFS partition on sata3 ssd connected via usb-c adapter. I haven’t customized anything except unchecking pruning option and selecting the external drive as a data folder.

    What about your wallet file, is it just created from scratch or did you put an existing, old wallet into the bitcoin folder?

  21. eriknylund commented at 5:47 am on January 2, 2020: contributor

    I am struggling to reproduce this issue. I am now around 43% downloading blocks progress and upon investigating the blocks folder, I see all my rev* files are between 17-22MB and blk* files are around 134MB each, so I guess all looks good.

    Tested on mac mini (2018) with macos 10.15.2 (19C57), 16GB RAM, i5. The drive selected for data is 500GB encrypted APFS partition on sata3 ssd connected via usb-c adapter. I haven’t customized anything except unchecking pruning option and selecting the external drive as a data folder.

    What about your wallet file, is it just created from scratch or did you put an existing, old wallet into the bitcoin folder?

    Interesting! Everything is completely from scratch, even the wallet. On the second machine my APFS is encrypted, and it’s also showing the same issue, the largest rev files is 184M.

     0$ du -ch rev*.dat | sort -n
 1 16M	rev00003.dat
 2 17M	rev00056.dat
 3 18M	rev00006.dat
 4 20M	rev00000.dat
 5...
 6174M	rev00256.dat
 7175M	rev00143.dat
 8175M	rev00275.dat
 9177M	rev00132.dat
10177M	rev00204.dat
11178M	rev00337.dat
12184M	rev00208.dat

    I’ve located a spare 500 GB external SSD so I will try to reproduce with an external drive to see what happens.

  22. eriknylund commented at 6:12 am on January 2, 2020: contributor

    As a workaround based on a tip from the IRC discussions, I did a small bash script to slim down the rev files one by one, by copying them once, allowing me to continue from 83% and finish the download on the mac mini. These are the steps I took, below reproduced on my second machine that was on 23% when I stopped downloading, hence the total size is lower, but you can see there is already an extra 40G of space being consumed:

     0$ cd Library/Application\ Support/Bitcoin/blocks
 1$ du -h
 2 91M	./index
 3105G	.
 4
 5$ cat slim.sh 
 6for file in ./rev*.dat
 7do
 8    if [[ -f $file ]]; then
 9	size_before=$(du -h $file | cut -f1)
10        cp $file $file.tmp
11        rm $file
12	mv $file.tmp $file
13	size_after=$(du -h $file | cut -f1)
14	echo $file $size_before " -> " $size_after
15    fi
16done
17
18$ bash slim.sh 
19./rev00000.dat 20M  ->  19M
20./rev00001.dat 33M  ->  17M
21./rev00002.dat 27M  ->  16M
22./rev00003.dat 16M  ->  16M
23...
24./rev00443.dat 108M  ->  19M
25./rev00444.dat 114M  ->  19M
26./rev00445.dat 52M  ->  19M
27./rev00446.dat 77M  ->  19M
28./rev00447.dat 54M  ->  19M
29./rev00448.dat 62M  ->  19M
30./rev00449.dat 79M  ->  17M
31./rev00450.dat 21M  ->  6.0M
32
33$ du -h
34 91M	./index
35 64G	.
  23. streamofstars commented at 12:19 pm on January 2, 2020: none

    I kinda need to retract what I said before. Today I woke up to a bitcoin core message “Error: Disk space is too low!”. This was at 91.20% of IBD. I checked the drive and About this Mac, Storage says only 80MB is left on the 500GB dedicated partition. Du and df commands and disk utility report the whole partition space used. Finder also reports whole disk space used when checking the whole drive but when checking the bitcoin folder only it says it takes only 273GB (there is really nothing else on this partition). Upon investigating further, I can see that Get Info option on a folder returns: 273 165 546 724 bytes (499,43 GB on disk) for 5 369 items similarly, all rev files in the blocks folder are visible in Finder as around 19MB files but when using Get info option the true size is revealed, e.g. 19 922 944 bytes (199,3 MB on disk) Same goes for ncdu, it reports true size of files on the disk and it seems that a lot of rev files actually take much more space than they should (e.g. 200MB).

    I checked the debug.log and noticed that the rev files that are the biggest have the most Pre-allocating up to position, messages, and the smallest ones have the least:

    • rev01037, 350.2 MiB, pre-allocating goes up to 0x1a00000
    • rev01767, 8.6MiB, pre-allocating goes up to 0x500000 only

    FYI, the Time Machine is disabled for this drive, later I disabled it completely, restarted the machine, this did not free up any space.

    I guess that some dev needs to investigate closer how this pre-allocating mechanism behaves on APFS drives. Perhaps it reserves more space than it needs and later on it does not release it properly?

  24. eriknylund commented at 12:47 pm on January 2, 2020: contributor

    I kinda need to retract what I said before. Today I woke up to a bitcoin core message “Error: Disk space is too low!”. This was at 91.20% of IBD. I checked the drive and About this Mac, Storage says only 80MB is left on the 500GB dedicated partition. Du and df commands and disk utility report the whole partition space used. Finder also reports whole disk space used when checking the whole drive but when checking the bitcoin folder only it says it takes only 273GB (there is really nothing else on this partition). Upon investigating further, I can see that Get Info option on a folder returns: 273 165 546 724 bytes (499,43 GB on disk) for 5 369 items similarly, all rev files in the blocks folder are visible in Finder as around 19MB files but when using Get info option the true size is revealed, e.g. 19 922 944 bytes (199,3 MB on disk) Same goes for ncdu, it reports true size of files on the disk and it seems that a lot of rev files actually take much more space than they should (e.g. 200MB).

    I checked the debug.log and noticed that the rev files that are the biggest have the most Pre-allocating up to position, messages, and the smallest ones have the least:

    * rev01037, 350.2 MiB, pre-allocating goes up to 0x1a00000

* rev01767, 8.6MiB, pre-allocating goes up to 0x500000 only

    FYI, the Time Machine is disabled for this drive, later I disabled it completely, restarted the machine, this did not free up any space.

    I guess that some dev needs to investigate closer how this pre-allocating mechanism behaves on APFS drives. Perhaps it reserves more space than it needs and later on it does not release it properly?

    Thanks for letting me know. Then I won’t spend time on testing with an external drive right now, but I’ll be prepared to do it later if needed.

  25. eriknylund commented at 10:51 pm on January 4, 2020: contributor

    To further pinpoint the problem I created two disk images using Disk Utility, one with Mac OS Extended (Journaled) and the other with APFS. I then symlinked the volumes to the default data directory:

    0ln -s /Volumes/MacOSExtended /Users/erik/Library/Application Support/Bitcoin

    After that I started bitcoind to download the first five blocks and stopped. Then I symlinked the other:

    0rm /Users/erik/Library/Application Support/Bitcoin
1ln -s /Volumes/APFS /Users/erik/Library/Application Support/Bitcoin

    and ran bitcoind again to sync the first five blocks.

    The problem only seems to exist on APFS:

     0$ cd /Volumes/APFS/blocks/
 1blocks erik$ du -ch *.dat
 2128M	blk00000.dat
 3128M	blk00001.dat
 4128M	blk00002.dat
 5128M	blk00003.dat
 6128M	blk00004.dat
 7 19M	rev00000.dat
 8 34M	rev00001.dat
 9 31M	rev00002.dat
10 32M	rev00003.dat
11 49M	rev00004.dat
12805M	total
13blocks erik$ ls -lh *.dat
14-rw-------  1 erik  staff   128M  5 Tam 00:25 blk00000.dat
15-rw-------  1 erik  staff   128M  5 Tam 00:26 blk00001.dat
16-rw-------  1 erik  staff   128M  5 Tam 00:27 blk00002.dat
17-rw-------  1 erik  staff   128M  5 Tam 00:28 blk00003.dat
18-rw-------  1 erik  staff   128M  5 Tam 00:30 blk00004.dat
19-rw-------  1 erik  staff    19M  5 Tam 00:25 rev00000.dat
20-rw-------  1 erik  staff    17M  5 Tam 00:26 rev00001.dat
21-rw-------  1 erik  staff    16M  5 Tam 00:28 rev00002.dat
22-rw-------  1 erik  staff    16M  5 Tam 00:29 rev00003.dat
23-rw-------  1 erik  staff    17M  5 Tam 00:30 rev00004.dat

    Mac OS Extended (HFS+):

     0$ cd /Volumes/MacOSExtended/blocks/
 1blocks erik$ du -ch *.dat
 2128M	blk00000.dat
 3128M	blk00001.dat
 4128M	blk00002.dat
 5128M	blk00003.dat
 6128M	blk00004.dat
 7 19M	rev00000.dat
 8 17M	rev00001.dat
 9 16M	rev00002.dat
10 16M	rev00003.dat
11 17M	rev00004.dat
12724M	total
13blocks erik$ ls -lh *.dat
14-rw-------  1 erik  staff   128M  5 Tam 00:02 blk00000.dat
15-rw-------  1 erik  staff   128M  5 Tam 00:03 blk00001.dat
16-rw-------  1 erik  staff   128M  5 Tam 00:05 blk00002.dat
17-rw-------  1 erik  staff   128M  5 Tam 00:06 blk00003.dat
18-rw-------  1 erik  staff   128M  5 Tam 00:07 blk00004.dat
19-rw-------  1 erik  staff    19M  5 Tam 00:02 rev00000.dat
20-rw-------  1 erik  staff    17M  5 Tam 00:04 rev00001.dat
21-rw-------  1 erik  staff    16M  5 Tam 00:05 rev00002.dat
22-rw-------  1 erik  staff    16M  5 Tam 00:06 rev00003.dat
23-rw-------  1 erik  staff    17M  5 Tam 00:07 rev00004.dat

    The diff:

    image

  26. sipa commented at 11:00 pm on January 4, 2020: member
    What are the exact file sizes you’re seeing for the rev*.dat files? Are they all multiples of some power of two? That could give a hint about where the problem is.
  27. IPGlider commented at 12:34 pm on January 6, 2020: contributor

    Checking the file content

    The file rev00337.dat has a reported size by du of 212MB before copying and 21MB after. However, the content of the files is exactly the same.

     0cd ~/Library/Application\ Support/Bitcoin/blocks
 1
 2du -ch ./rev00337.dat
 3 212M	./rev00337.dat
 4 212M	total
 5
 6hexdump -C rev00337.dat | tail -4
 70141a770  e0 92 b0 c3 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 80141a780  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 9*
1001500000
11
12cp rev00337.dat ~/Desktop
13
14cd ~/Desktop
15
16hexdump -C rev00337.dat | tail -4
170141a770  e0 92 b0 c3 00 00 00 00  00 00 00 00 00 00 00 00  |................|
180141a780  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
19*
2001500000

    Checking that the files content is the same after copy

    I have modified the previously posted script to show the files hashes before and after copy.

     0for file in ./rev*.dat
 1do
 2    if [[ -f $file ]]; then
 3	size_before=$(du -h $file | cut -f1)
 4	hash_before=$(shasum -a 256 $file | cut -d" " -f1)
 5        cp $file $file.tmp
 6        rm $file
 7	mv $file.tmp $file
 8	size_after=$(du -h $file | cut -f1)
 9        hash_after=$(shasum -a 256 $file | cut -d" " -f1)
10	echo $file $size_before " -> " $size_after " hash_before: " $hash_before "hash_after: " $hash_after
11    fi
12done

    Here is a sample output:

    0./rev01590.dat 3.9M  ->  4.0M  hash_before:  9fa0f20cfc1d9593b873399130cfce323aa574473f3a6b7caa58a970e41dc45b hash_after:  9fa0f20cfc1d9593b873399130cfce323aa574473f3a6b7caa58a970e41dc45b
1./rev01591.dat 844K  ->  1.0M  hash_before:  ab8811a65806921492c63c85c3d05643a0a7cba9fb5b3a677cb16fabb1167918 hash_after:  ab8811a65806921492c63c85c3d05643a0a7cba9fb5b3a677cb16fabb1167918

    All the hashes are the same before and after copying the files.

    Difference in file size reporting

    The file size reported by du is different from the one reported by ls and stat.

     0cd ~/Library/Application\ Support/Bitcoin/blocks
 1
 2du -ch ./rev01668.dat
 3 90M	./rev01668.dat
 4 90M	total
 5
 6ls -lah | grep rev01668.dat 
 7-rw-------     1 something  something    17M Jan  6 11:18 rev01668.dat
 8
 9stat -x rev01668.dat          
10  File: "rev01668.dat"
11  Size: 17825792     FileType: Regular File
12  Mode: (0600/-rw-------)         Uid: (  502/  something)  Gid: (   20/   something)
13Device: 1,4   Inode: 65189970    Links: 1
14Access: Mon Jan  6 11:53:20 2020
15Modify: Mon Jan  6 11:18:58 2020
16Change: Mon Jan  6 11:18:58 2020

    Diagnosis

    du is reporting the size in used blocks, ls the size of the files content.

     0diskutil info / | grep "Block Size"
 1   Device Block Size:         4096 Bytes
 2   Allocation Block Size:     4096 Bytes
 3
 4cd ~/Library/Application\ Support/Bitcoin/blocks
 5
 6du -ch ./rev01668.dat
 7 90M	./rev01668.dat
 8 90M	total
 9
10BLOCKSIZE=4096 du -c ./rev01668.dat
1122916	./rev01668.dat
1222916	total
13
14BLOCKSIZE=4096 ls -lahs | grep rev01668.dat
15 22916 -rw-------     1 something  something    17M Jan  6 11:18 rev01668.dat
16
17cp rev01668.dat ~/Desktop
18
19cd ~/Desktop
20
21du -ch ./rev01668.dat
22 17M	./rev01668.dat
23 17M	total
24
25BLOCKSIZE=4096 du -c ./rev01668.dat
264352	./rev01668.dat
274352	total
28
29BLOCKSIZE=4096 ls -lahs | grep rev01668.dat
30  4352 -rw-------   1 something  something    17M Jan  6 12:08 rev01668.dat

    It can be seen that, after copying the file, the number of blocks used is reduced drastically and the file size is as expected.

    Hypothesis and troubleshooting

    ftruncate function is not working as expected when used in an APFS formated volume.

    I have tried to reproduce the problem in an isolated environment wirhout any success. The files sizes were always correct.

    Possible fix

    If the function AllocateFileRange is modified to use the fallback method in macOS the file sizes are as expected and the problem dissapear.

  28. eriknylund commented at 12:43 pm on January 6, 2020: contributor

    Possible fix

    If the function AllocateFileRange is modified to use the fallback method in macOS the file sizes are as expected and the problem dissapear.

    I can confirm that I’ve tried this fix and using the fallback works. I also lean towards something with the ftruncate being broken, but wouldn’t you also see the same problem in the blk*.dat files then?

  29. eriknylund commented at 12:46 pm on January 6, 2020: contributor

    What are the exact file sizes you’re seeing for the rev*.dat files? Are they all multiples of some power of two? That could give a hint about where the problem is.

    I cannot see any indication that they would be multiples of power of two, unless I’m missing something.

  30. kallewoof commented at 5:35 am on January 8, 2020: member

    @sipa

    What are the exact file sizes you’re seeing for the rev*.dat files? Are they all multiples of some power of two? That could give a hint about where the problem is.

    This is not how it ends up, actually. Every pre-allocation will append “expected size + 1 MB” to the actual size for each allocation time. I.e.

    0(once)
1a = 1024*1024
2b = a
3(for each increase)
4a = a + 1024*1024; b = b+a; b/1024/1024

    This gives the sizes 6, 10, 15, 21, 28, 36, 45, 55, 66, 78, 91, 105, 120, 136, 153, 171, … which exactly matches the sizes I see.

  31. laanwj closed this on Jan 22, 2020
  32. sidhujag referenced this in commit 6234d8cce9 on Jan 24, 2020
  33. earthsound commented at 3:26 pm on June 17, 2020: none

    As a workaround based on a tip from the IRC discussions, I did a small bash script to slim down the rev files one by one, by copying them once, allowing me to continue from 83% and finish the download on the mac mini. These are the steps I took, below reproduced on my second machine that was on 23% when I stopped downloading, hence the total size is lower, but you can see there is already an extra 40G of space being consumed:

    Thanks for the workaround of copying them out. I was able to reduce disk usage by 237G. My before & after:

    0$ du -h
1 92M	./index
2538G	.

    0$ du -h
1 92M	./index
2301G
  34. sidhujag referenced this in commit 3ffeb8e979 on Nov 10, 2020
  35. PastaPastaPasta referenced this in commit 1b5ea905a3 on Sep 17, 2021
  36. thelazier referenced this in commit c4ce0d1cb6 on Sep 25, 2021
  37. DrahtBot locked this on Feb 15, 2022


eriknylund sipa fanquake streamofstars IPGlider kallewoof earthsound

Labels
Bug macOS

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-22 06:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me