util: detect and warn when using exFAT on MacOS #31453

willcl-ark commented at 10:29 AM on December 10, 2024: member

exFAT is known to cause intermittent corruption on MacOS.

Therefore we should warn when using this fs format for either the blocks or data directories.

See #28552 for more context.

DrahtBot commented at 10:30 AM on December 10, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31453.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	l0rinc, ismaelsadeeq, marcofleon
Concept ACK	laanwj, sipa, helloscoopa, hebasto, edilmedeiros, davidgumberg, Zero-1729, pablomartin4btc
Stale ACK	tdb3

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

No conflicts as of last run.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

“macOS,the” -> “macOS, the” [missing space after comma]

<sup>drahtbot_id_4_m</sup>

DrahtBot added the label Utils/log/libs on Dec 10, 2024

willcl-ark commented at 10:32 AM on December 10, 2024: member

I did toy with having this trigger a full on InitError, but the behaviour is intermittent and seems worst during IBD. Exiting would also likely break things for a fair few users, due to exFAT being used in cross-platform storage devices. Therefore a warning seems more apt.

The root cause seems to be that some calls to F_FULLFSYNC fcntl silently failing on exFAT. This appears to stem from the fact that, whilst supported, F_FULLFSYNC may not be fully implemented for exFAT:

F_FULLFSYNC Does the same thing as fsync(2) then asks the drive to flush all buffered data to the permanent storage device (arg is ignored). This is currently imple-mented implemented mented on HFS, MS-DOS (FAT), and Universal Disk Format (UDF) file systems. The operation may take quite a while to complete. Certain FireWire drives have also been known to ignore the request to flush their buffered data.

link: https://newosxbook.com/src.jl?tree=xnu&file=/bsd/man/man2/fcntl.2 and: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fcntl.2.html#//apple_ref/doc/man/2/fcntl

From debugging this seems to manifest from fast processing speed; if I add enough LogPrintf's then it eventually goes away, presumably due to the reduced IBD speed. This may also indicate that users on slower (CPU) machines may suffer from this less frequently.

We do read and write to the block files from the correct position/offsets at failure but the wrong data is found there. So this appears to be an issue at the kernel level.

I will open a new issue with some outputs from my debugging on this.

in src/common/init.cpp:81 in a7d52d023e outdated

  76 | +        }};
  77 | +        for (const auto& check : paths_to_check) {
  78 | +            FSType fs_type = GetFilesystemType(check.path);
  79 | +            switch(fs_type) {
  80 | +                case FSType::EXFAT:
  81 | +                    LogWarning("Specified %s \"%s\" is exFAT which is known to have intermittent corruption problems on MacOS.", check.description, fs::PathToString(check.path));

maflcko commented at 10:48 AM on December 10, 2024:

Shouldn't this be an InitWarning? Otherwise gui users (basically everyone on macos?) will never see this on their screen, unless they dig up the debug log at the right position?

willcl-ark commented at 11:43 AM on December 10, 2024:

Thanks for catching marco. It was indeed a mistake to change this from an InitError in my first draft to LogWarning.

Will fixup shortly

willcl-ark commented at 10:46 AM on December 11, 2024:

Done in 83839a35f07055dd03924b5bdab46bf31df33b35

in src/common/init.cpp:66 in a7d52d023e outdated

  62 | @@ -62,6 +63,32 @@ std::optional<ConfigError> InitConfig(ArgsManager& args, SettingsAbortFn setting
  63 |              fs::create_directories(net_path / "wallets");
  64 |          }
  65 |  
  66 | +        // Warn if we are trying to put the datadir on an exFAT fs on MacOS
  67 | +        // This is an upstream issue known to cause bugs, see #28552
  68 | +#ifdef __APPLE__

l0rinc commented at 11:15 AM on December 10, 2024:

Q: is this related to a macs or just generally using flash drives as storage?

willcl-ark commented at 11:19 AM on December 10, 2024:

I haven't encountered the same error using this drive to sync when formatted with APFS, so I in my opinion, no.

I have come across comments online purporting that certain brands of nvme drives don't play well with apple's (internal) nvme controllers, which can also cause similar issues. But here I feel pretty certain this is an exFAT x MacOS issue, and we may well see the same issues with an internal exFAT drive.

If anyone has a mac with an internal sata/nvme drive which they could format to exFAT and test IBD with, that would be great! (I haven't)

willcl-ark commented at 11:20 AM on December 10, 2024:

Oh I should add here, (and probably in #31454) that I tested this same drive using exFAT on Linux, and did not see any corruption to block 450k. By which point every MacOS test had corrupted some data and failed.

edilmedeiros commented at 5:32 PM on January 3, 2025:

If my memory is not failing me, I have seem only macOS related issues reporting problems with exFAT external drivers. I tried exFAT on the internal drive of an old 2016 Intel Macbook Pro (original 1GB SSD, APFS on a primary partition and exFAT on a secondary partition for Bitcoin data) and IBD failed on v26. It was about a year ago, I was trying to understand if it was a problem with the external drive I was using at the time or a software problem. Unfortunately, I can't try that again with that computer now.

l0rinc commented at 4:10 PM on February 16, 2025:

Even when there isn't any corruption, maybe it would help to warn when SD cards are used for storage

philmb3487 commented at 2:59 PM on February 17, 2025:

Is that really judicious? Plenty of people run Bitcoind on Raspberry Pi's!

l0rinc commented at 3:32 PM on February 17, 2025:

Yes, but mostly with SSDs, not SD cards.

l0rinc commented at 2:00 PM on February 18, 2025:

Please resolve the comment, seems it's not strictly related to the issue

laanwj commented at 11:24 AM on April 2, 2025:

Oh I should add here, (and probably in #31454) that I tested this same drive using exFAT on Linux, and did not see any corruption to block 450k. By which point every MacOS test had corrupted some data and failed.

The bug is specific to macOS and exFAT. Though, exFAT is a trival filesystem meant for things like storing photos (for cameras) and other archival purposes, using it for database-heavy access patterns such as a bitcoin node is a bad idea in general.

marcofleon approved

marcofleon commented at 11:35 AM on December 10, 2024: contributor

utACK a7d52d023eee6559d04e2e191f17176f2fe92c00. This happened to me earlier this year during IBD using a mac M3 and an external SSD formatted by default as exFAT. Took me a bit to figure it out, so a warning would've helped.

Reformatting to APFS solved the issue for me. Would that be worth adding to the warning as a suggestion for a solution?

willcl-ark commented at 11:38 AM on December 10, 2024: member

Reformatting to APFS solved the issue for me. Would that be worth adding to the warning as a suggestion for a solution?

Yes that could be nice. Another idea that I had suggested was linking to a help page. Perhaps I could dd a section to https://github.com/bitcoin/bitcoin/blob/master/doc/files.md describing the problematic FS/OS combination and add the url to the warning to make things super-easy for users to fix themselves?

cc @fanquake

laanwj commented at 12:13 PM on December 10, 2024: member

Concept ACK, i think this makes sense in general. exFAT is meant for storing photos, not as a mature filesystem for database workloads, so subpar reliability and performance are to be expected.

l0rinc commented at 2:10 PM on December 10, 2024: contributor

Concept ACK

willcl-ark force-pushed on Dec 10, 2024

sipa commented at 4:19 PM on December 10, 2024: member

Concept ACK

willcl-ark force-pushed on Dec 10, 2024

DrahtBot added the label CI failed on Dec 10, 2024

DrahtBot commented at 4:47 PM on December 10, 2024: contributor

🚧 At least one of the CI tasks failed. <sub>Debug: https://github.com/bitcoin/bitcoin/runs/34204028171</sub>

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

DrahtBot removed the label CI failed on Dec 10, 2024

marcofleon commented at 11:39 AM on December 11, 2024: contributor

Nice, the warning and new addition to the docs lgtm.

ACK ee05e730466f250e131eb92165bcdb8bab68c864

DrahtBot requested review from laanwj on Dec 11, 2024

DrahtBot requested review from l0rinc on Dec 11, 2024

DrahtBot requested review from sipa on Dec 11, 2024

helloscoopa commented at 12:09 PM on December 11, 2024: none

Concept ACK

hebasto commented at 10:36 AM on December 12, 2024: member

Concept ACK.

in src/common/init.cpp:75 in ee05e73046 outdated

  72 | +            std::string_view description;
  73 | +        };
  74 | +        std::array<PathCheck, 2> paths_to_check{{
  75 | +            {args.GetDataDirNet(), "data directory"},
  76 | +            {args.GetBlocksDirPath(), "blocks directory"}
  77 | +        }};

tdb3 commented at 2:27 AM on December 15, 2024:

Makes sense to cover blocksdir and datadir. Since the user could also set -walletdir=<dir> when starting bitcoind, do you think it would be worth warning for it as well? Much lighter disk usage than blocksdir and datadir, but if we're doubtful of macOS's exFAT implementation, then maybe we would want to warn the user when storing wallets with it?

edilmedeiros commented at 5:14 PM on January 3, 2025:

Agree with this, even though I have never encountered any problems with the wallet files on a Mac with a FAT formatted disk, only on the blocks dir.

willcl-ark commented at 12:40 PM on January 20, 2025:

I am not convinced that walletdir is a problem, do we have any reports of this?

I can add it, but it looks cleanest to first refactor GetWalletDir into ArgsMan, so that we can fetch this directory here in common/init.cpp, which will make this change quite a bit bigger...

tdb3 commented at 10:59 PM on January 20, 2025:

Fair point. Either approach (the refactor, or deferring) sounds reasonable. In the big picture, the warnings for blocksdir and datadir add value and I don't think walletdir warning should hold up the PR.

I do wonder how common it actually is that someone (using macOS) is using exFAT for just the walletdir, and would therefore miss the warning. I would imagine most users would be using the default walletdir location, and could therefore see the warning as it pertains to datadir.

tdb3 commented at 2:29 AM on December 15, 2024: contributor

Concept ACK Great find!

in src/common/init.cpp:82 in ee05e73046 outdated

  77 | +        }};
  78 | +        for (const auto& check : paths_to_check) {
  79 | +            FSType fs_type = GetFilesystemType(check.path);
  80 | +            switch(fs_type) {
  81 | +                case FSType::EXFAT:
  82 | +                    InitWarning(strprintf(_("Specified %s \"%s\" is exFAT which is known to have intermittent corruption problems on MacOS. "

luke-jr commented at 12:55 AM on December 25, 2024:

For GUI, maybe we should check this while they still have the opportunity to change it...

philmb3487 commented at 8:05 AM on January 1, 2025:

just one nit, the OS is named 'macOS'

Zero-1729 commented at 9:23 PM on January 14, 2025:

I'd be good to add a recommendation here, too, so the user is aware of possible immediate solutions.

                    InitWarning(strprintf(_("Specified %s \"%s\" is exFAT which is known to have intermittent corruption problems on MacOS, use APFS instead. "

willcl-ark commented at 12:47 PM on January 20, 2025:

For GUI, maybe we should check this while they still have the opportunity to change it...

To add this check to the qt/intro.cpp flow would require duplicating some of the init checks, as well as:

first creating the chosen directory
check what FS Type the created dir is
if EXFAT prompt the user to change filesystems
either remove the just-created directories (or keep them) depending on response.
etc.

If we want that kind of logic then I'm happy to add it, but not sure if we need much more than a loud InitWarning myself.

just one nit, the OS is named 'macOS'

Renamed in all places

gentlee commented at 8:02 PM on April 10, 2025:

May be let's not insist so much and use ", consider using APFS instead."

luke-jr commented at 12:55 AM on December 25, 2024: member

Would be ideal if we had a way to reproduce the bug, and can test if the user is actually affected, in case Apple fixes it. But this is better than nothing for now

edilmedeiros commented at 5:26 PM on January 3, 2025: contributor

Concept ACK.

Already lost many hours trying to debug this and my conclusion so far is that the FAT driver on macOS is flawed. Also, it's quite difficult to reproduce this reliably, IBD will fail in a different point every time. Maybe we can detect a broken block in persistent memory and restart IBD from there?

davidgumberg commented at 11:36 PM on January 8, 2025: contributor

Concept ACK

I second the suggestion by @tdb3 above to also check -walletdir, although I have looked and have not been able to find similar reports of corruption with SQLite and exFAT drives on macOS.

To add a little data pointing in the direction of this bug only existing on macOS, I have synced nodes with datadirs on exFAT drives over both SATA and USB on Fedora 40 and have a long-running bitcoind node that has been running off of an exFAT SATA SSD on Fedora 40 for the past ~6 months and have never experienced any of the symptoms described in #28552 or #31454.

in src/util/fs_helpers.cpp:34 in ee05e73046 outdated

  40 | @@ -41,6 +41,10 @@
  41 |  #include <shlobj.h> /* For SHGetSpecialFolderPathW */
  42 |  #endif // WIN32
  43 |  
  44 | +#ifdef __APPLE__
  45 | +#include <sys/mount.h>

luke-jr commented at 9:12 PM on January 9, 2025:

The manpage says we need #include <sys/param.h> too

https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/statfs.2.html

willcl-ark commented at 12:42 PM on January 20, 2025:

added in df1ba101419729aafa382d970ee605e2a6273e26

in doc/files.md:132 in ee05e73046 outdated

 124 | @@ -123,6 +125,12 @@ Path           | Description | Repository notes
 125 |  `addr.dat`     | Peer IP address BDB database; replaced by `peers.dat` in [0.7.0](https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.7.0.md) | [PR #1198](https://github.com/bitcoin/bitcoin/pull/1198), [`928d3a01`](https://github.com/bitcoin/bitcoin/commit/928d3a011cc66c7f907c4d053f674ea77dc611cc)
 126 |  `onion_private_key` | Cached Tor onion service private key for `-listenonion` option. Was used for Tor v2 services; replaced by `onion_v3_private_key` in [0.21.0](https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.21.0.md) | [PR #19954](https://github.com/bitcoin/bitcoin/pull/19954)
 127 |  
 128 | +## Filesystem recommendations
 129 | +
 130 | +When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`), some filesystems should be avoided:
 131 | +
 132 | +- **MacOS**: The exFAT filesystem should not be used. There have been multiple reports of database corruption when using exFAT on MacOS for Bitcoin Core. This appears to be due to filesystem-level issues with exFAT on MacOS. See [Issue #31454](https://github.com/bitcoin/bitcoin/issues/31454) for more details.

Zero-1729 commented at 9:22 PM on January 14, 2025:

It'd be good to clarify here that it is recommended to use APFS instead.

- **MacOS**: The exFAT filesystem should not be used, it is recommended to use the APFS filesystem instead. There have been multiple reports of database corruption when using exFAT on MacOS for Bitcoin Core. This appears to be due to filesystem-level issues with exFAT on MacOS. See [Issue [#31454](/bitcoin-bitcoin/31454/)](https://github.com/bitcoin/bitcoin/issues/31454) for more details.

willcl-ark commented at 10:09 AM on August 4, 2025:

I've avoided explicit recommendation of a FS on purpose here; as far as I know exFAT is the only known-problematic FS on macOS?

Can be convinced otherwise, but leaving as-is for now

Zero-1729 commented at 9:33 PM on January 14, 2025: contributor

Concept ACK

I encountered this error late last year as well. Thank you for working on this. I think having the warning reported is a good measure.

I've dropped a few comments above.

I tested the patch on an M1 MBP running MacOS v15.2 with an exFAT drive (83839a35f07055dd03924b5bdab46bf31df33b35):

$ ./build/src/qt/bitcoin-qt --datadir=/Volumes/Little\ Test/

Warning: Specified data directory "/Volumes/Little Test" is exFAT which is known to have intermittent corruption problems on MacOS. See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.
Warning: Specified blocks directory "/Volumes/Little Test/blocks" is exFAT which is known to have intermittent corruption problems on MacOS. See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.
2025-01-14 19:45:56.627 bitcoin-qt[55978:5075996] +[IMKClient subclass]: chose IMKClient_Modern
2025-01-14 19:45:56.627 bitcoin-qt[55978:5075996] +[IMKInputSession subclass]: chose IMKInputSession_Modern
...

As a minor nit, I do think given this adds a whole bunch of references to macOS, it should be formatted as macOS instead of MacOS for consistency, as macOS is used more often across the repo.

Also, I think it'll be helpful to have the disk format warning displayed as a dialog for those running the GUI, same way we get a warning if the disk is too small to accommodate blocks:

willcl-ark force-pushed on Jan 20, 2025

willcl-ark commented at 1:55 PM on January 20, 2025: member

df1ba101419729aafa382d970ee605e2a6273e26 squashes the commits together, and also removes the duplicate warning messages, instead printing a single warning with a list of violating directories, e.g. going from:

Warning: Specified data directory "/Volumes/sandisk1/signet" is exFAT which is known to have intermittent corruption problems on MacOS. See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.
Warning: Specified blocks directory "/Volumes/sandisk1/signet/blocks" is exFAT which is known to have intermittent corruption problems on MacOS. See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.

to

Warning: The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: data directory ("/Volumes/sandisk1/signet"), blocks directory ("/Volumes/sandisk1/signet/blocks"). See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.

... if both datadir and blocksdir are on exfat on macOS.

tdb3 approved

tdb3 commented at 11:37 PM on January 20, 2025: contributor

code review and untested ACK df1ba101419729aafa382d970ee605e2a6273e26

I also think a GUI warning will be beneficial (and would guess is pertinent to a significant number of the users running macOS and exFAT)

While I don't have the hardware to test locally, fwiw cross compilation was successful with Ubuntu 24.04.

Cross compiling ....................... TRUE, for Darwin, aarch64
C++ compiler .......................... Clang 18.1.3, /usr/bin/clang++

DrahtBot requested review from edilmedeiros on Jan 20, 2025

DrahtBot requested review from davidgumberg on Jan 20, 2025

DrahtBot requested review from hebasto on Jan 20, 2025

DrahtBot requested review from Zero-1729 on Jan 20, 2025

DrahtBot requested review from marcofleon on Jan 20, 2025

philmb3487 commented at 1:33 PM on February 18, 2025: contributor

I don’t think we should bother the user with these gratuitous messages. Has it been demonstrated that SD cards lead to corruption? From what I understand, this issue is about Apple-specific software bugs and it should stick to this.

On Mon, Feb 17, 2025 at 22:32, l0rinc @.***(mailto:On Mon, Feb 17, 2025 at 22:32, l0rinc <<a href=)> wrote:

@l0rinc commented on this pull request.

In src/common/init.cpp:

@@ -62,6 +63,32 @@ std::optional<ConfigError> InitConfig(ArgsManager& args, SettingsAbortFn setting fs::create_directories(net_path / "wallets"); }
   // Warn if we are trying to put the datadir on an exFAT fs on MacOS
   // This is an upstream issue known to cause bugs, see [#28552](/bitcoin-bitcoin/28552/)
+#ifdef APPLE
Yes, but mostly with SSDs, not SD cards.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

sipa commented at 1:53 PM on February 18, 2025: member

@l0rinc @philmb3487 I think we shouldn't conflate these issues. From the linked #28552 issue, it appears that there is an actual demonstrable corruption that happens specifically due to exFAT on macOS, and that is what this PR is addressing. In addition, we have vague reports over the years of corruption when running Bitcoin Core on relatively cheap consumer storage (sdcards, external spinning drives, ...), but it's far less of a clear-cut case there.

philmb3487 commented at 8:08 PM on February 18, 2025: contributor

@sipa Thanks for bringing this up. Apple moved the exFat filesystem to userspace and it had nasty side effects. What a terrible idea, what is Apple even thinking? To be sure, have other drivers been moved to user-space? For all we know they might also be broken. It needs to be checked, it seems doubtful they would only move one driver over.

On Tue, Feb 18, 2025 at 21:00, l0rinc @.***(mailto:On Tue, Feb 18, 2025 at 21:00, l0rinc <<a href=)> wrote:

@l0rinc commented on this pull request.

In src/common/init.cpp:

@@ -62,6 +63,32 @@ std::optional<ConfigError> InitConfig(ArgsManager& args, SettingsAbortFn setting fs::create_directories(net_path / "wallets"); }
   // Warn if we are trying to put the datadir on an exFAT fs on MacOS
   // This is an upstream issue known to cause bugs, see [#28552](/bitcoin-bitcoin/28552/)
+#ifdef APPLE
Please resolve the comment, seems it's not strictly related to the issue

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

Zero-1729 commented at 11:55 AM on February 19, 2025: contributor

I agree with @sipa here, I think the scope and focus of this PR should remain the macOS external SSD exFAT issue which is well documented and easily testable (both the error and @willcl-ark's warning).

Once there is a similar level of replicability for the SD card or similar issue, there can be a separate follow up PR that addresses that.

luke-jr referenced this in commit d4d9bdfa49 on Feb 22, 2025

in src/common/init.cpp:99 in df1ba10141 outdated

  94 | +                    break;
  95 | +            }
  96 | +        }
  97 | +
  98 | +        if (!exfat_paths.empty()) {
  99 | +            InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: %s. "

pablomartin4btc commented at 11:07 AM on April 2, 2025:

small nit:

            InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on macOS: %s. "

willcl-ark commented at 10:08 AM on August 4, 2025:

MacOS corrected to macOS in according to #31453 (comment)

pablomartin4btc commented at 3:45 AM on August 13, 2025:

Sorry, my bad here, I should've checked properly. That's correct, Apple rebranded MacOS to macOS in 2016.

in src/util/fs_helpers.h:26 in df1ba10141 outdated

  21 | +    ERROR
  22 | +};
  23 | +
  24 | +/**
  25 | + * Detect filesystem type for a given path.
  26 | + * Currently identifies exFAT filesystems which cause issues on MacOS.

pablomartin4btc commented at 11:07 AM on April 2, 2025:

same:

 * Currently identifies exFAT filesystems which cause issues on macOS.

l0rinc commented at 12:15 AM on August 8, 2025:

+1

pablomartin4btc commented at 3:46 AM on August 13, 2025:

As per my previous comment, macOS is correct.

pablomartin4btc commented at 11:13 AM on April 2, 2025: member

Concept ACK

In case more reviewers consider -walletdir and GUI, please consider this and this author's comments respectively.

Left a tiny nit, I see was also mentioned by @Zero-1729.

philmb3487 commented at 12:48 PM on April 2, 2025: contributor

The bug is specific to macOS and exFAT. Though, exFAT is a trival filesystem meant for things like storing photos (for cameras) and other archival purposes, using it for database-heavy access patterns such as a bitcoin node is a bad idea in general.

I don't know. exFAT is the standard filesystem on Android for instance, the world's most widely used OS.

laanwj commented at 1:00 PM on April 2, 2025: member

I don't know. exFAT is the standard filesystem on Android for instance, the world's most widely used OS.

No one is preventing you from doing so, but it's still better (for reliability, and probably for performance too) to use a journaling fs.

Edit: i'm also fairly sure that Android, as a Linux kernel OS, uses ext4 by default for internal file systems, but this is off topic anyhow.

gentlee commented at 8:06 PM on April 10, 2025: none

This PR would save WEEEEKS for me, when it will be merged?

achow101 commented at 1:19 AM on April 25, 2025: member

It looks like this is happening early enough in init that the warning never actually makes it to the GUI, so GUI users (probably a majority of MacOS users) won't actually see the warning.

DrahtBot added the label Needs rebase on May 7, 2025

in doc/files.md:132 in df1ba10141 outdated

 124 | @@ -123,6 +125,12 @@ Path           | Description | Repository notes
 125 |  `addr.dat`     | Peer IP address BDB database; replaced by `peers.dat` in [0.7.0](https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.7.0.md) | [PR #1198](https://github.com/bitcoin/bitcoin/pull/1198), [`928d3a01`](https://github.com/bitcoin/bitcoin/commit/928d3a011cc66c7f907c4d053f674ea77dc611cc)
 126 |  `onion_private_key` | Cached Tor onion service private key for `-listenonion` option. Was used for Tor v2 services; replaced by `onion_v3_private_key` in [0.21.0](https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.21.0.md) | [PR #19954](https://github.com/bitcoin/bitcoin/pull/19954)
 127 |  
 128 | +## Filesystem recommendations
 129 | +
 130 | +When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`), some filesystems should be avoided:
 131 | +
 132 | +- **macOS**: The exFAT filesystem should not be used. There have been multiple reports of database and other corruption when using exFAT on macOS for Bitcoin Core. This appears to be due to filesystem-level issues with exFAT on the macOS operating system. See [Issue #31454](https://github.com/bitcoin/bitcoin/issues/31454) for more details.

maflcko commented at 3:23 PM on June 20, 2025:

Looks like this pull needs rebase, but the issue is quite common. Maybe this doc-only fixup can be split up and merged before the other code changes?

ismaelsadeeq commented at 3:33 PM on June 20, 2025:

Wouldn't it be better if they are merged together, the log message will be more helpful than the doc I think.

maflcko commented at 9:58 AM on June 21, 2025:

Sure, but it doesn't seem to work right now, according to #31453 (comment) and it also needs rebase. The doc should be trivially correct, whereas the code needs review and testing.

willcl-ark commented at 10:07 AM on August 4, 2025:

FWIW it is now working for GUI startup according to #31453 (comment) in eb68bb48ab2bb163f20dc7430d0401d88ca43f4a

ismaelsadeeq commented at 3:29 PM on June 20, 2025: member

Concept ACK on this.

This is really helpful for preventing redundant sync attempts pending when the actual issue will be addressed.

Sigh, I wouldn’t have synced past 80% three times if this was in.

willcl-ark force-pushed on Jun 21, 2025

DrahtBot removed the label Needs rebase on Jun 21, 2025

willcl-ark commented at 8:07 AM on June 23, 2025: member

@maflcko I'd be happy to split off the doc change if that's preferred (or have someone else take it), but I think that the log warning is actually the most effective part of this PR.

I rebased and moved the checks later in the startup sequence when the GUI is started, so (MacOS) GUI users will see this warning in a big pop-up box like they do with the disk space warnings.

I don't particularly like how this check is so much later in startup; I put it as early as possible before which is cleaner, but I don't know a good way to pass this message to the GUI after it's started. (I also started with it early partially because I think we should not support exFAT at all and have this InitError rather than just warn).

maflcko commented at 9:52 AM on June 23, 2025: member

InitError

seems fine too. It could be guarded by an if(!std::getenv("REALLY_USE_EXFAT")), if needed.

ismaelsadeeq commented at 3:57 PM on June 23, 2025: member

Code review 690b57560da11a3c25072d4232601bde0cfbc89d

I've tested this on macOS using a custom exFAT filesystem for:

datadir
-blocksdir
Both blocksdir and datadir

I think the current implementation is redundant because when the blocksdir is just a folder inside datadir, we emit two warnings. I think it should be a single warning instead.

I've modified the implementation to achieve this — see: https://github.com/ismaelsadeeq/bitcoin/commit/ddb02a6beb0ec892c18b43a55d53dc2daf11e555

I've also made GetFilesystemType available on all operating systems, not just macOS, as I think it could be useful.

Additionally, I've adjusted the warnings to remove the external link and instead directly include the recommended action. I believe this is clearer and eliminates the need for users to check an external link, which they may likely not do. This is also consistent with the format used for other warnings, such as when the blocks directory size is less than 750 GiB. @willcl-ark would it be possible to have a functional test for this so that it can be validated in CI i.e we dont have to rely on manual test? If so, it would be great to include it here( although the test is not a blocker though).

in src/init.cpp:1883 in 690b57560d outdated

1873 | +                break;
1874 | +        }
1875 | +    }
1876 | +
1877 | +    if (!exfat_paths.empty()) {
1878 | +        InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: %s. "

ismaelsadeeq commented at 4:02 PM on June 23, 2025:

Not just in this PR I’ve noticed InitWarning emits duplicate warnings when running bitcoind: one with a timestamp and another without.

When running Qt from the command line: one popup warning and one log entry without a timestamp.

ismaelsadeeq commented at 6:27 AM on June 25, 2025:

e.g

2025-06-25T06:13:16Z [warning] The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: blocks directory ("/Volumes/StoreJet/blocks_temp/blocks"). Move these directories to a non-exFAT formatted drive to avoid corruption.
Warning: The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: blocks directory ("/Volumes/StoreJet/blocks_temp/blocks"). Move these directories to a non-exFAT formatted drive to avoid corruption.

willcl-ark commented at 6:44 AM on June 25, 2025:

I'm pretty sure this is actually just two different warning styles both being logged to the same place by the user. I haven't tested, but this code: https://github.com/bitcoin/bitcoin/blob/ad654a4807cd584be9ffcd8640f628ab40cb5170/src/noui.cpp#L33-L46 logs one error to log (with timestamp) and when running the daemon the same error (without timestamp) to stderr.

When running daemon in the foreground, both messages appear printed in the console. But IMO this makes sense for the daemonised process to outpue a message like this not only to the logfile, but also stderr (to be read/processed by other tools).

ismaelsadeeq commented at 6:46 AM on June 25, 2025:

Makes sense, thanks for explaining.

maflcko commented at 8:28 AM on August 4, 2025:

can be resolved?

ismaelsadeeq commented at 8:50 AM on August 4, 2025:

Yes

willcl-ark commented at 4:29 PM on June 23, 2025: member

Thanks for testing.

I think the current implementation is redundant because when the blocksdir is just a folder inside datadir, we emit two warnings. I think it should be a single warning instead.

Could you provide an example of the two warnings? It should print one warning listing the two paths.

@willcl-ark would it be possible to have a functional test for this so that it can be validated in CI i.e we dont have to rely on manual test? If so, it would be great to include it here( although the test is not a blocker though).

I suppose it might be possible to add, but having a test which repartitions a drive, and creates a new exFAT volume feels icky to me! I'm happy with having this warning documented and a best-effort check for it on startup.

ismaelsadeeq commented at 6:59 PM on June 23, 2025: member

Could you provide an example of the two warnings? It should print one warning listing the two paths.

yeah you are right sorry thats what I meant :) this is the message on 690b57560da11a3c25072d4232601bde0cfbc89d

Warning: The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: data directory ("/Volumes/StoreJet/bitcoin_temp"), blocks directory ("/Volumes/StoreJet/bitcoin_temp/blocks"). See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.

No need to mention the warning for the blocksdir since it just a folder in the datadir; It's whats being checked in https://github.com/ismaelsadeeq/bitcoin/commit/ddb02a6beb0ec892c18b43a55d53dc2daf11e555

2025-06-23T18:57:40Z [warning] The specified data directory "/Volumes/StoreJet/bitcoin_temp" are on exFAT which is known to have intermittent corruption problems on macOS. Use of exFAT is discouraged. Consider restarting with different filesystem

willcl-ark force-pushed on Jun 24, 2025

willcl-ark commented at 7:57 AM on June 24, 2025: member

No need to mention the warning for the blocksdir since it just a folder in the datadir; It's whats being checked in ismaelsadeeq@ddb02a6

Ok, we now check if blocksdir is inside datadir, and removed the long URL from the warning.

in src/init.cpp:1868 in e48fe7dbef outdated

1862 | +
1863 | +    for (const auto& check : paths) {
1864 | +        FSType fs_type = GetFilesystemType(check.path);
1865 | +        switch(fs_type) {
1866 | +            case FSType::EXFAT:
1867 | +                exfat_paths.push_back(strprintf("%s (\"%s\")",

ismaelsadeeq commented at 10:55 AM on June 24, 2025:

nit: use fs::quoted here and other places, will avoid the slashes?

willcl-ark commented at 3:37 PM on June 24, 2025:

Not going to take this for now as I think I need to create and use a stream to read into? two backslashes feels easier IMO. :)

in src/util/fs_helpers.h:31 in e48fe7dbef outdated

  26 | + * Currently identifies exFAT filesystems which cause issues on MacOS.
  27 | + *
  28 | + * @param[in] path The directory path to check
  29 | + * @return FSType enum indicating the filesystem type
  30 | + */
  31 | +FSType GetFilesystemType(const fs::path& path);

ismaelsadeeq commented at 10:57 AM on June 24, 2025:

nit: would be nice to be just generic not specific to macOs

willcl-ark commented at 3:38 PM on June 24, 2025:

I agree and think this can trivially be made generic in the future if needed, but I don't see the need now?

in src/init.cpp:1848 in e48fe7dbef outdated

1840 | @@ -1841,6 +1841,54 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
1841 |          }
1842 |      }
1843 |  
1844 | +#ifdef __APPLE__
1845 | +    // Warn about exFAT filesystem usage on macOS
1846 | +    struct PathCheck {
1847 | +        fs::path path;
1848 | +        std::string_view description;

ismaelsadeeq commented at 11:08 AM on June 24, 2025:

Having a constructor here will prevent the C.I failure I think, emplace_back will forward the values to constructor which is missing now.

willcl-ark commented at 3:36 PM on June 24, 2025:

done in 2ad18fc9784e6a37e453bf844ba10c4e46f54754

ismaelsadeeq commented at 11:10 AM on June 24, 2025: member

Thanks for taking my suggestion @willcl-ark

Your modified string is even better!

Just a couple of comments and then I will re-test again.

willcl-ark force-pushed on Jun 24, 2025

in src/init.cpp:1891 in 2ad18fc978 outdated

1881 | +
1882 | +    if (!exfat_paths.empty()) {
1883 | +        InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: %s. "
1884 | +            "Move these directories to a non-exFAT formatted drive to avoid corruption."),
1885 | +            util::Join(exfat_paths, ", ")));
1886 | +    }

ismaelsadeeq commented at 6:24 AM on June 25, 2025:

nitty-nit: right now the ouput seems to be plural, i.e directories when it's one so this might be better?

if (!exfat_paths.empty()) {Add commentMore actions
        for (const auto& path : exfat_paths) {
            InitWarning(strprintf(_("The specified %s is on exFAT which is known to have intermittent corruption problems on macOS. "
                                    "Restart with non-exFAT formatted drive to avoid corruption."),
                                    path));
        }
    }

willcl-ark commented at 10:06 AM on August 4, 2025:

Going to leave this nit, as I don't think it's worth exatra logic here

ismaelsadeeq commented at 6:31 AM on June 25, 2025: member

Code review and tested ACK 2ad18fc9784e6a37e453bf844ba10c4e46f54754

bitcoind

2025-06-25T06:12:15Z [warning] The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: data directory ("/Volumes/StoreJet/bitcoin_temp"). Move these directories to a non-exFAT formatted drive to avoid corruption.

gui <img width="577" alt="Screenshot 2025-06-25 at 07 29 37" src="https://github.com/user-attachments/assets/1e1340e6-9e3e-4aed-beb6-8ab5a5eed678" />

DrahtBot requested review from tdb3 on Jun 25, 2025

DrahtBot requested review from pablomartin4btc on Jun 25, 2025

in src/init.cpp:1857 in 2ad18fc978 outdated

1852 | +
1853 | +    fs::path data_dir = args.GetDataDirNet();
1854 | +    fs::path blocks_dir = args.GetBlocksDirPath();
1855 | +    std::vector<PathCheck> paths{{data_dir, "data directory"}};
1856 | +    // Don't mention blocks dir if it's inside datadir
1857 | +    if (fs::relative(blocks_dir, data_dir / "blocks").string() != ".") {

luke-jr commented at 11:29 AM on June 27, 2025:

What if it's datadir/blocks-alternative-name or something?

in src/init.cpp:1856 in 2ad18fc978 outdated

1851 | +
1852 | +
1853 | +    fs::path data_dir = args.GetDataDirNet();
1854 | +    fs::path blocks_dir = args.GetBlocksDirPath();
1855 | +    std::vector<PathCheck> paths{{data_dir, "data directory"}};
1856 | +    // Don't mention blocks dir if it's inside datadir

luke-jr commented at 11:29 AM on June 27, 2025:

Does macOS not allow mounting subdirectories? (Maybe a bind mount) eg, maybe datadir isn't exFAT, but blocks is?

willcl-ark commented at 10:06 AM on August 4, 2025:

Agree. I removed this "optimisation" in eb68bb48ab2bb163f20dc7430d0401d88ca43f4a

We now just check both directories to avoid missing any as you describe (and if they are renamed as per #31453 (review) )

luke-jr changes_requested

luke-jr referenced this in commit 070dbfbd6d on Jul 17, 2025

ismaelsadeeq commented at 4:00 PM on August 3, 2025: member

It will be nice if this get in to v30, cc all previous reviewers @maflcko perhaps add the v30.0 milestone

maflcko added this to the milestone 30.0 on Aug 4, 2025

willcl-ark force-pushed on Aug 4, 2025

willcl-ark commented at 10:11 AM on August 4, 2025: member

While I was re-touching this for the outstanding comments I took the chance to inline the check for a smaller diff and generally a little cleaner/better code (IMO).

ismaelsadeeq approved

ismaelsadeeq commented at 7:20 PM on August 4, 2025: member

reACK eb68bb48ab2bb163f20dc7430d0401d88ca43f4a

Also retested on both bitcoin-qt and bitcoind binaries release build and I get the same warnings when datadir/blocksdir or both are on exFat drive format

in src/init.cpp:1876 in eb68bb48ab outdated

1846 | +    std::vector<std::string> exfat_paths;
1847 | +    std::vector<std::string> error_paths;
1848 | +
1849 | +    auto check_fs = [&](const fs::path& path, std::string_view desc) {
1850 | +        FSType fs_type = GetFilesystemType(path);
1851 | +        std::string path_desc = strprintf("%s (\"%s\")", desc, fs::PathToString(path));

luke-jr commented at 2:26 AM on August 5, 2025:

Kinda ugly to build this string when it's likely not going to be used.

l0rinc commented at 11:36 PM on August 7, 2025:

Agree, we could make it a lambda:

auto check_fs{[&](const fs::path& path, std::string_view desc) {
    const auto fs_type{GetFilesystemType(path)};
    const auto path_desc{[&] { return strprintf("%s (\"%s\")", desc, fs::PathToString(path)); }};
    if (fs_type == FSType::EXFAT) {
        exfat_paths.push_back(path_desc());
    } else if (fs_type == FSType::ERROR) {
        error_paths.push_back(path_desc());
    }
}};

or alternatively even use a switch to inline fs_type:

auto check_fs{[&](const fs::path& path, std::string_view desc) {
    const auto path_desc{[&] { return strprintf("%s (\"%s\")", desc, fs::PathToString(path)); }};
    switch (GetFilesystemType(path)) {
    case FSType::EXFAT: exfat_paths.push_back(path_desc()); break;
    case FSType::ERROR: error_paths.push_back(path_desc()); break;
    case FSType::OTHER: break;
    }
}};

or even simpler, to avoid all the vectors, since this is an exceptional situation in the first place, it shouldn't matter if we're getting multiple warnings for multiple folders:

auto check_and_warn_fs{[&](const fs::path& path, std::string_view desc) {
    const auto path_desc{strprintf("%s (\"%s\")", desc, fs::PathToString(path))};
    switch (GetFilesystemType(path)) {
    case FSType::EXFAT:
        InitWarning(strprintf(_("The %s path uses exFAT, which is known to have intermittent corruption problems on macOS. "
            "Move this directory to a different filesystem to avoid data loss."), path_desc));
        break;
    case FSType::ERROR:
        LogInfo("Failed to detect filesystem type for %s", path_desc);
        break;
    case FSType::OTHER:
        break;
    }
}};

check_and_warn_fs(args.GetDataDirNet(), "data directory");
check_and_warn_fs(args.GetBlocksDirPath(), "blocks directory");

luke-jr referenced this in commit e33534b061 on Aug 5, 2025

marcofleon commented at 11:08 AM on August 5, 2025: contributor

ACK eb68bb48ab2bb163f20dc7430d0401d88ca43f4a

Tested using bitcoind and the warning appears, although gets a bit lost among the output. There might be discussion I missed, but I don't see why we wouldn't just exit in this case. If the GUI warning is clear enough (as it seems like it is based on other reviews), then leaving the option to continue after the pop-up warning is probably fine. Either way, not a blocker, as having a warning is better than nothing.

DrahtBot added the label Needs rebase on Aug 7, 2025

in src/init.cpp:1869 in eb68bb48ab outdated

1864 | +            "Move these directories to a non-exFAT formatted drive to avoid corruption."),
1865 | +            util::Join(exfat_paths, ", ")));
1866 | +    }
1867 | +
1868 | +    if (!error_paths.empty()) {
1869 | +        LogInfo("Failed to detect filesystem type for: %s\n", util::Join(error_paths, ", "));

maflcko commented at 2:47 PM on August 7, 2025:

        LogInfo("Failed to detect filesystem type for: %s", util::Join(error_paths, ", "));

nit: no need for newline in new code

willcl-ark force-pushed on Aug 7, 2025

DrahtBot removed the label Needs rebase on Aug 7, 2025

in doc/files.md:170 in ac49712b7d outdated

 161 | @@ -160,3 +162,9 @@ This table describes the files installed by Bitcoin Core across different platfo
 162 |  - *Italicized* files are only installed in source builds if relevant CMake options are enabled. They are not included in binary releases.
 163 |  - README and bitcoin.conf files are included in binary releases but not installed in source builds.
 164 |  - On Windows, binaries have a `.exe` suffix (e.g., `bitcoin-cli.exe`).
 165 | +
 166 | +## Filesystem recommendations
 167 | +
 168 | +When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`), some filesystems should be avoided:
 169 | +
 170 | +- **macOS**: The exFAT filesystem should not be used. There have been multiple reports of database and other corruption when using exFAT on macOS for Bitcoin Core. This appears to be due to filesystem-level issues with exFAT on the macOS operating system. See [Issue #31454](https://github.com/bitcoin/bitcoin/issues/31454) for more details.

l0rinc commented at 11:31 PM on August 7, 2025:

Line seems to state the same thing multiple times. The previous line already states that some should be avoided:

## Filesystem recommendations

When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`), some filesystems should be avoided:

- **macOS**: The exFAT filesystem should not be used. There have been multiple reports of database corruption and data loss when using this filesystem with Bitcoin Core due to filesystem-level compatibility issues. See [Issue [#31454](/bitcoin-bitcoin/31454/)](https://github.com/bitcoin/bitcoin/issues/31454) for more details.

and I'm not sure why we're even creating a single-entry list. When new filesystems are encountered, we can generalize it:

## Filesystem recommendations

When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`) on **macOS**, the `exFAT` filesystem should be avoided.
There have been multiple reports of database corruption and data loss when using this filesystem with Bitcoin Core, see [Issue [#31454](/bitcoin-bitcoin/31454/)](https://github.com/bitcoin/bitcoin/issues/31454) for more details.

in doc/files.md:21 in ac49712b7d outdated

  17 | @@ -18,6 +18,8 @@
  18 |  
  19 |  - [Installed Files](#installed-files)
  20 |  
  21 | +- [Filesystem recommendations](#filesystem-recommendations)

l0rinc commented at 11:33 PM on August 7, 2025:

In other cases we've just removed the whole TOC - looks like this was kept since it's not technically called that, but consider removing the whole thing as an alternative.

in src/init.cpp:1870 in ac49712b7d outdated

1865 | @@ -1866,6 +1866,35 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
1866 |          }
1867 |      }
1868 |  
1869 | +#ifdef __APPLE__
1870 | +    // Warn about exFAT filesystem usage on macOS

l0rinc commented at 11:46 PM on August 7, 2025:

This is either redundant or incomplete: we should either delete it or mention the error_paths check as well

in src/init.cpp:1889 in ac49712b7d outdated

1884 | +    check_fs(args.GetDataDirNet(), "data directory");
1885 | +    check_fs(args.GetBlocksDirPath(), "blocks directory");
1886 | +
1887 | +    if (!exfat_paths.empty()) {
1888 | +        InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on macOS: %s. "
1889 | +            "Move these directories to a non-exFAT formatted drive to avoid corruption."),

l0rinc commented at 11:46 PM on August 7, 2025:

Weird negation + repetition:

            "Move these directories to a different filesystem to avoid data loss."),

in src/init.cpp:1894 in ac49712b7d outdated

1889 | +            "Move these directories to a non-exFAT formatted drive to avoid corruption."),
1890 | +            util::Join(exfat_paths, ", ")));
1891 | +    }
1892 | +
1893 | +    if (!error_paths.empty()) {
1894 | +        LogInfo("Failed to detect filesystem type for: %s", util::Join(error_paths, ", "));

l0rinc commented at 11:48 PM on August 7, 2025:

We already have using util::Join; at the top:

        LogInfo("Failed to detect filesystem type for: %s", Join(error_paths, ", "));

in src/util/fs_helpers.cpp:314 in ac49712b7d outdated

 309 | +    struct statfs fs_info;
 310 | +    if (statfs(path.c_str(), &fs_info) != 0) {
 311 | +        return FSType::ERROR;
 312 | +    }
 313 | +
 314 | +    if (strcmp(fs_info.f_fstypename, "exfat") == 0) {

l0rinc commented at 11:57 PM on August 7, 2025:

First, while I believe this to be correct, could you provide a documentation that shows the possible values? I'm wondering if it's always written like this or if it's sometimes exFAT (like you refer to it) or ExFAT (like the apple doc does sometimes) - or even if it would/should work with something like https://github.com/relan/exfat (found this accidentally, ignore if not important)

Second: nit, we don't have to keep this C-style comparison, we can also compare string_views, something like:

FSType GetFilesystemType(const fs::path& path)
{
    if (struct statfs fs_info; statfs(path.c_str(), &fs_info)) {
        return FSType::ERROR;
    } else if (std::string_view{fs_info.f_fstypename} == "exfat") {
        return FSType::EXFAT;
    }
    return FSType::OTHER;
}

willcl-ark commented at 8:25 AM on August 8, 2025:

Yeah, I also could not find such documentation when I first wrote the patch, and ended up making a simple c program to just call getfsstat on my mounted exfat disk, to get the info out to use here. I am not at home now though so can't even pass said program to you right now.

I'm wondering if it's always written like this or if it's sometimes exFAT (like you refer to it) or ExFAT (like the apple doc does sometimes) - or even if it would/should work with something like relan/exfat (found this accidentally, ignore if not important)

It's a fair question, I don't know how brittle this might be, but can't imagine that it would change much?

in src/init.cpp:1888 in ac49712b7d outdated

1883 | +
1884 | +    check_fs(args.GetDataDirNet(), "data directory");
1885 | +    check_fs(args.GetBlocksDirPath(), "blocks directory");
1886 | +
1887 | +    if (!exfat_paths.empty()) {
1888 | +        InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on macOS: %s. "

l0rinc commented at 12:11 AM on August 8, 2025:

nit: "are on exFAT" sounds weird to me + missing comma:

        InitWarning(strprintf(_("The following paths are using exFAT, which is known to have intermittent corruption problems on macOS: %s. "

in src/util/fs_helpers.cpp:308 in ac49712b7d outdated

 302 | @@ -298,3 +303,17 @@ std::optional<fs::perms> InterpretPermString(const std::string& s)
 303 |          return std::nullopt;
 304 |      }
 305 |  }
 306 | +
 307 | +#ifdef __APPLE__
 308 | +FSType GetFilesystemType(const fs::path& path) {

l0rinc commented at 12:12 AM on August 8, 2025:

nit: non-standard formatting

l0rinc changes_requested

l0rinc commented at 12:22 AM on August 8, 2025: contributor

Seeing this planned for V30 I reviewed it a bit more thoroughly. My Concept ACK remains, but I think we can document and simplify and modernize it a bit more.

<details> <summary>Suggestions patch</summary>

diff --git a/doc/files.md b/doc/files.md
index 21a73d7757..9bdb660b57 100644
--- a/doc/files.md
+++ b/doc/files.md
@@ -164,7 +164,5 @@ This table describes the files installed by Bitcoin Core across different platfo
 - On Windows, binaries have a `.exe` suffix (e.g., `bitcoin-cli.exe`).
 
 ## Filesystem recommendations
-
-When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`), some filesystems should be avoided:
-
-- **macOS**: The exFAT filesystem should not be used. There have been multiple reports of database and other corruption when using exFAT on macOS for Bitcoin Core. This appears to be due to filesystem-level issues with exFAT on the macOS operating system. See [Issue [#31454](/bitcoin-bitcoin/31454/)](https://github.com/bitcoin/bitcoin/issues/31454) for more details.
+When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`) on **macOS**,the `exFAT` filesystem should be avoided.
+There have been multiple reports of database corruption and data loss when using this filesystem with Bitcoin Core, see [Issue [#31454](/bitcoin-bitcoin/31454/)](https://github.com/bitcoin/bitcoin/issues/31454) for more details.
diff --git a/src/init.cpp b/src/init.cpp
index e136a32b49..2c2e3b77d5 100644
--- a/src/init.cpp
+++ b/src/init.cpp
@@ -1867,32 +1867,23 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
     }
 
 #ifdef __APPLE__
-    // Warn about exFAT filesystem usage on macOS
-    std::vector<std::string> exfat_paths;
-    std::vector<std::string> error_paths;
-
-    auto check_fs = [&](const fs::path& path, std::string_view desc) {
-        FSType fs_type = GetFilesystemType(path);
-        std::string path_desc = strprintf("%s (\"%s\")", desc, fs::PathToString(path));
-        if (fs_type == FSType::EXFAT) {
-            exfat_paths.push_back(path_desc);
-        } else if (fs_type == FSType::ERROR) {
-            error_paths.push_back(path_desc);
-        }
-    };
-
-    check_fs(args.GetDataDirNet(), "data directory");
-    check_fs(args.GetBlocksDirPath(), "blocks directory");
-
-    if (!exfat_paths.empty()) {
-        InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on macOS: %s. "
-            "Move these directories to a non-exFAT formatted drive to avoid corruption."),
-            util::Join(exfat_paths, ", ")));
-    }
-
-    if (!error_paths.empty()) {
-        LogInfo("Failed to detect filesystem type for: %s", util::Join(error_paths, ", "));
-    }
+    auto check_and_warn_fs{[&](const fs::path& path, std::string_view desc) {
+        const auto path_desc{strprintf("%s (\"%s\")", desc, fs::PathToString(path))};
+        switch (GetFilesystemType(path)) {
+        case FSType::EXFAT:
+            InitWarning(strprintf(_("The %s path uses exFAT, which is known to have intermittent corruption problems on macOS. "
+                "Move this directory to a different filesystem to avoid data loss."), path_desc));
+            break;
+        case FSType::ERROR:
+            LogInfo("Failed to detect filesystem type for %s", path_desc);
+            break;
+        case FSType::OTHER:
+            break;
+        }
+    }};
+
+    check_and_warn_fs(args.GetDataDirNet(), "data directory");
+    check_and_warn_fs(args.GetBlocksDirPath(), "blocks directory");
 #endif
 
 #if HAVE_SYSTEM
diff --git a/src/util/fs_helpers.cpp b/src/util/fs_helpers.cpp
index ef7cc78896..b5dc1cb332 100644
--- a/src/util/fs_helpers.cpp
+++ b/src/util/fs_helpers.cpp
@@ -305,13 +305,11 @@ std::optional<fs::perms> InterpretPermString(const std::string& s)
 }
 
 #ifdef __APPLE__
-FSType GetFilesystemType(const fs::path& path) {
-    struct statfs fs_info;
-    if (statfs(path.c_str(), &fs_info) != 0) {
+FSType GetFilesystemType(const fs::path& path)
+{
+    if (struct statfs fs_info; statfs(path.c_str(), &fs_info)) {
         return FSType::ERROR;
-    }
-
-    if (strcmp(fs_info.f_fstypename, "exfat") == 0) {
+    } else if (std::string_view{fs_info.f_fstypename} == "exfat") {
         return FSType::EXFAT;
     }
     return FSType::OTHER;
diff --git a/src/util/fs_helpers.h b/src/util/fs_helpers.h
index 7423da82a4..4d11f82809 100644
--- a/src/util/fs_helpers.h
+++ b/src/util/fs_helpers.h
@@ -23,7 +23,7 @@ enum class FSType {
 
 /**
  * Detect filesystem type for a given path.
- * Currently identifies exFAT filesystems which cause issues on MacOS.
+ * Currently identifies exFAT filesystems which cause issues on macOS.
  *
  * [@param](/bitcoin-bitcoin/contributor/param/)[in] path The directory path to check
  * [@return](/bitcoin-bitcoin/contributor/return/) FSType enum indicating the filesystem type

</details>

willcl-ark force-pushed on Aug 8, 2025

luke-jr referenced this in commit 691e886865 on Aug 8, 2025

in doc/files.md:168 in 504fe8a3e0 outdated

 161 | @@ -160,3 +162,7 @@ This table describes the files installed by Bitcoin Core across different platfo
 162 |  - *Italicized* files are only installed in source builds if relevant CMake options are enabled. They are not included in binary releases.
 163 |  - README and bitcoin.conf files are included in binary releases but not installed in source builds.
 164 |  - On Windows, binaries have a `.exe` suffix (e.g., `bitcoin-cli.exe`).
 165 | +
 166 | +## Filesystem recommendations
 167 | +When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`) on **macOS**,the `exFAT` filesystem should be avoided.

l0rinc commented at 5:22 PM on August 8, 2025:

nit, if you touch again:

## Filesystem recommendations

When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`) on **macOS**, the `exFAT` filesystem should be avoided.

(my mistake)

willcl-ark commented at 6:25 PM on August 8, 2025:

Done in db3228042b2278faf7713a6ed40cdbf8e62b8fd4

in src/util/fs_helpers.cpp:312 in 504fe8a3e0 outdated

 307 | +#ifdef __APPLE__
 308 | +FSType GetFilesystemType(const fs::path& path)
 309 | +{
 310 | +    if (struct statfs fs_info; statfs(path.c_str(), &fs_info)) {
 311 | +        return FSType::ERROR;
 312 | +    } else if (std::string_view{fs_info.f_fstypename} == "exfat") {

l0rinc commented at 5:38 PM on August 8, 2025:

Few references I found that confirm this is the correct value:

in src/util/fs_helpers.cpp:308 in 504fe8a3e0 outdated

 302 | @@ -298,3 +303,15 @@ std::optional<fs::perms> InterpretPermString(const std::string& s)
 303 |          return std::nullopt;
 304 |      }
 305 |  }
 306 | +
 307 | +#ifdef __APPLE__
 308 | +FSType GetFilesystemType(const fs::path& path)

l0rinc commented at 5:45 PM on August 8, 2025:

nit: this may still be a bit too general for the current problem, I'm fine with it as it is, but if you touch again consider:

bool IsExfatFilesystem(const std::filesystem::path& p)

Which would simplify the PR even more.

willcl-ark commented at 6:25 PM on August 8, 2025:

Going to leave this as-is because this might want to be extended in the future to other fs types if the need ever arises.

l0rinc approved

l0rinc commented at 5:47 PM on August 8, 2025: contributor

Modified:

FSType GetFilesystemType(const fs::path& path)
{
    return FSType::EXFAT;
}

it correctly logs:

Warning: The data directory ("/Users/lorinc/bitcoin/demo") path uses exFAT, which is known to have intermittent corruption problems on macOS. Move this directory to a different filesystem to avoid data loss.
Warning: The blocks directory ("/Users/lorinc/bitcoin/demo/blocks") path uses exFAT, which is known to have intermittent corruption problems on macOS. Move this directory to a different filesystem to avoid data loss.

(note the lack of timestamp)

and for:

FSType GetFilesystemType(const fs::path& path)
{
    return FSType::ERROR;
}

it logs:

2025-08-08T17:41:12Z Failed to detect filesystem type for data directory ("/Users/lorinc/bitcoin/demo")
2025-08-08T17:41:12Z Failed to detect filesystem type for blocks directory ("/Users/lorinc/bitcoin/demo/blocks")

and for the original code or:

FSType GetFilesystemType(const fs::path& path)
{
    return FSType::OTHER;
}

I'm not getting any additional warnings.

ACK 504fe8a3e0a3def67a1ed3e309992d0115cf13cf

DrahtBot requested review from ismaelsadeeq on Aug 8, 2025

util: detect and warn when using exFAT on macOS

exFAT is known to cause corruption on macOS. See #28552.

Therefore we should warn when using this fs format for either the blocks
or data directories on macOS.

Co-authored-by: l0rinc <pap.lorinc@gmail.com>

db3228042b

willcl-ark force-pushed on Aug 8, 2025

willcl-ark commented at 6:24 PM on August 8, 2025: member

Seeing this planned for V30 I reviewed it a bit more thoroughly. My Concept ACK remains, but I think we can document and simplify and modernize it a bit more. Suggestions patch

Thanks for your review @l0rinc.

Have taken your entire patch (and one fixup) and added you as co-author, thanks!

l0rinc commented at 7:00 PM on August 8, 2025: contributor

ACK db3228042b2278faf7713a6ed40cdbf8e62b8fd4

in src/init.cpp:1886 in db3228042b

1881 | +            break;
1882 | +        }
1883 | +    }};
1884 | +
1885 | +    check_and_warn_fs(args.GetDataDirNet(), "data directory");
1886 | +    check_and_warn_fs(args.GetBlocksDirPath(), "blocks directory");

achow101 commented at 9:50 PM on August 8, 2025:

This now causes two warning dialogs to appear consecutively when the blocksdir is a subdirectory of the datadir.

l0rinc commented at 9:57 PM on August 8, 2025:

Do you think that will be confusing? We don't expect this to happen all the time, having separate warnings draws more attention in my opinion.

achow101 commented at 10:06 PM on August 8, 2025:

I think it's confusing, especially since most users probably don't know that blocksdir can be specified separately.

This was also discussed earlier in the PR with the decision to have one warning: #31453 (comment)

ismaelsadeeq commented at 5:26 PM on August 10, 2025: member

reACK db3228042b2278faf7713a6ed40cdbf8e62b8fd4

Could be nice if #31453 (review) will be fixed as well, but not a blocker for me.

marcofleon commented at 12:32 PM on August 12, 2025: contributor

reACK db3228042b2278faf7713a6ed40cdbf8e62b8fd4

glozow merged this on Aug 12, 2025

glozow closed this on Aug 12, 2025

fanquake commented at 1:01 PM on August 15, 2025: member

Translations followup in #33193.