util: detect and warn when using exFAT on MacOS #31453

pull willcl-ark wants to merge 1 commits into bitcoin:master from willcl-ark:macos-exfat changing 4 files +61 −0
  1. willcl-ark commented at 10:29 am on December 10, 2024: member

    exFAT is known to cause intermittent corruption on MacOS.

    Therefore we should warn when using this fs format for either the blocks or data directories.

    See #28552 for more context.

  2. DrahtBot commented at 10:30 am on December 10, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31453.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK l0rinc, ismaelsadeeq, marcofleon
    Concept ACK laanwj, sipa, helloscoopa, hebasto, edilmedeiros, davidgumberg, Zero-1729, pablomartin4btc
    Stale ACK tdb3

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    No conflicts as of last run.

    LLM Linter (✨ experimental)

    Possible typos and grammar issues:

    • “macOS,the” -> “macOS, the” [missing space after comma]

    drahtbot_id_4_m

  3. DrahtBot added the label Utils/log/libs on Dec 10, 2024
  4. willcl-ark commented at 10:32 am on December 10, 2024: member

    I did toy with having this trigger a full on InitError, but the behaviour is intermittent and seems worst during IBD. Exiting would also likely break things for a fair few users, due to exFAT being used in cross-platform storage devices. Therefore a warning seems more apt.

    The root cause seems to be that some calls to F_FULLFSYNC fcntl silently failing on exFAT. This appears to stem from the fact that, whilst supported, F_FULLFSYNC may not be fully implemented for exFAT:

    F_FULLFSYNC Does the same thing as fsync(2) then asks the drive to flush all buffered data to the permanent storage device (arg is ignored). This is currently imple-mented implemented mented on HFS, MS-DOS (FAT), and Universal Disk Format (UDF) file systems. The operation may take quite a while to complete. Certain FireWire drives have also been known to ignore the request to flush their buffered data.

    link: https://newosxbook.com/src.jl?tree=xnu&file=/bsd/man/man2/fcntl.2 and: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fcntl.2.html#//apple_ref/doc/man/2/fcntl

    From debugging this seems to manifest from fast processing speed; if I add enough LogPrintf’s then it eventually goes away, presumably due to the reduced IBD speed. This may also indicate that users on slower (CPU) machines may suffer from this less frequently.

    We do read and write to the block files from the correct position/offsets at failure but the wrong data is found there. So this appears to be an issue at the kernel level.

    I will open a new issue with some outputs from my debugging on this.

  5. in src/common/init.cpp:81 in a7d52d023e outdated 
    76+        }};
77+        for (const auto& check : paths_to_check) {
78+            FSType fs_type = GetFilesystemType(check.path);
79+            switch(fs_type) {
80+                case FSType::EXFAT:
81+                    LogWarning("Specified %s \"%s\" is exFAT which is known to have intermittent corruption problems on MacOS.", check.description, fs::PathToString(check.path));
    maflcko commented at 10:48 am on December 10, 2024:
    Shouldn’t this be an InitWarning? Otherwise gui users (basically everyone on macos?) will never see this on their screen, unless they dig up the debug log at the right position?
    willcl-ark commented at 11:43 am on December 10, 2024:

    Thanks for catching marco. It was indeed a mistake to change this from an InitError in my first draft to LogWarning.

    Will fixup shortly

    willcl-ark commented at 10:46 am on December 11, 2024:
    Done in 83839a35f07055dd03924b5bdab46bf31df33b35
  6. in src/common/init.cpp:66 in a7d52d023e outdated 
    62@@ -62,6 +63,32 @@ std::optional<ConfigError> InitConfig(ArgsManager& args, SettingsAbortFn setting
63             fs::create_directories(net_path / "wallets");
64         }
65 
66+        // Warn if we are trying to put the datadir on an exFAT fs on MacOS
67+        // This is an upstream issue known to cause bugs, see #28552
68+#ifdef __APPLE__
    l0rinc commented at 11:15 am on December 10, 2024:
    Q: is this related to a macs or just generally using flash drives as storage?
    willcl-ark commented at 11:19 am on December 10, 2024:

    I haven’t encountered the same error using this drive to sync when formatted with APFS, so I in my opinion, no.

    I have come across comments online purporting that certain brands of nvme drives don’t play well with apple’s (internal) nvme controllers, which can also cause similar issues. But here I feel pretty certain this is an exFAT x MacOS issue, and we may well see the same issues with an internal exFAT drive.

    If anyone has a mac with an internal sata/nvme drive which they could format to exFAT and test IBD with, that would be great! (I haven’t)

    willcl-ark commented at 11:20 am on December 10, 2024:
    Oh I should add here, (and probably in #31454) that I tested this same drive using exFAT on Linux, and did not see any corruption to block 450k. By which point every MacOS test had corrupted some data and failed.
    edilmedeiros commented at 5:32 pm on January 3, 2025:
    If my memory is not failing me, I have seem only macOS related issues reporting problems with exFAT external drivers. I tried exFAT on the internal drive of an old 2016 Intel Macbook Pro (original 1GB SSD, APFS on a primary partition and exFAT on a secondary partition for Bitcoin data) and IBD failed on v26. It was about a year ago, I was trying to understand if it was a problem with the external drive I was using at the time or a software problem. Unfortunately, I can’t try that again with that computer now.
    l0rinc commented at 4:10 pm on February 16, 2025:
    Even when there isn’t any corruption, maybe it would help to warn when SD cards are used for storage
    philmb3487 commented at 2:59 pm on February 17, 2025:
    Is that really judicious? Plenty of people run Bitcoind on Raspberry Pi’s!
    l0rinc commented at 3:32 pm on February 17, 2025:
    Yes, but mostly with SSDs, not SD cards.
    l0rinc commented at 2:00 pm on February 18, 2025:
    Please resolve the comment, seems it’s not strictly related to the issue
    laanwj commented at 11:24 am on April 2, 2025:

    Oh I should add here, (and probably in #31454) that I tested this same drive using exFAT on Linux, and did not see any corruption to block 450k. By which point every MacOS test had corrupted some data and failed.

    The bug is specific to macOS and exFAT. Though, exFAT is a trival filesystem meant for things like storing photos (for cameras) and other archival purposes, using it for database-heavy access patterns such as a bitcoin node is a bad idea in general.

  7. marcofleon approved
  8. marcofleon commented at 11:35 am on December 10, 2024: contributor

    utACK a7d52d023eee6559d04e2e191f17176f2fe92c00. This happened to me earlier this year during IBD using a mac M3 and an external SSD formatted by default as exFAT. Took me a bit to figure it out, so a warning would’ve helped.

    Reformatting to APFS solved the issue for me. Would that be worth adding to the warning as a suggestion for a solution?

  9. willcl-ark commented at 11:38 am on December 10, 2024: member

    Reformatting to APFS solved the issue for me. Would that be worth adding to the warning as a suggestion for a solution?

    Yes that could be nice. Another idea that I had suggested was linking to a help page. Perhaps I could dd a section to https://github.com/bitcoin/bitcoin/blob/master/doc/files.md describing the problematic FS/OS combination and add the url to the warning to make things super-easy for users to fix themselves?

    cc @fanquake

  10. laanwj commented at 12:13 pm on December 10, 2024: member
    Concept ACK, i think this makes sense in general. exFAT is meant for storing photos, not as a mature filesystem for database workloads, so subpar reliability and performance are to be expected.
  11. l0rinc commented at 2:10 pm on December 10, 2024: contributor
    Concept ACK
  12. willcl-ark force-pushed on Dec 10, 2024
  13. sipa commented at 4:19 pm on December 10, 2024: member
    Concept ACK
  14. willcl-ark force-pushed on Dec 10, 2024
  15. DrahtBot added the label CI failed on Dec 10, 2024
  16. DrahtBot commented at 4:47 pm on December 10, 2024: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/34204028171

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  17. DrahtBot removed the label CI failed on Dec 10, 2024
  18. marcofleon commented at 11:39 am on December 11, 2024: contributor

    Nice, the warning and new addition to the docs lgtm.

    ACK ee05e730466f250e131eb92165bcdb8bab68c864

  19. DrahtBot requested review from laanwj on Dec 11, 2024
  20. DrahtBot requested review from l0rinc on Dec 11, 2024
  21. DrahtBot requested review from sipa on Dec 11, 2024
  22. helloscoopa commented at 12:09 pm on December 11, 2024: none
    Concept ACK
  23. hebasto commented at 10:36 am on December 12, 2024: member
    Concept ACK.
  24. in src/common/init.cpp:75 in ee05e73046 outdated 
    72+            std::string_view description;
73+        };
74+        std::array<PathCheck, 2> paths_to_check{{
75+            {args.GetDataDirNet(), "data directory"},
76+            {args.GetBlocksDirPath(), "blocks directory"}
77+        }};
    tdb3 commented at 2:27 am on December 15, 2024:
    Makes sense to cover blocksdir and datadir. Since the user could also set -walletdir=<dir> when starting bitcoind, do you think it would be worth warning for it as well? Much lighter disk usage than blocksdir and datadir, but if we’re doubtful of macOS’s exFAT implementation, then maybe we would want to warn the user when storing wallets with it?
    edilmedeiros commented at 5:14 pm on January 3, 2025:
    Agree with this, even though I have never encountered any problems with the wallet files on a Mac with a FAT formatted disk, only on the blocks dir.
    willcl-ark commented at 12:40 pm on January 20, 2025:

    I am not convinced that walletdir is a problem, do we have any reports of this?

    I can add it, but it looks cleanest to first refactor GetWalletDir into ArgsMan, so that we can fetch this directory here in common/init.cpp, which will make this change quite a bit bigger…

    tdb3 commented at 10:59 pm on January 20, 2025:

    Fair point. Either approach (the refactor, or deferring) sounds reasonable. In the big picture, the warnings for blocksdir and datadir add value and I don’t think walletdir warning should hold up the PR.

    I do wonder how common it actually is that someone (using macOS) is using exFAT for just the walletdir, and would therefore miss the warning. I would imagine most users would be using the default walletdir location, and could therefore see the warning as it pertains to datadir.

  25. tdb3 commented at 2:29 am on December 15, 2024: contributor
    Concept ACK Great find!
  26. in src/common/init.cpp:82 in ee05e73046 outdated 
    77+        }};
78+        for (const auto& check : paths_to_check) {
79+            FSType fs_type = GetFilesystemType(check.path);
80+            switch(fs_type) {
81+                case FSType::EXFAT:
82+                    InitWarning(strprintf(_("Specified %s \"%s\" is exFAT which is known to have intermittent corruption problems on MacOS. "
    luke-jr commented at 0:55 am on December 25, 2024:
    For GUI, maybe we should check this while they still have the opportunity to change it…
    philmb3487 commented at 8:05 am on January 1, 2025:
    just one nit, the OS is named ‘macOS’
    Zero-1729 commented at 9:23 pm on January 14, 2025:

    I’d be good to add a recommendation here, too, so the user is aware of possible immediate solutions.

    0                    InitWarning(strprintf(_("Specified %s \"%s\" is exFAT which is known to have intermittent corruption problems on MacOS, use APFS instead. "
    willcl-ark commented at 12:47 pm on January 20, 2025:

    For GUI, maybe we should check this while they still have the opportunity to change it…

    To add this check to the qt/intro.cpp flow would require duplicating some of the init checks, as well as:

    • first creating the chosen directory
    • check what FS Type the created dir is
    • if EXFAT prompt the user to change filesystems
    • either remove the just-created directories (or keep them) depending on response.
    • etc.

    If we want that kind of logic then I’m happy to add it, but not sure if we need much more than a loud InitWarning myself.

    just one nit, the OS is named ‘macOS’

    Renamed in all places

    gentlee commented at 8:02 pm on April 10, 2025:
    May be let’s not insist so much and use “, consider using APFS instead.”
  27. luke-jr commented at 0:55 am on December 25, 2024: member
    Would be ideal if we had a way to reproduce the bug, and can test if the user is actually affected, in case Apple fixes it. But this is better than nothing for now
  28. edilmedeiros commented at 5:26 pm on January 3, 2025: contributor

    Concept ACK.

    Already lost many hours trying to debug this and my conclusion so far is that the FAT driver on macOS is flawed. Also, it’s quite difficult to reproduce this reliably, IBD will fail in a different point every time. Maybe we can detect a broken block in persistent memory and restart IBD from there?

  29. davidgumberg commented at 11:36 pm on January 8, 2025: contributor

    Concept ACK

    I second the suggestion by @tdb3 above to also check -walletdir, although I have looked and have not been able to find similar reports of corruption with SQLite and exFAT drives on macOS.

    To add a little data pointing in the direction of this bug only existing on macOS, I have synced nodes with datadirs on exFAT drives over both SATA and USB on Fedora 40 and have a long-running bitcoind node that has been running off of an exFAT SATA SSD on Fedora 40 for the past ~6 months and have never experienced any of the symptoms described in #28552 or #31454.

  30. in src/util/fs_helpers.cpp:34 in ee05e73046 outdated 
    40@@ -41,6 +41,10 @@
41 #include <shlobj.h> /* For SHGetSpecialFolderPathW */
42 #endif // WIN32
43 
44+#ifdef __APPLE__
45+#include <sys/mount.h>
    luke-jr commented at 9:12 pm on January 9, 2025:

    The manpage says we need #include <sys/param.h> too

    https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/statfs.2.html

    willcl-ark commented at 12:42 pm on January 20, 2025:
    added in df1ba101419729aafa382d970ee605e2a6273e26
  31. in doc/files.md:132 in ee05e73046 outdated 
    124@@ -123,6 +125,12 @@ Path           | Description | Repository notes
125 `addr.dat`     | Peer IP address BDB database; replaced by `peers.dat` in [0.7.0](https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.7.0.md) | [PR #1198](https://github.com/bitcoin/bitcoin/pull/1198), [`928d3a01`](https://github.com/bitcoin/bitcoin/commit/928d3a011cc66c7f907c4d053f674ea77dc611cc)
126 `onion_private_key` | Cached Tor onion service private key for `-listenonion` option. Was used for Tor v2 services; replaced by `onion_v3_private_key` in [0.21.0](https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.21.0.md) | [PR #19954](https://github.com/bitcoin/bitcoin/pull/19954)
127 
128+## Filesystem recommendations
129+
130+When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`), some filesystems should be avoided:
131+
132+- **MacOS**: The exFAT filesystem should not be used. There have been multiple reports of database corruption when using exFAT on MacOS for Bitcoin Core. This appears to be due to filesystem-level issues with exFAT on MacOS. See [Issue #31454](https://github.com/bitcoin/bitcoin/issues/31454) for more details.
    Zero-1729 commented at 9:22 pm on January 14, 2025:

    It’d be good to clarify here that it is recommended to use APFS instead.

    0- **MacOS**: The exFAT filesystem should not be used, it is recommended to use the APFS filesystem instead. There have been multiple reports of database corruption when using exFAT on MacOS for Bitcoin Core. This appears to be due to filesystem-level issues with exFAT on MacOS. See [Issue [#31454](/bitcoin-bitcoin/31454/)](https://github.com/bitcoin/bitcoin/issues/31454) for more details.
    willcl-ark commented at 10:09 am on August 4, 2025:

    I’ve avoided explicit recommendation of a FS on purpose here; as far as I know exFAT is the only known-problematic FS on macOS?

    Can be convinced otherwise, but leaving as-is for now

  32. Zero-1729 commented at 9:33 pm on January 14, 2025: contributor

    Concept ACK

    I encountered this error late last year as well. Thank you for working on this. I think having the warning reported is a good measure.

    I’ve dropped a few comments above.

    I tested the patch on an M1 MBP running MacOS v15.2 with an exFAT drive (83839a35f07055dd03924b5bdab46bf31df33b35):

    0$ ./build/src/qt/bitcoin-qt --datadir=/Volumes/Little\ Test/
1
2Warning: Specified data directory "/Volumes/Little Test" is exFAT which is known to have intermittent corruption problems on MacOS. See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.
3Warning: Specified blocks directory "/Volumes/Little Test/blocks" is exFAT which is known to have intermittent corruption problems on MacOS. See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.
42025-01-14 19:45:56.627 bitcoin-qt[55978:5075996] +[IMKClient subclass]: chose IMKClient_Modern
52025-01-14 19:45:56.627 bitcoin-qt[55978:5075996] +[IMKInputSession subclass]: chose IMKInputSession_Modern
6...

    As a minor nit, I do think given this adds a whole bunch of references to macOS, it should be formatted as macOS instead of MacOS for consistency, as macOS is used more often across the repo.

    Also, I think it’ll be helpful to have the disk format warning displayed as a dialog for those running the GUI, same way we get a warning if the disk is too small to accommodate blocks:

  33. willcl-ark force-pushed on Jan 20, 2025
  34. willcl-ark commented at 1:55 pm on January 20, 2025: member

    df1ba101419729aafa382d970ee605e2a6273e26 squashes the commits together, and also removes the duplicate warning messages, instead printing a single warning with a list of violating directories, e.g. going from:

    0Warning: Specified data directory "/Volumes/sandisk1/signet" is exFAT which is known to have intermittent corruption problems on MacOS. See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.
1Warning: Specified blocks directory "/Volumes/sandisk1/signet/blocks" is exFAT which is known to have intermittent corruption problems on MacOS. See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.

    to

    0Warning: The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: data directory ("/Volumes/sandisk1/signet"), blocks directory ("/Volumes/sandisk1/signet/blocks"). See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.

    … if both datadir and blocksdir are on exfat on macOS.

  35. tdb3 approved
  36. tdb3 commented at 11:37 pm on January 20, 2025: contributor

    code review and untested ACK df1ba101419729aafa382d970ee605e2a6273e26

    I also think a GUI warning will be beneficial (and would guess is pertinent to a significant number of the users running macOS and exFAT)

    While I don’t have the hardware to test locally, fwiw cross compilation was successful with Ubuntu 24.04.

    0Cross compiling ....................... TRUE, for Darwin, aarch64
1C++ compiler .......................... Clang 18.1.3, /usr/bin/clang++
  37. DrahtBot requested review from edilmedeiros on Jan 20, 2025
  38. DrahtBot requested review from davidgumberg on Jan 20, 2025
  39. DrahtBot requested review from hebasto on Jan 20, 2025
  40. DrahtBot requested review from Zero-1729 on Jan 20, 2025
  41. DrahtBot requested review from marcofleon on Jan 20, 2025
  42. philmb3487 commented at 1:33 pm on February 18, 2025: contributor

    I don’t think we should bother the user with these gratuitous messages. Has it been demonstrated that SD cards lead to corruption? From what I understand, this issue is about Apple-specific software bugs and it should stick to this.

    On Mon, Feb 17, 2025 at 22:32, l0rinc @.***(mailto:On Mon, Feb 17, 2025 at 22:32, l0rinc «a href=)> wrote:

    @l0rinc commented on this pull request.

    In src/common/init.cpp:

    @@ -62,6 +63,32 @@ std::optional InitConfig(ArgsManager& args, SettingsAbortFn setting fs::create_directories(net_path / “wallets”); }

    •    // Warn if we are trying to put the datadir on an exFAT fs on MacOS

    •    // This is an upstream issue known to cause bugs, see [#28552](/bitcoin-bitcoin/28552/)

    +#ifdef APPLE

    Yes, but mostly with SSDs, not SD cards.

    — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

  43. sipa commented at 1:53 pm on February 18, 2025: member
    @l0rinc @philmb3487 I think we shouldn’t conflate these issues. From the linked #28552 issue, it appears that there is an actual demonstrable corruption that happens specifically due to exFAT on macOS, and that is what this PR is addressing. In addition, we have vague reports over the years of corruption when running Bitcoin Core on relatively cheap consumer storage (sdcards, external spinning drives, …), but it’s far less of a clear-cut case there.
  44. philmb3487 commented at 8:08 pm on February 18, 2025: contributor

    @sipa Thanks for bringing this up. Apple moved the exFat filesystem to userspace and it had nasty side effects. What a terrible idea, what is Apple even thinking? To be sure, have other drivers been moved to user-space? For all we know they might also be broken. It needs to be checked, it seems doubtful they would only move one driver over.

    On Tue, Feb 18, 2025 at 21:00, l0rinc @.***(mailto:On Tue, Feb 18, 2025 at 21:00, l0rinc «a href=)> wrote:

    @l0rinc commented on this pull request.

    In src/common/init.cpp:

    @@ -62,6 +63,32 @@ std::optional InitConfig(ArgsManager& args, SettingsAbortFn setting fs::create_directories(net_path / “wallets”); }

    •    // Warn if we are trying to put the datadir on an exFAT fs on MacOS

    •    // This is an upstream issue known to cause bugs, see [#28552](/bitcoin-bitcoin/28552/)

    +#ifdef APPLE

    Please resolve the comment, seems it’s not strictly related to the issue

    — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

  45. Zero-1729 commented at 11:55 am on February 19, 2025: contributor

    I agree with @sipa here, I think the scope and focus of this PR should remain the macOS external SSD exFAT issue which is well documented and easily testable (both the error and @willcl-ark’s warning).

    Once there is a similar level of replicability for the SD card or similar issue, there can be a separate follow up PR that addresses that.

  46. luke-jr referenced this in commit d4d9bdfa49 on Feb 22, 2025
  47. in src/common/init.cpp:99 in df1ba10141 outdated 
    94+                    break;
95+            }
96+        }
97+
98+        if (!exfat_paths.empty()) {
99+            InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: %s. "
    pablomartin4btc commented at 11:07 am on April 2, 2025:

    small nit:

    0            InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on macOS: %s. "
    willcl-ark commented at 10:08 am on August 4, 2025:
    MacOS corrected to macOS in according to #31453 (comment)
    pablomartin4btc commented at 3:45 am on August 13, 2025:
    Sorry, my bad here, I should’ve checked properly. That’s correct, Apple rebranded MacOS to macOS in 2016.
  48. in src/util/fs_helpers.h:26 in df1ba10141 outdated 
    21+    ERROR
22+};
23+
24+/**
25+ * Detect filesystem type for a given path.
26+ * Currently identifies exFAT filesystems which cause issues on MacOS.
    pablomartin4btc commented at 11:07 am on April 2, 2025:

    same:

    0 * Currently identifies exFAT filesystems which cause issues on macOS.
    l0rinc commented at 0:15 am on August 8, 2025:
    +1
    pablomartin4btc commented at 3:46 am on August 13, 2025:
    As per my previous comment, macOS is correct.
  49. pablomartin4btc commented at 11:13 am on April 2, 2025: member

    Concept ACK

    In case more reviewers consider -walletdir and GUI, please consider this and this author’s comments respectively.

    Left a tiny nit, I see was also mentioned by @Zero-1729.

  50. philmb3487 commented at 12:48 pm on April 2, 2025: contributor

    The bug is specific to macOS and exFAT. Though, exFAT is a trival filesystem meant for things like storing photos (for cameras) and other archival purposes, using it for database-heavy access patterns such as a bitcoin node is a bad idea in general.

    I don’t know. exFAT is the standard filesystem on Android for instance, the world’s most widely used OS.

  51. laanwj commented at 1:00 pm on April 2, 2025: member

    I don’t know. exFAT is the standard filesystem on Android for instance, the world’s most widely used OS.

    No one is preventing you from doing so, but it’s still better (for reliability, and probably for performance too) to use a journaling fs.

    Edit: i’m also fairly sure that Android, as a Linux kernel OS, uses ext4 by default for internal file systems, but this is off topic anyhow.

  52. gentlee commented at 8:06 pm on April 10, 2025: none
    This PR would save WEEEEKS for me, when it will be merged?
  53. achow101 commented at 1:19 am on April 25, 2025: member
    It looks like this is happening early enough in init that the warning never actually makes it to the GUI, so GUI users (probably a majority of MacOS users) won’t actually see the warning.
  54. DrahtBot added the label Needs rebase on May 7, 2025
  55. in doc/files.md:132 in df1ba10141 outdated 
    124@@ -123,6 +125,12 @@ Path           | Description | Repository notes
125 `addr.dat`     | Peer IP address BDB database; replaced by `peers.dat` in [0.7.0](https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.7.0.md) | [PR #1198](https://github.com/bitcoin/bitcoin/pull/1198), [`928d3a01`](https://github.com/bitcoin/bitcoin/commit/928d3a011cc66c7f907c4d053f674ea77dc611cc)
126 `onion_private_key` | Cached Tor onion service private key for `-listenonion` option. Was used for Tor v2 services; replaced by `onion_v3_private_key` in [0.21.0](https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.21.0.md) | [PR #19954](https://github.com/bitcoin/bitcoin/pull/19954)
127 
128+## Filesystem recommendations
129+
130+When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`), some filesystems should be avoided:
131+
132+- **macOS**: The exFAT filesystem should not be used. There have been multiple reports of database and other corruption when using exFAT on macOS for Bitcoin Core. This appears to be due to filesystem-level issues with exFAT on the macOS operating system. See [Issue #31454](https://github.com/bitcoin/bitcoin/issues/31454) for more details.
    maflcko commented at 3:23 pm on June 20, 2025:
    Looks like this pull needs rebase, but the issue is quite common. Maybe this doc-only fixup can be split up and merged before the other code changes?
    ismaelsadeeq commented at 3:33 pm on June 20, 2025:
    Wouldn’t it be better if they are merged together, the log message will be more helpful than the doc I think.
    maflcko commented at 9:58 am on June 21, 2025:
    Sure, but it doesn’t seem to work right now, according to #31453 (comment) and it also needs rebase. The doc should be trivially correct, whereas the code needs review and testing.
    willcl-ark commented at 10:07 am on August 4, 2025:
    FWIW it is now working for GUI startup according to #31453 (comment) in eb68bb48ab2bb163f20dc7430d0401d88ca43f4a
  56. ismaelsadeeq commented at 3:29 pm on June 20, 2025: member

    Concept ACK on this.

    This is really helpful for preventing redundant sync attempts pending when the actual issue will be addressed.

    Sigh, I wouldn’t have synced past 80% three times if this was in.

  57. willcl-ark force-pushed on Jun 21, 2025
  58. DrahtBot removed the label Needs rebase on Jun 21, 2025
  59. willcl-ark commented at 8:07 am on June 23, 2025: member

    @maflcko I’d be happy to split off the doc change if that’s preferred (or have someone else take it), but I think that the log warning is actually the most effective part of this PR.

    I rebased and moved the checks later in the startup sequence when the GUI is started, so (MacOS) GUI users will see this warning in a big pop-up box like they do with the disk space warnings.

    I don’t particularly like how this check is so much later in startup; I put it as early as possible before which is cleaner, but I don’t know a good way to pass this message to the GUI after it’s started. (I also started with it early partially because I think we should not support exFAT at all and have this InitError rather than just warn).

  60. maflcko commented at 9:52 am on June 23, 2025: member

    InitError

    seems fine too. It could be guarded by an if(!std::getenv("REALLY_USE_EXFAT")), if needed.

  61. ismaelsadeeq commented at 3:57 pm on June 23, 2025: member

    Code review 690b57560da11a3c25072d4232601bde0cfbc89d

    I’ve tested this on macOS using a custom exFAT filesystem for:

    1. datadir
    2. -blocksdir
    3. Both blocksdir and datadir

    I think the current implementation is redundant because when the blocksdir is just a folder inside datadir, we emit two warnings. I think it should be a single warning instead.

    I’ve modified the implementation to achieve this — see: https://github.com/ismaelsadeeq/bitcoin/commit/ddb02a6beb0ec892c18b43a55d53dc2daf11e555

    I’ve also made GetFilesystemType available on all operating systems, not just macOS, as I think it could be useful.

    Additionally, I’ve adjusted the warnings to remove the external link and instead directly include the recommended action. I believe this is clearer and eliminates the need for users to check an external link, which they may likely not do. This is also consistent with the format used for other warnings, such as when the blocks directory size is less than 750 GiB. @willcl-ark would it be possible to have a functional test for this so that it can be validated in CI i.e we dont have to rely on manual test? If so, it would be great to include it here( although the test is not a blocker though).

  62. in src/init.cpp:1883 in 690b57560d outdated 
    1873+                break;
1874+        }
1875+    }
1876+
1877+    if (!exfat_paths.empty()) {
1878+        InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: %s. "
    ismaelsadeeq commented at 4:02 pm on June 23, 2025:

    Not just in this PR I’ve noticed InitWarning emits duplicate warnings when running bitcoind: one with a timestamp and another without.

    When running Qt from the command line: one popup warning and one log entry without a timestamp.

    ismaelsadeeq commented at 6:27 am on June 25, 2025:

    e.g

    02025-06-25T06:13:16Z [warning] The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: blocks directory ("/Volumes/StoreJet/blocks_temp/blocks"). Move these directories to a non-exFAT formatted drive to avoid corruption.
1Warning: The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: blocks directory ("/Volumes/StoreJet/blocks_temp/blocks"). Move these directories to a non-exFAT formatted drive to avoid corruption.
    willcl-ark commented at 6:44 am on June 25, 2025:

    I’m pretty sure this is actually just two different warning styles both being logged to the same place by the user. I haven’t tested, but this code: https://github.com/bitcoin/bitcoin/blob/ad654a4807cd584be9ffcd8640f628ab40cb5170/src/noui.cpp#L33-L46 logs one error to log (with timestamp) and when running the daemon the same error (without timestamp) to stderr.

    When running daemon in the foreground, both messages appear printed in the console. But IMO this makes sense for the daemonised process to outpue a message like this not only to the logfile, but also stderr (to be read/processed by other tools).

    ismaelsadeeq commented at 6:46 am on June 25, 2025:
    Makes sense, thanks for explaining.
    maflcko commented at 8:28 am on August 4, 2025:
    can be resolved?
    ismaelsadeeq commented at 8:50 am on August 4, 2025:
    Yes
  63. willcl-ark commented at 4:29 pm on June 23, 2025: member

    Thanks for testing.

    I think the current implementation is redundant because when the blocksdir is just a folder inside datadir, we emit two warnings. I think it should be a single warning instead.

    Could you provide an example of the two warnings? It should print one warning listing the two paths.

    @willcl-ark would it be possible to have a functional test for this so that it can be validated in CI i.e we dont have to rely on manual test? If so, it would be great to include it here( although the test is not a blocker though).

    I suppose it might be possible to add, but having a test which repartitions a drive, and creates a new exFAT volume feels icky to me! I’m happy with having this warning documented and a best-effort check for it on startup.

  64. ismaelsadeeq commented at 6:59 pm on June 23, 2025: member

    Could you provide an example of the two warnings? It should print one warning listing the two paths.

    yeah you are right sorry thats what I meant :) this is the message on 690b57560da11a3c25072d4232601bde0cfbc89d

    0Warning: The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: data directory ("/Volumes/StoreJet/bitcoin_temp"), blocks directory ("/Volumes/StoreJet/bitcoin_temp/blocks"). See https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#filesystem-recommendations for more information.

    No need to mention the warning for the blocksdir since it just a folder in the datadir; It’s whats being checked in https://github.com/ismaelsadeeq/bitcoin/commit/ddb02a6beb0ec892c18b43a55d53dc2daf11e555

    02025-06-23T18:57:40Z [warning] The specified data directory "/Volumes/StoreJet/bitcoin_temp" are on exFAT which is known to have intermittent corruption problems on macOS. Use of exFAT is discouraged. Consider restarting with different filesystem
  65. willcl-ark force-pushed on Jun 24, 2025
  66. willcl-ark commented at 7:57 am on June 24, 2025: member

    No need to mention the warning for the blocksdir since it just a folder in the datadir; It’s whats being checked in ismaelsadeeq@ddb02a6

    Ok, we now check if blocksdir is inside datadir, and removed the long URL from the warning.

  67. in src/init.cpp:1868 in e48fe7dbef outdated 
    1862+
1863+    for (const auto& check : paths) {
1864+        FSType fs_type = GetFilesystemType(check.path);
1865+        switch(fs_type) {
1866+            case FSType::EXFAT:
1867+                exfat_paths.push_back(strprintf("%s (\"%s\")",
    ismaelsadeeq commented at 10:55 am on June 24, 2025:
    nit: use fs::quoted here and other places, will avoid the slashes?
    willcl-ark commented at 3:37 pm on June 24, 2025:
    Not going to take this for now as I think I need to create and use a stream to read into? two backslashes feels easier IMO. :)
  68. in src/util/fs_helpers.h:31 in e48fe7dbef outdated 
    26+ * Currently identifies exFAT filesystems which cause issues on MacOS.
27+ *
28+ * @param[in] path The directory path to check
29+ * @return FSType enum indicating the filesystem type
30+ */
31+FSType GetFilesystemType(const fs::path& path);
    ismaelsadeeq commented at 10:57 am on June 24, 2025:
    nit: would be nice to be just generic not specific to macOs
    willcl-ark commented at 3:38 pm on June 24, 2025:
    I agree and think this can trivially be made generic in the future if needed, but I don’t see the need now?
  69. in src/init.cpp:1848 in e48fe7dbef outdated 
    1840@@ -1841,6 +1841,54 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
1841         }
1842     }
1843 
1844+#ifdef __APPLE__
1845+    // Warn about exFAT filesystem usage on macOS
1846+    struct PathCheck {
1847+        fs::path path;
1848+        std::string_view description;
    ismaelsadeeq commented at 11:08 am on June 24, 2025:
    Having a constructor here will prevent the C.I failure I think, emplace_back will forward the values to constructor which is missing now.
    willcl-ark commented at 3:36 pm on June 24, 2025:
    done in 2ad18fc9784e6a37e453bf844ba10c4e46f54754
  70. ismaelsadeeq commented at 11:10 am on June 24, 2025: member

    Thanks for taking my suggestion @willcl-ark

    Your modified string is even better!

    Just a couple of comments and then I will re-test again.

  71. willcl-ark force-pushed on Jun 24, 2025
  72. in src/init.cpp:1891 in 2ad18fc978 outdated 
    1881+
1882+    if (!exfat_paths.empty()) {
1883+        InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: %s. "
1884+            "Move these directories to a non-exFAT formatted drive to avoid corruption."),
1885+            util::Join(exfat_paths, ", ")));
1886+    }
    ismaelsadeeq commented at 6:24 am on June 25, 2025:

    nitty-nit: right now the ouput seems to be plural, i.e directories when it’s one so this might be better?

    0if (!exfat_paths.empty()) {Add commentMore actions
1        for (const auto& path : exfat_paths) {
2            InitWarning(strprintf(_("The specified %s is on exFAT which is known to have intermittent corruption problems on macOS. "
3                                    "Restart with non-exFAT formatted drive to avoid corruption."),
4                                    path));
5        }
6    }
    willcl-ark commented at 10:06 am on August 4, 2025:
    Going to leave this nit, as I don’t think it’s worth exatra logic here
  73. ismaelsadeeq commented at 6:31 am on June 25, 2025: member

    Code review and tested ACK 2ad18fc9784e6a37e453bf844ba10c4e46f54754

    bitcoind

    02025-06-25T06:12:15Z [warning] The following paths are on exFAT which is known to have intermittent corruption problems on MacOS: data directory ("/Volumes/StoreJet/bitcoin_temp"). Move these directories to a non-exFAT formatted drive to avoid corruption.

    gui

  74. DrahtBot requested review from tdb3 on Jun 25, 2025
  75. DrahtBot requested review from pablomartin4btc on Jun 25, 2025
  76. in src/init.cpp:1857 in 2ad18fc978 outdated 
    1852+
1853+    fs::path data_dir = args.GetDataDirNet();
1854+    fs::path blocks_dir = args.GetBlocksDirPath();
1855+    std::vector<PathCheck> paths{{data_dir, "data directory"}};
1856+    // Don't mention blocks dir if it's inside datadir
1857+    if (fs::relative(blocks_dir, data_dir / "blocks").string() != ".") {
    luke-jr commented at 11:29 am on June 27, 2025:
    What if it’s datadir/blocks-alternative-name or something?
  77. in src/init.cpp:1856 in 2ad18fc978 outdated 
    1851+
1852+
1853+    fs::path data_dir = args.GetDataDirNet();
1854+    fs::path blocks_dir = args.GetBlocksDirPath();
1855+    std::vector<PathCheck> paths{{data_dir, "data directory"}};
1856+    // Don't mention blocks dir if it's inside datadir
    luke-jr commented at 11:29 am on June 27, 2025:
    Does macOS not allow mounting subdirectories? (Maybe a bind mount) eg, maybe datadir isn’t exFAT, but blocks is?
    willcl-ark commented at 10:06 am on August 4, 2025:

    Agree. I removed this “optimisation” in eb68bb48ab2bb163f20dc7430d0401d88ca43f4a

    We now just check both directories to avoid missing any as you describe (and if they are renamed as per #31453 (review) )

  78. luke-jr changes_requested
  79. luke-jr referenced this in commit 070dbfbd6d on Jul 17, 2025
  80. ismaelsadeeq commented at 4:00 pm on August 3, 2025: member
    It will be nice if this get in to v30, cc all previous reviewers @maflcko perhaps add the v30.0 milestone
  81. maflcko added this to the milestone 30.0 on Aug 4, 2025
  82. willcl-ark force-pushed on Aug 4, 2025
  83. willcl-ark commented at 10:11 am on August 4, 2025: member
    While I was re-touching this for the outstanding comments I took the chance to inline the check for a smaller diff and generally a little cleaner/better code (IMO).
  84. ismaelsadeeq approved
  85. ismaelsadeeq commented at 7:20 pm on August 4, 2025: member

    reACK eb68bb48ab2bb163f20dc7430d0401d88ca43f4a

    Also retested on both bitcoin-qt and bitcoind binaries release build and I get the same warnings when datadir/blocksdir or both are on exFat drive format

  86. in src/init.cpp:1876 in eb68bb48ab outdated 
    1846+    std::vector<std::string> exfat_paths;
1847+    std::vector<std::string> error_paths;
1848+
1849+    auto check_fs = [&](const fs::path& path, std::string_view desc) {
1850+        FSType fs_type = GetFilesystemType(path);
1851+        std::string path_desc = strprintf("%s (\"%s\")", desc, fs::PathToString(path));
    luke-jr commented at 2:26 am on August 5, 2025:
    Kinda ugly to build this string when it’s likely not going to be used.
    l0rinc commented at 11:36 pm on August 7, 2025:

    Agree, we could make it a lambda:

    0auto check_fs{[&](const fs::path& path, std::string_view desc) {
1    const auto fs_type{GetFilesystemType(path)};
2    const auto path_desc{[&] { return strprintf("%s (\"%s\")", desc, fs::PathToString(path)); }};
3    if (fs_type == FSType::EXFAT) {
4        exfat_paths.push_back(path_desc());
5    } else if (fs_type == FSType::ERROR) {
6        error_paths.push_back(path_desc());
7    }
8}};

    or alternatively even use a switch to inline fs_type:

    0auto check_fs{[&](const fs::path& path, std::string_view desc) {
1    const auto path_desc{[&] { return strprintf("%s (\"%s\")", desc, fs::PathToString(path)); }};
2    switch (GetFilesystemType(path)) {
3    case FSType::EXFAT: exfat_paths.push_back(path_desc()); break;
4    case FSType::ERROR: error_paths.push_back(path_desc()); break;
5    case FSType::OTHER: break;
6    }
7}};

    or even simpler, to avoid all the vectors, since this is an exceptional situation in the first place, it shouldn’t matter if we’re getting multiple warnings for multiple folders:

     0auto check_and_warn_fs{[&](const fs::path& path, std::string_view desc) {
 1    const auto path_desc{strprintf("%s (\"%s\")", desc, fs::PathToString(path))};
 2    switch (GetFilesystemType(path)) {
 3    case FSType::EXFAT:
 4        InitWarning(strprintf(_("The %s path uses exFAT, which is known to have intermittent corruption problems on macOS. "
 5            "Move this directory to a different filesystem to avoid data loss."), path_desc));
 6        break;
 7    case FSType::ERROR:
 8        LogInfo("Failed to detect filesystem type for %s", path_desc);
 9        break;
10    case FSType::OTHER:
11        break;
12    }
13}};
14
15check_and_warn_fs(args.GetDataDirNet(), "data directory");
16check_and_warn_fs(args.GetBlocksDirPath(), "blocks directory");
  87. luke-jr referenced this in commit e33534b061 on Aug 5, 2025
  88. marcofleon commented at 11:08 am on August 5, 2025: contributor

    ACK eb68bb48ab2bb163f20dc7430d0401d88ca43f4a

    Tested using bitcoind and the warning appears, although gets a bit lost among the output. There might be discussion I missed, but I don’t see why we wouldn’t just exit in this case. If the GUI warning is clear enough (as it seems like it is based on other reviews), then leaving the option to continue after the pop-up warning is probably fine. Either way, not a blocker, as having a warning is better than nothing.

  89. DrahtBot added the label Needs rebase on Aug 7, 2025
  90. in src/init.cpp:1869 in eb68bb48ab outdated 
    1864+            "Move these directories to a non-exFAT formatted drive to avoid corruption."),
1865+            util::Join(exfat_paths, ", ")));
1866+    }
1867+
1868+    if (!error_paths.empty()) {
1869+        LogInfo("Failed to detect filesystem type for: %s\n", util::Join(error_paths, ", "));
    maflcko commented at 2:47 pm on August 7, 2025: 
    0        LogInfo("Failed to detect filesystem type for: %s", util::Join(error_paths, ", "));

    nit: no need for newline in new code

  91. willcl-ark force-pushed on Aug 7, 2025
  92. DrahtBot removed the label Needs rebase on Aug 7, 2025
  93. in doc/files.md:170 in ac49712b7d outdated 
    161@@ -160,3 +162,9 @@ This table describes the files installed by Bitcoin Core across different platfo
162 - *Italicized* files are only installed in source builds if relevant CMake options are enabled. They are not included in binary releases.
163 - README and bitcoin.conf files are included in binary releases but not installed in source builds.
164 - On Windows, binaries have a `.exe` suffix (e.g., `bitcoin-cli.exe`).
165+
166+## Filesystem recommendations
167+
168+When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`), some filesystems should be avoided:
169+
170+- **macOS**: The exFAT filesystem should not be used. There have been multiple reports of database and other corruption when using exFAT on macOS for Bitcoin Core. This appears to be due to filesystem-level issues with exFAT on the macOS operating system. See [Issue #31454](https://github.com/bitcoin/bitcoin/issues/31454) for more details.
    l0rinc commented at 11:31 pm on August 7, 2025:

    Line seems to state the same thing multiple times. The previous line already states that some should be avoided:

    0## Filesystem recommendations
1
2When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`), some filesystems should be avoided:
3
4- **macOS**: The exFAT filesystem should not be used. There have been multiple reports of database corruption and data loss when using this filesystem with Bitcoin Core due to filesystem-level compatibility issues. See [Issue [#31454](/bitcoin-bitcoin/31454/)](https://github.com/bitcoin/bitcoin/issues/31454) for more details.

    and I’m not sure why we’re even creating a single-entry list. When new filesystems are encountered, we can generalize it:

    0## Filesystem recommendations
1
2When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`) on **macOS**, the `exFAT` filesystem should be avoided.
3There have been multiple reports of database corruption and data loss when using this filesystem with Bitcoin Core, see [Issue [#31454](/bitcoin-bitcoin/31454/)](https://github.com/bitcoin/bitcoin/issues/31454) for more details.
  94. in doc/files.md:21 in ac49712b7d outdated 
    17@@ -18,6 +18,8 @@
18 
19 - [Installed Files](#installed-files)
20 
21+- [Filesystem recommendations](#filesystem-recommendations)
    l0rinc commented at 11:33 pm on August 7, 2025:
    In other cases we’ve just removed the whole TOC - looks like this was kept since it’s not technically called that, but consider removing the whole thing as an alternative.
  95. in src/init.cpp:1870 in ac49712b7d outdated 
    1865@@ -1866,6 +1866,35 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
1866         }
1867     }
1868 
1869+#ifdef __APPLE__
1870+    // Warn about exFAT filesystem usage on macOS
    l0rinc commented at 11:46 pm on August 7, 2025:
    This is either redundant or incomplete: we should either delete it or mention the error_paths check as well
  96. in src/init.cpp:1889 in ac49712b7d outdated 
    1884+    check_fs(args.GetDataDirNet(), "data directory");
1885+    check_fs(args.GetBlocksDirPath(), "blocks directory");
1886+
1887+    if (!exfat_paths.empty()) {
1888+        InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on macOS: %s. "
1889+            "Move these directories to a non-exFAT formatted drive to avoid corruption."),
    l0rinc commented at 11:46 pm on August 7, 2025:

    Weird negation + repetition:

    0            "Move these directories to a different filesystem to avoid data loss."),
  97. in src/init.cpp:1894 in ac49712b7d outdated 
    1889+            "Move these directories to a non-exFAT formatted drive to avoid corruption."),
1890+            util::Join(exfat_paths, ", ")));
1891+    }
1892+
1893+    if (!error_paths.empty()) {
1894+        LogInfo("Failed to detect filesystem type for: %s", util::Join(error_paths, ", "));
    l0rinc commented at 11:48 pm on August 7, 2025:

    We already have using util::Join; at the top:

    0        LogInfo("Failed to detect filesystem type for: %s", Join(error_paths, ", "));
  98. in src/util/fs_helpers.cpp:314 in ac49712b7d outdated 
    309+    struct statfs fs_info;
310+    if (statfs(path.c_str(), &fs_info) != 0) {
311+        return FSType::ERROR;
312+    }
313+
314+    if (strcmp(fs_info.f_fstypename, "exfat") == 0) {
    l0rinc commented at 11:57 pm on August 7, 2025:

    First, while I believe this to be correct, could you provide a documentation that shows the possible values? I’m wondering if it’s always written like this or if it’s sometimes exFAT (like you refer to it) or ExFAT (like the apple doc does sometimes) - or even if it would/should work with something like https://github.com/relan/exfat (found this accidentally, ignore if not important)

    Second: nit, we don’t have to keep this C-style comparison, we can also compare string_views, something like:

    0FSType GetFilesystemType(const fs::path& path)
1{
2    if (struct statfs fs_info; statfs(path.c_str(), &fs_info)) {
3        return FSType::ERROR;
4    } else if (std::string_view{fs_info.f_fstypename} == "exfat") {
5        return FSType::EXFAT;
6    }
7    return FSType::OTHER;
8}
    willcl-ark commented at 8:25 am on August 8, 2025:

    Yeah, I also could not find such documentation when I first wrote the patch, and ended up making a simple c program to just call getfsstat on my mounted exfat disk, to get the info out to use here. I am not at home now though so can’t even pass said program to you right now.

    I’m wondering if it’s always written like this or if it’s sometimes exFAT (like you refer to it) or ExFAT (like the apple doc does sometimes) - or even if it would/should work with something like relan/exfat (found this accidentally, ignore if not important)

    It’s a fair question, I don’t know how brittle this might be, but can’t imagine that it would change much?

  99. in src/init.cpp:1888 in ac49712b7d outdated 
    1883+
1884+    check_fs(args.GetDataDirNet(), "data directory");
1885+    check_fs(args.GetBlocksDirPath(), "blocks directory");
1886+
1887+    if (!exfat_paths.empty()) {
1888+        InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on macOS: %s. "
    l0rinc commented at 0:11 am on August 8, 2025:

    nit: “are on exFAT” sounds weird to me + missing comma:

    0        InitWarning(strprintf(_("The following paths are using exFAT, which is known to have intermittent corruption problems on macOS: %s. "
  100. in src/util/fs_helpers.cpp:308 in ac49712b7d outdated 
    302@@ -298,3 +303,17 @@ std::optional<fs::perms> InterpretPermString(const std::string& s)
303         return std::nullopt;
304     }
305 }
306+
307+#ifdef __APPLE__
308+FSType GetFilesystemType(const fs::path& path) {
    l0rinc commented at 0:12 am on August 8, 2025:
    nit: non-standard formatting
  101. l0rinc changes_requested
  102. l0rinc commented at 0:22 am on August 8, 2025: contributor

    Seeing this planned for V30 I reviewed it a bit more thoroughly. My Concept ACK remains, but I think we can document and simplify and modernize it a bit more.

      0diff --git a/doc/files.md b/doc/files.md
  1index 21a73d7757..9bdb660b57 100644
  2--- a/doc/files.md
  3+++ b/doc/files.md
  4@@ -164,7 +164,5 @@ This table describes the files installed by Bitcoin Core across different platfo
  5 - On Windows, binaries have a `.exe` suffix (e.g., `bitcoin-cli.exe`).
  6 
  7 ## Filesystem recommendations
  8-
  9-When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`), some filesystems should be avoided:
 10-
 11-- **macOS**: The exFAT filesystem should not be used. There have been multiple reports of database and other corruption when using exFAT on macOS for Bitcoin Core. This appears to be due to filesystem-level issues with exFAT on the macOS operating system. See [Issue [#31454](/bitcoin-bitcoin/31454/)](https://github.com/bitcoin/bitcoin/issues/31454) for more details.
 12+When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`) on **macOS**,the `exFAT` filesystem should be avoided.
 13+There have been multiple reports of database corruption and data loss when using this filesystem with Bitcoin Core, see [Issue [#31454](/bitcoin-bitcoin/31454/)](https://github.com/bitcoin/bitcoin/issues/31454) for more details.
 14diff --git a/src/init.cpp b/src/init.cpp
 15index e136a32b49..2c2e3b77d5 100644
 16--- a/src/init.cpp
 17+++ b/src/init.cpp
 18@@ -1867,32 +1867,23 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
 19     }
 20 
 21 #ifdef __APPLE__
 22-    // Warn about exFAT filesystem usage on macOS
 23-    std::vector<std::string> exfat_paths;
 24-    std::vector<std::string> error_paths;
 25-
 26-    auto check_fs = [&](const fs::path& path, std::string_view desc) {
 27-        FSType fs_type = GetFilesystemType(path);
 28-        std::string path_desc = strprintf("%s (\"%s\")", desc, fs::PathToString(path));
 29-        if (fs_type == FSType::EXFAT) {
 30-            exfat_paths.push_back(path_desc);
 31-        } else if (fs_type == FSType::ERROR) {
 32-            error_paths.push_back(path_desc);
 33-        }
 34-    };
 35-
 36-    check_fs(args.GetDataDirNet(), "data directory");
 37-    check_fs(args.GetBlocksDirPath(), "blocks directory");
 38-
 39-    if (!exfat_paths.empty()) {
 40-        InitWarning(strprintf(_("The following paths are on exFAT which is known to have intermittent corruption problems on macOS: %s. "
 41-            "Move these directories to a non-exFAT formatted drive to avoid corruption."),
 42-            util::Join(exfat_paths, ", ")));
 43-    }
 44-
 45-    if (!error_paths.empty()) {
 46-        LogInfo("Failed to detect filesystem type for: %s", util::Join(error_paths, ", "));
 47-    }
 48+    auto check_and_warn_fs{[&](const fs::path& path, std::string_view desc) {
 49+        const auto path_desc{strprintf("%s (\"%s\")", desc, fs::PathToString(path))};
 50+        switch (GetFilesystemType(path)) {
 51+        case FSType::EXFAT:
 52+            InitWarning(strprintf(_("The %s path uses exFAT, which is known to have intermittent corruption problems on macOS. "
 53+                "Move this directory to a different filesystem to avoid data loss."), path_desc));
 54+            break;
 55+        case FSType::ERROR:
 56+            LogInfo("Failed to detect filesystem type for %s", path_desc);
 57+            break;
 58+        case FSType::OTHER:
 59+            break;
 60+        }
 61+    }};
 62+
 63+    check_and_warn_fs(args.GetDataDirNet(), "data directory");
 64+    check_and_warn_fs(args.GetBlocksDirPath(), "blocks directory");
 65 #endif
 66 
 67 #if HAVE_SYSTEM
 68diff --git a/src/util/fs_helpers.cpp b/src/util/fs_helpers.cpp
 69index ef7cc78896..b5dc1cb332 100644
 70--- a/src/util/fs_helpers.cpp
 71+++ b/src/util/fs_helpers.cpp
 72@@ -305,13 +305,11 @@ std::optional<fs::perms> InterpretPermString(const std::string& s)
 73 }
 74 
 75 #ifdef __APPLE__
 76-FSType GetFilesystemType(const fs::path& path) {
 77-    struct statfs fs_info;
 78-    if (statfs(path.c_str(), &fs_info) != 0) {
 79+FSType GetFilesystemType(const fs::path& path)
 80+{
 81+    if (struct statfs fs_info; statfs(path.c_str(), &fs_info)) {
 82         return FSType::ERROR;
 83-    }
 84-
 85-    if (strcmp(fs_info.f_fstypename, "exfat") == 0) {
 86+    } else if (std::string_view{fs_info.f_fstypename} == "exfat") {
 87         return FSType::EXFAT;
 88     }
 89     return FSType::OTHER;
 90diff --git a/src/util/fs_helpers.h b/src/util/fs_helpers.h
 91index 7423da82a4..4d11f82809 100644
 92--- a/src/util/fs_helpers.h
 93+++ b/src/util/fs_helpers.h
 94@@ -23,7 +23,7 @@ enum class FSType {
 95 
 96 /**
 97  * Detect filesystem type for a given path.
 98- * Currently identifies exFAT filesystems which cause issues on MacOS.
 99+ * Currently identifies exFAT filesystems which cause issues on macOS.
100  *
101  * [@param](/bitcoin-bitcoin/contributor/param/)[in] path The directory path to check
102  * [@return](/bitcoin-bitcoin/contributor/return/) FSType enum indicating the filesystem type
  103. willcl-ark force-pushed on Aug 8, 2025
  104. luke-jr referenced this in commit 691e886865 on Aug 8, 2025
  105. in doc/files.md:168 in 504fe8a3e0 outdated 
    161@@ -160,3 +162,7 @@ This table describes the files installed by Bitcoin Core across different platfo
162 - *Italicized* files are only installed in source builds if relevant CMake options are enabled. They are not included in binary releases.
163 - README and bitcoin.conf files are included in binary releases but not installed in source builds.
164 - On Windows, binaries have a `.exe` suffix (e.g., `bitcoin-cli.exe`).
165+
166+## Filesystem recommendations
167+When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`) on **macOS**,the `exFAT` filesystem should be avoided.
    l0rinc commented at 5:22 pm on August 8, 2025:

    nit, if you touch again:

    0## Filesystem recommendations
1
2When choosing a filesystem for the data directory (`datadir`) or blocks directory (`blocksdir`) on **macOS**, the `exFAT` filesystem should be avoided.

    (my mistake)

    willcl-ark commented at 6:25 pm on August 8, 2025:
    Done in db3228042b2278faf7713a6ed40cdbf8e62b8fd4
  106. in src/util/fs_helpers.cpp:312 in 504fe8a3e0 outdated 
    307+#ifdef __APPLE__
308+FSType GetFilesystemType(const fs::path& path)
309+{
310+    if (struct statfs fs_info; statfs(path.c_str(), &fs_info)) {
311+        return FSType::ERROR;
312+    } else if (std::string_view{fs_info.f_fstypename} == "exfat") {
  107. in src/util/fs_helpers.cpp:308 in 504fe8a3e0 outdated 
    302@@ -298,3 +303,15 @@ std::optional<fs::perms> InterpretPermString(const std::string& s)
303         return std::nullopt;
304     }
305 }
306+
307+#ifdef __APPLE__
308+FSType GetFilesystemType(const fs::path& path)
    l0rinc commented at 5:45 pm on August 8, 2025:

    nit: this may still be a bit too general for the current problem, I’m fine with it as it is, but if you touch again consider:

    0bool IsExfatFilesystem(const std::filesystem::path& p)

    Which would simplify the PR even more.

    willcl-ark commented at 6:25 pm on August 8, 2025:
    Going to leave this as-is because this might want to be extended in the future to other fs types if the need ever arises.
  108. l0rinc approved
  109. l0rinc commented at 5:47 pm on August 8, 2025: contributor

    Modified:

    0FSType GetFilesystemType(const fs::path& path)
1{
2    return FSType::EXFAT;
3}

    it correctly logs:

    0Warning: The data directory ("/Users/lorinc/bitcoin/demo") path uses exFAT, which is known to have intermittent corruption problems on macOS. Move this directory to a different filesystem to avoid data loss.
1Warning: The blocks directory ("/Users/lorinc/bitcoin/demo/blocks") path uses exFAT, which is known to have intermittent corruption problems on macOS. Move this directory to a different filesystem to avoid data loss.

    (note the lack of timestamp)

    and for:

    0FSType GetFilesystemType(const fs::path& path)
1{
2    return FSType::ERROR;
3}

    it logs:

    02025-08-08T17:41:12Z Failed to detect filesystem type for data directory ("/Users/lorinc/bitcoin/demo")
12025-08-08T17:41:12Z Failed to detect filesystem type for blocks directory ("/Users/lorinc/bitcoin/demo/blocks")

    and for the original code or:

    0FSType GetFilesystemType(const fs::path& path)
1{
2    return FSType::OTHER;
3}

    I’m not getting any additional warnings.

    ACK 504fe8a3e0a3def67a1ed3e309992d0115cf13cf

  110. DrahtBot requested review from ismaelsadeeq on Aug 8, 2025
  111. util: detect and warn when using exFAT on macOS 
    exFAT is known to cause corruption on macOS. See #28552.

Therefore we should warn when using this fs format for either the blocks
or data directories on macOS.

Co-authored-by: l0rinc <pap.lorinc@gmail.com>
    db3228042b
  112. willcl-ark force-pushed on Aug 8, 2025
  113. willcl-ark commented at 6:24 pm on August 8, 2025: member

    Seeing this planned for V30 I reviewed it a bit more thoroughly. My Concept ACK remains, but I think we can document and simplify and modernize it a bit more. Suggestions patch

    Thanks for your review @l0rinc.

    Have taken your entire patch (and one fixup) and added you as co-author, thanks!

  114. l0rinc commented at 7:00 pm on August 8, 2025: contributor
    ACK db3228042b2278faf7713a6ed40cdbf8e62b8fd4
  115. in src/init.cpp:1886 in db3228042b 
    1881+            break;
1882+        }
1883+    }};
1884+
1885+    check_and_warn_fs(args.GetDataDirNet(), "data directory");
1886+    check_and_warn_fs(args.GetBlocksDirPath(), "blocks directory");
    achow101 commented at 9:50 pm on August 8, 2025:
    This now causes two warning dialogs to appear consecutively when the blocksdir is a subdirectory of the datadir.
    l0rinc commented at 9:57 pm on August 8, 2025:
    Do you think that will be confusing? We don’t expect this to happen all the time, having separate warnings draws more attention in my opinion.
    achow101 commented at 10:06 pm on August 8, 2025:

    I think it’s confusing, especially since most users probably don’t know that blocksdir can be specified separately.

    This was also discussed earlier in the PR with the decision to have one warning: #31453 (comment)

  116. ismaelsadeeq commented at 5:26 pm on August 10, 2025: member

    reACK db3228042b2278faf7713a6ed40cdbf8e62b8fd4

    Could be nice if #31453 (review) will be fixed as well, but not a blocker for me.

  117. marcofleon commented at 12:32 pm on August 12, 2025: contributor
    reACK db3228042b2278faf7713a6ed40cdbf8e62b8fd4
  118. glozow merged this on Aug 12, 2025
  119. glozow closed this on Aug 12, 2025


willcl-ark DrahtBot maflcko l0rinc edilmedeiros philmb3487 laanwj marcofleon sipa helloscoopa hebasto tdb3 luke-jr Zero-1729 gentlee davidgumberg pablomartin4btc achow101 ismaelsadeeq


laanwj sipa davidgumberg Zero-1729 hebasto edilmedeiros marcofleon tdb3 pablomartin4btc

Labels
Utils/log/libs

Milestone
30.0

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-08-13 09:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me