bitcoind and bitcoin-cli does not supports symbolic link anymore

nymkappa commented at 2:44 am on February 4, 2022: none

I just pulled latest master at https://github.com/bitcoin/bitcoin/commit/3ace3a17c9bce606cea05192f0da3ac62ac69dda and I cannot run bitcoind or bitcoin-cli without providing a full path for data directory using -datadir option. It was working fine couple of days ago (while running https://github.com/bitcoin/bitcoin/commit/807169e10b4a18324356ed6ee4d69587b96a7c70). I suspect it may be related to latest switch to the standard std::filesystem library #20744

Expected behavior

I use a symbolic link to point my /home/me/.bitcoin to an external hard drive. When running bitcoind from command line, or executing bitcoin-cli commands, without using the -datadir option, it should not terminate or crash unexpectedly. Those commands should be able to follow the symbolic link and be able to read/write into the final folder.

Actual behavior

bitcoind and bitcoin-cli are not able to follow the symbolic link. bitcoin-qt runs fine, even when running it from command line. See below commands examples.

 0# Create a symbolic link
 1$~/btc-things/bitcoin$ ln -s /media/DATA/Bitcoin /home/me/.bitcoin
 2
 3# Try to run bitcoind - Crashes
 4$~/btc-things/bitcoin$ ./bitcoind 
 5
 6
 7************************
 8EXCEPTION: NSt10filesystem7__cxx1116filesystem_errorE       
 9filesystem error: cannot create directories: Not a directory [/home/me/.bitcoin]       
10bitcoin in AppInit()       
11
12bitcoind: chainparamsbase.cpp:35: const CBaseChainParams& BaseParams(): Assertion `globalChainBaseParams' failed.
13Aborted (core dumped)
14
15# Run bitcoin-qt, it works fine, the node runs
16$~/btc-things/bitcoin$ ./bitcoin-qt 
17qt5ct: using qt5ct plugin
18
19# Try to run a bitcoin-cli command to interact with the running node, does not work
20$~/btc-things/bitcoin$ ./bitcoin-cli getblockhash 501726
21
22
23************************
24EXCEPTION: NSt10filesystem7__cxx1116filesystem_errorE       
25filesystem error: cannot create directories: Not a directory [/home/me/.bitcoin]       
26bitcoin in AppInitRPC()       
27
28# Try the same previous command but give an absolute path for -datadir, works fine
29$~/btc-things/bitcoin$ ./bitcoin-cli -datadir=/media/DATA/Bitcoin getblockhash 501726
300000000000000000004b27f9ee7ba33d6f048f684aaeb0eea4befd80f1701126

To reproduce

Steps are indicated previously.

System information

Running Bitcoin Core version v22.99.0-3ace3a17c9bc. Self compiled from master branch.

I’m using Linux Mint with the following kernel Linux 5.4.0-96-generic x86_64. I’m using an additional hard drive to store the full blockchain (SSD).

I had no prior issues before pulling latest changes. I was previously running at https://github.com/bitcoin/bitcoin/commit/807169e10b4a18324356ed6ee4d69587b96a7c70

nymkappa added the label Bug on Feb 4, 2022

hebasto commented at 11:57 am on February 4, 2022: member

@nymkappa

What is result of running unit and functional tests?

MarcoFalke added this to the milestone 23.0 on Feb 4, 2022

hebasto commented at 12:24 pm on February 4, 2022: member

Confirming the bug on Linux Mint 20.2 (x86_64):

 0$ mkdir /home/hebasto/NVMe/TEST_DATADIR
 1$ ln -s /home/hebasto/NVMe/TEST_DATADIR /home/hebasto/.bitcoin
 2$ ./src/bitcoind -regtest
 3
 4
 5************************
 6EXCEPTION: NSt10filesystem7__cxx1116filesystem_errorE       
 7filesystem error: cannot create directories: Not a directory [/home/hebasto/.bitcoin]       
 8bitcoin in AppInit()       
 9
10bitcoind: chainparamsbase.cpp:35: const CBaseChainParams &BaseParams(): Assertion `globalChainBaseParams' failed.
11Aborted (core dumped)

UPDATE: The bug is not reproducible on Ubuntu 21.10.

UPDATE 2: Confirming on Ubuntu 20.04.3 LTS.

nymkappa commented at 1:34 pm on February 4, 2022: none

@nymkappa

What is result of running unit and functional tests?

I actually had the same issue that you’ve described here #20869 (comment). Removing qt5ct allowed me to run make check. No error shown, all tests seem to pass.

I’m still waiting for feature_dbcrash.py test to complete, but all others went through without any issue.

UPDATE: All tests passes

hebasto commented at 2:06 pm on February 4, 2022: member

Looks like a libstdc++ bug in std::filesystem::create_directories implementation.

After configuring with CC=clang-12 CXX='clang++-12 -stdlib=libc++' and compiling, everything works just fine.

sebbavw commented at 9:13 pm on February 4, 2022: none

I have the same issue on a RaspberryPi 4b 8gig: Bitcoin Core version v22.99.0-b2a8371913a8

0bitcoin@raspberrypi:/media/pi/PiData/bitcoin/.bitcoin $ bitcoind -regtest
1************************
2EXCEPTION: NSt10filesystem7__cxx1116filesystem_errorE       
3filesystem error: cannot create directories: Not a directory [/home/bitcoin/.bitcoin]       
4bitcoin in AppInit()       
5
6bitcoind: chainparamsbase.cpp:35: const CBaseChainParams& BaseParams(): Assertion `globalChainBaseParams' failed.

UPDATE: Manually implemented #24266 locally, it fixed the issue here Bitcoin Core version v22.99.0-b2a8371913a8-dirty

hebasto commented at 9:23 pm on February 4, 2022: member

@nymkappa @sebbavw

Mind testing #24266?

hebasto commented at 3:33 pm on February 5, 2022: member

Looks like a libstdc++ bug in std::filesystem::create_directories implementation.

The bug has been fixed:

upstream 124eaa50e0a34f5f89572c1aa812c50979da58fc
backported to gcc-11 bde28c60c70079416dd2181f882c0694e019eaef (not in 11.2.0)
backported to gcc-10 86bdcd21a1be008df28648b185c12221d917a166 (not in 10.3.0)
backported to gcc-9 f8ac77533951d79d4e6a65841aa30293b2f11fdd (not in 9.4.0)

Some distros backporting info (as for 2022-02-05)

Debian 12 Bookworm, libstdc++6

Backported since gcc-11 (11.2.0-2).

Debian 11 Bullseye, libstdc++6

No backports to gcc-10.

Ubuntu 22.04 LTS, libstdc++6

Backported since gcc-11 (11.2.0-1ubuntu3).

Ubuntu 21.10, libstdc++6

Backported since gcc-11 (11.2.0-1ubuntu3).

Ubuntu 20.04 LTS, libstdc++6

No backports to gcc-10.

Did not check gcc-10.3.0 we use for GUIX builds.

MarcoFalke closed this on Feb 8, 2022

sidhujag referenced this in commit 7b80800859 on Feb 9, 2022

laanwj commented at 2:33 pm on February 11, 2022: member

I don’t think this is fixed (as of bcecde64b4ad7bb8e717d5709af8fd17532755ab)? My blocks directory is a symlink this used to work but now I get the following error on a run-of-the-mill Ubuntu 20.04 VM.

0$ src/bitcoind
1
2************************
3EXCEPTION: NSt10filesystem7__cxx1116filesystem_errorE
4filesystem error: cannot create directories: Not a directory [/home/…/.bitcoin/blocks]
5bitcoin in AppInit()

laanwj reopened this on Feb 11, 2022

w0xlt commented at 7:32 pm on February 11, 2022: contributor

I could reproduce the problem (when the blocks directory is a symlink) on Ubuntu 20.04.3 LTS I tried the patch below and it fixed the mentioned error but another error still occurs when creating blocks/index.

filesystem error: cannot create directories: File exists [/home/.../.bitcoin/signet/blocks/index]

 0
 1// src/util/system.cpp
 2path /= fs::PathFromString(BaseParams().DataDir());
 3path /= "blocks";
 4-if (!fs::exists(path)) {
 5+if (!fs::exists(path) && !fs::is_symlink(path)) {
 6        fs::create_directories(path);
 7}
 8
 9// src/init.cpp
10-if (!fs::is_directory(gArgs.GetBlocksDirPath())) {
11+if (!fs::is_directory(gArgs.GetBlocksDirPath()) && !fs::is_symlink(gArgs.GetBlocksDirPath())) {
12        return InitError(strprintf(_("Specified blocks directory \"%s\" does not exist."), args.GetArg("-blocksdir", "")));
13    }

laanwj commented at 11:31 am on February 14, 2022: member

I think it would be the most thorough solution to make our own fs::create_directories wrapper and use that instead of fs::create_directories, everywhere it’s used now. It’s used in a lot of places in the code, fixing them one by one is error prone.

laanwj referenced this in commit 36a681581e on Feb 14, 2022

laanwj referenced this in commit e6fafbdfd0 on Feb 14, 2022

laanwj referenced this in commit 01c5aefa03 on Feb 14, 2022

laanwj referenced this in commit b9dff100ff on Feb 14, 2022

ghost commented at 11:44 am on February 16, 2022: none

I don’t know enough about symlink and related issues to review #24338 however read the details of https://blog.rust-lang.org/2022/01/20/cve-2022-21658.html, hopefully we don’t have any such race conditions that could be exploited.

laanwj referenced this in commit 2df2c10ee1 on Feb 16, 2022

laanwj referenced this in commit b15bde7530 on Feb 16, 2022

laanwj referenced this in commit c936fc30ec on Feb 16, 2022

hebasto referenced this in commit 1a1f795f74 on Feb 16, 2022

laanwj referenced this in commit 1f46b6e46e on Feb 17, 2022

fanquake closed this on Feb 17, 2022

fanquake referenced this in commit 003523d239 on Feb 17, 2022

sidhujag referenced this in commit fd145549cb on Feb 18, 2022

hmel referenced this in commit 273f3fbc06 on Feb 20, 2022