Alternative to #33562, which was adding a native FreeBSD job; however that had issues with permissions/caching, as well as potential determinism issues. This adds a FreeBSD cross job using Linux and Clang.
Would close #33438. The same changes here could also be used to produce FreeBSD binaries out of Guix.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
See the guideline for information on the review process.
If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.
No conflicts as of last run.
🚧 At least one of the CI tasks failed.
Task lint: https://github.com/bitcoin/bitcoin/actions/runs/21630222694/job/62340895630
LLM reason (✨ experimental): Lint failure: a script lacks executable permission (ci/test/00_setup_env_freebsd_cross.sh) causing lint-files.py to fail.
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
103@@ -104,4 +104,17 @@ if [ -n "$XCODE_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${OSX_SDK_BASENAME}" ]
104 tar -C "${DEPENDS_DIR}/SDKs" -xf "$OSX_SDK_PATH"
105 fi
106
107+mkdir -p "${DEPENDS_DIR}/SDKs/freebsd-cross-${FREEBSD_VERSION}"
108+
109+FREEBSD_SDK_BASENAME="freebsd-cross-${XCODE_VERSION}"
110+
111+if [ -n "$FREEBSD_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}" ]; then
112+ FREEBSD_SDK_FILENAME="base-15.0.txz"
0 FREEBSD_SDK_FILENAME="base-${FREEBSD_VERSION}.txz"
0@@ -1,31 +1,39 @@
1+FREEBSD_VERSION=15.0
make -C depends/ HOST=x86_64-unknown-freebsd FREEBSD_VERSION=14.4.
The current assignment only allows overrides via command-line arguments, ignoring environment variables. This can be counter-intuitive. Suggesting the following change:
0FREEBSD_VERSION ?= 15.0
7+export LC_ALL=C.UTF-8
8+
9+export CONTAINER_NAME=ci_freebsd_cross
10+export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:24.04"
11+export APT_LLVM_V="21"
12+export FREEBSD_VERSION=15.0
maybe use 14.3 for now, to avoid the failure?
Yea, that works, so I’ve pushed it up for now.
potential determinism issues
You can drop this from the OP, given #34093 (comment)
potential determinism issues
You can drop this from the OP, given #34093 (comment)
This refers to the alternative approach, which was using unpinned third-party deps, so I think the pull request description is correct.
103@@ -104,4 +104,17 @@ if [ -n "$XCODE_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${OSX_SDK_BASENAME}" ]
104 tar -C "${DEPENDS_DIR}/SDKs" -xf "$OSX_SDK_PATH"
105 fi
106
107+mkdir -p "${DEPENDS_DIR}/SDKs/freebsd-cross-${FREEBSD_VERSION}"
108+
109+FREEBSD_SDK_BASENAME="freebsd-cross-${XCODE_VERSION}"
0FREEBSD_SDK_BASENAME="freebsd-cross-${FREEBSD_VERSION}"
0@@ -1,31 +1,39 @@
1+FREEBSD_VERSION=15.0
2+FREEBSD_SDK=$(SDK_PATH)/freebsd-cross-$(FREEBSD_VERSION)/
109+FREEBSD_SDK_BASENAME="freebsd-${HOST}-${FREEBSD_VERSION}"
110+
111+if [ -n "$FREEBSD_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}" ]; then
112+ FREEBSD_SDK_FILENAME="base-${FREEBSD_VERSION}.txz"
113+ FREEBSD_SDK_PATH="${DEPENDS_DIR}/sdk-sources/${FREEBSD_SDK_FILENAME}"
114+ if [ ! -f "$OSX_SDK_PATH" ]; then
0 if [ ! -f "$FREEBSD_SDK_PATH" ]; then
112+ FREEBSD_SDK_FILENAME="base-${FREEBSD_VERSION}.txz"
113+ FREEBSD_SDK_PATH="${DEPENDS_DIR}/sdk-sources/${FREEBSD_SDK_FILENAME}"
114+ if [ ! -f "$OSX_SDK_PATH" ]; then
115+ ${CI_RETRY_EXE} curl --location --fail "https://download.freebsd.org/releases/amd64/${FREEBSD_VERSION}-RELEASE/base.txz" -o "$FREEBSD_SDK_PATH"
116+ fi
117+ tar -C "${DEPENDS_DIR}/SDKs/freebsd-cross-${FREEBSD_VERSION}" -xf "$FREEBSD_SDK_PATH"
0 tar -C "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}" -xf "$FREEBSD_SDK_PATH"
103@@ -104,4 +104,17 @@ if [ -n "$XCODE_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${OSX_SDK_BASENAME}" ]
104 tar -C "${DEPENDS_DIR}/SDKs" -xf "$OSX_SDK_PATH"
105 fi
106
107+mkdir -p "${DEPENDS_DIR}/SDKs/freebsd-${HOST}-${FREEBSD_VERSION}"
108+
109+FREEBSD_SDK_BASENAME="freebsd-${HOST}-${FREEBSD_VERSION}"
110+
111+if [ -n "$FREEBSD_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}" ]; then
To avoid duplication we could use this in making the dir above - which should likely be after we’re checking for it, i.e.
0FREEBSD_SDK_BASENAME="freebsd-${HOST}-${FREEBSD_VERSION}"
1
2if [ -n "$FREEBSD_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}" ]; then
3 mkdir -p "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}"
6+# prior to 1af314465e5dfe3e8baa839a32a72e83c04f26ef, first released in v4.2.90.
7+# At the time of writing, GNU Make v4.2.1 is still being used in supported
8+# distro releases.
9+#
10+# Source: https://lists.gnu.org/archive/html/bug-make/2017-11/msg00017.html
11+clang_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang")
freebsd_ prefix
103@@ -104,4 +104,16 @@ if [ -n "$XCODE_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${OSX_SDK_BASENAME}" ]
104 tar -C "${DEPENDS_DIR}/SDKs" -xf "$OSX_SDK_PATH"
105 fi
106
107+FREEBSD_SDK_BASENAME="freebsd-${HOST}-${FREEBSD_VERSION}"
108+
109+if [ -n "$FREEBSD_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}" ]; then
110+ FREEBSD_SDK_FILENAME="base-${FREEBSD_VERSION}.txz"
we already have a basename above, could we use it here?
0 FREEBSD_SDK_FILENAME="${FREEBSD_SDK_BASENAME}-base.txz"
111+ FREEBSD_SDK_PATH="${DEPENDS_DIR}/sdk-sources/${FREEBSD_SDK_FILENAME}"
112+ if [ ! -f "$FREEBSD_SDK_PATH" ]; then
113+ ${CI_RETRY_EXE} curl --location --fail "https://download.freebsd.org/releases/amd64/${FREEBSD_VERSION}-RELEASE/base.txz" -o "$FREEBSD_SDK_PATH"
114+ fi
115+ mkdir -p "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}"
116+ tar -C "${DEPENDS_DIR}/SDKs/freebsd-${HOST}-${FREEBSD_VERSION}" -xf "$FREEBSD_SDK_PATH"
we could use the FREEBSD_SDK_BASENAME here as well:
0 tar -C "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}" -xf "$FREEBSD_SDK_PATH"
539@@ -540,6 +540,13 @@ jobs:
540 timeout-minutes: 120
541 file-env: './ci/test/00_setup_env_native_msan.sh'
542
543+
544+ - name: 'FreeBSD Cross'
539@@ -540,6 +540,13 @@ jobs:
540 timeout-minutes: 120
541 file-env: './ci/test/00_setup_env_native_msan.sh'
542
543+
544+ - name: 'FreeBSD Cross'
545+ cirrus-runner: 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-md'
if we’re just compiling (not even running tests), would a small instance suffice?
0 cirrus-runner: 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-sm'
19+ -DREDUCE_EXPORTS=ON \
20+ -DBUILD_GUI=OFF \
21+ -DWITH_USDT=OFF \
22+"
23+export RUN_UNIT_TESTS="false"
24+export RUN_FUNCTIONAL_TESTS="false"
Are we planning on enabling the tests later? Having BSD tests would have helped with #34256
nit: based on other usages the values don’t have to be strings: https://github.com/bitcoin/bitcoin/blob/e4c04f7759b0b390189410f5ef3ad5faa5354698/ci/test/00_setup_env_mac_native_fuzz.sh#L15-L17
Are we planning on enabling the tests later?
Not sure. This is just adding cross compilation.
22+
23+freebsd_CC=$(clang_prog) --target=$(host) \
24+ --sysroot=$(FREEBSD_SDK)
25+
26+freebsd_CXX=$(clangxx_prog) --target=$(host) \
27+ --sysroot=$(FREEBSD_SDK) -stdlib=libc++
isn’t libc++ the default here?
0 --sysroot=$(FREEBSD_SDK)
10+# Source: https://lists.gnu.org/archive/html/bug-make/2017-11/msg00017.html
11+clang_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang")
12+clangxx_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang++")
13+
14+freebsd_AR=$(shell $(SHELL) $(.SHELLFLAGS) "command -v llvm-ar")
15+freebsd_DSYMUTIL=$(shell $(SHELL) $(.SHELLFLAGS) "command -v dsymutil")
command -v dsymutil returns nothing,
8+
9+export CONTAINER_NAME=ci_freebsd_cross
10+export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:24.04"
11+export APT_LLVM_V="21"
12+export FREEBSD_VERSION=15.0
13+export PACKAGES="clang-${APT_LLVM_V} llvm-${APT_LLVM_V} lld"
doesn’t lld need the same version?
0export PACKAGES="clang-${APT_LLVM_V} llvm-${APT_LLVM_V} lld-${APT_LLVM_V}"
this will probably also need:
0diff --git a/ci/test/01_base_install.sh b/ci/test/01_base_install.sh
1--- a/ci/test/01_base_install.sh (revision e8df2be3b9fbd8cfdf10e3a5f0e7c344d10a2673)
2+++ b/ci/test/01_base_install.sh (date 1773780463181)
3@@ -53,6 +53,7 @@
4 if [ -n "${APT_LLVM_V}" ]; then
5 update-alternatives --install /usr/bin/clang++ clang++ "/usr/bin/clang++-${APT_LLVM_V}" 100
6 update-alternatives --install /usr/bin/clang clang "/usr/bin/clang-${APT_LLVM_V}" 100
7+ update-alternatives --install /usr/bin/ld.lld ld.lld "/usr/bin/ld.lld-${APT_LLVM_V}" 100
8 update-alternatives --install /usr/bin/llvm-symbolizer llvm-symbolizer "/usr/bin/llvm-symbolizer-${APT_LLVM_V}" 100
9 fi
10+export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:24.04"
11+export APT_LLVM_V="21"
12+export FREEBSD_VERSION=15.0
13+export PACKAGES="clang-${APT_LLVM_V} llvm-${APT_LLVM_V} lld"
14+export HOST=x86_64-unknown-freebsd
15+export DEP_OPTS="build_CC=clang build_CXX=clang++ AR=llvm-ar-${APT_LLVM_V} STRIP=llvm-strip-${APT_LLVM_V} NM=llvm-nm-${APT_LLVM_V} RANLIB=llvm-ranlib-${APT_LLVM_V} NO_QT=1"
CC and CXX here, too, isn’t the freebsd_CC/freebsd_CXX enough?
build_CC and build_CXX are for native tools.
11+export APT_LLVM_V="21"
12+export FREEBSD_VERSION=15.0
13+export PACKAGES="clang-${APT_LLVM_V} llvm-${APT_LLVM_V} lld"
14+export HOST=x86_64-unknown-freebsd
15+export DEP_OPTS="build_CC=clang build_CXX=clang++ AR=llvm-ar-${APT_LLVM_V} STRIP=llvm-strip-${APT_LLVM_V} NM=llvm-nm-${APT_LLVM_V} RANLIB=llvm-ranlib-${APT_LLVM_V} NO_QT=1"
16+export GOAL="install"
103@@ -104,4 +104,16 @@ if [ -n "$XCODE_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${OSX_SDK_BASENAME}" ]
104 tar -C "${DEPENDS_DIR}/SDKs" -xf "$OSX_SDK_PATH"
105 fi
106
107+FREEBSD_SDK_BASENAME="freebsd-${HOST}-${FREEBSD_VERSION}"
108+
109+if [ -n "$FREEBSD_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}" ]; then
110+ FREEBSD_SDK_FILENAME="base-${FREEBSD_VERSION}.txz"
111+ FREEBSD_SDK_PATH="${DEPENDS_DIR}/sdk-sources/${FREEBSD_SDK_FILENAME}"
sdk-sources also be included in the explicit CI caches?
It looks like the first commit introduces a regression for FreeBSD users cross-compiling to other architectures:
0$ gmake -j 8 -C depends NO_QT=1 HOST=i686-unknown-freebsd
1$ cmake -B build --toolchain depends/i686-unknown-freebsd/toolchain.cmake
2$ cmake --build build -t bitcoind
3[323/323] Linking CXX executable bin/bitcoind
4FAILED: [code=1] bin/bitcoind
5: && /usr/bin/clang++ --target=i686-unknown-freebsd -O2 -O2 -g -fuse-ld=lld -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -fPIE -pie -Xlinker --dependency-file=src/CMakeFiles/bitcoind.dir/link.d src/CMakeFiles/bitcoind.dir/bitcoind.cpp.o src/CMakeFiles/bitcoind.dir/init/bitcoind.cpp.o -o bin/bitcoind lib/libbitcoin_node.a lib/libbitcoin_wallet.a lib/libbitcoin_zmq.a /home/hebasto/dev/bitcoin/depends/i686-unknown-freebsd/lib/libzmq.a src/libleveldb.a src/libcrc32c.a src/libminisketch.a /home/hebasto/dev/bitcoin/depends/i686-unknown-freebsd/lib/libevent_extra.a /home/hebasto/dev/bitcoin/depends/i686-unknown-freebsd/lib/libevent_pthreads.a /home/hebasto/dev/bitcoin/depends/i686-unknown-freebsd/lib/libevent_core.a -lpthread lib/libbitcoin_common.a lib/libbitcoin_util.a lib/libbitcoin_clientversion.a lib/libbitcoin_consensus.a lib/libbitcoin_crypto.a src/secp256k1/lib/libsecp256k1.a src/univalue/libunivalue.a -pthread /home/hebasto/dev/bitcoin/depends/i686-unknown-freebsd/lib/libsqlite3.a && :
6ld.lld: error: /home/hebasto/dev/bitcoin/depends/i686-unknown-freebsd/lib/libsqlite3.a(sqlite3.o) is incompatible with elf_i386_fbsd
7clang++: error: linker command failed with exit code 1 (use -v to see invocation)
8ninja: build stopped: subcommand failed.
It looks like the first commit introduces a regression for FreeBSD users cross-compiling to other architectures: HOST=i686-unknown-freebsd
What SDK are you using for i386; I don’t see a 15.0 release under https://download.freebsd.org/releases/i386/?
If I use the 14.4 version, and your commands, it builds fine (compiling on x86_64):
0clang --version
1Ubuntu clang version 18.1.3 (1ubuntu1)
2Target: x86_64-pc-linux-gnu
3...
4file build/bin/bitcoind
5build/bin/bitcoind: ELF 32-bit LSB pie executable, Intel 80386, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.4, FreeBSD-style, with debug_info, not stripped
It looks like the first commit introduces a regression for FreeBSD users cross-compiling to other architectures: HOST=i686-unknown-freebsd
What SDK are you using for
i386; I don’t see a 15.0 release under https://download.freebsd.org/releases/i386/?
Currently, on the master branch, there is no need to specify an SDK to cross-compile on FreeBSD x86_64 to i686.
6+
7+export LC_ALL=C.UTF-8
8+
9+export CONTAINER_NAME=ci_freebsd_cross
10+export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:24.04"
11+export APT_LLVM_V="21"
19.x. We can try pin to 19, or just use 22, I don’t think I mind either way. One downside of pinning to older, could be poorer support in the apt repo (for current, or newer images).
APT_LLVM_V, having one less place to bump. But again, this doesn’t really matter, so anything is fine here.
15+export DEP_OPTS="build_CC=clang build_CXX=clang++ AR=llvm-ar-${APT_LLVM_V} STRIP=llvm-strip-${APT_LLVM_V} NM=llvm-nm-${APT_LLVM_V} RANLIB=llvm-ranlib-${APT_LLVM_V} NO_QT=1"
16+export GOAL="install"
17+export BITCOIN_CONFIG="\
18+ --preset=dev-mode \
19+ -DREDUCE_EXPORTS=ON \
20+ -DBUILD_GUI=OFF \
I’ll add a comment, for reference at least one Qt compile failure is:
0ld.lld: error: undefined symbol: QKqueueFileSystemWatcherEngine::create(QObject*)
1>>> referenced by qfilesystemwatcher.cpp
2>>> qfilesystemwatcher.cpp.o:(QFileSystemWatcher::QFileSystemWatcher(QObject*)) in archive /root/ci_scratch/depends/aarch64-unknown-freebsd/lib/libQt6Core.a
3clang++: error: linker command failed with exit code 1 (use -v to see invocation)
Could be fixed with the following patch:
0--- a/depends/packages/qt.mk
1+++ b/depends/packages/qt.mk
2@@ -167,6 +167,8 @@ $(package)_config_env += CXX="$$($(package)_cxx)"
3 $(package)_config_env_darwin := OBJC="$$($(package)_cc)"
4 $(package)_config_env_darwin += OBJCXX="$$($(package)_cxx)"
5
6+$(package)_config_opts_freebsd += -no-feature-inotify
7+
8 $(package)_cmake_opts := -DCMAKE_PREFIX_PATH=$(host_prefix)
9 $(package)_cmake_opts += -DQT_FEATURE_cxx20=ON
10 $(package)_cmake_opts += -DQT_GENERATE_SBOM=OFF
Could be fixed with the following patch:
Did this work for you? The build is still broken here, after applying the above.
Could be fixed with the following patch:
Did this work for you? The build is still broken here, after applying the above.
I successfully built on Ubuntu 25.10. -no-feature-inotify is required to enabling compiling of the required source file:
0qt_internal_extend_target(Core CONDITION QT_FEATURE_filesystemwatcher AND UNIX AND NOT MACOS AND NOT QT_FEATURE_inotify AND (APPLE OR FREEBSD OR NETBSD OR OPENBSD)
1 SOURCES
2 io/qfilesystemwatcher_kqueue.cpp io/qfilesystemwatcher_kqueue_p.h
3)
Concept ACK
Though, I am not familiar with freebsd, nor the build system, so I can’t really review this
The same changes here could also be used to produce FreeBSD binaries out of Guix.
https://download.freebsd.org/releases/amd64/15.0-RELEASE/base.txz doesn’t seem to include static libraries like libc.a, libc++.a and libcxxrt.a. How should we handle this for the release binaries?
How should we handle this for the release binaries?
I’ll look at that when we look at adding FreeBSD bins to releases, not as part of this PR.
bitcoind, test_bitcoin and bitcoin-qt) were tested on FreeBSD 14.4 locally.
0@@ -1,31 +1,38 @@
1+FREEBSD_VERSION ?= 15.0
2+FREEBSD_SDK=$(SDK_PATH)/freebsd-$(host)-$(FREEBSD_VERSION)/
3+
4+# We can't just use $(shell command -v clang) because GNU Make handles builtins
5+# in a special way and doesn't know that `command` is a POSIX-standard builtin
6+# prior to 1af314465e5dfe3e8baa839a32a72e83c04f26ef, first released in v4.2.90.
7+# At the time of writing, GNU Make v4.2.1 is still being used in supported
8+# distro releases.
darwin.mk. I’d imagine it was in regards to Debian, or Ubuntu 20.04 (at the time) Maybe we can drop this at this point, as I don’t think it’s likely that Ubuntu 20.04 is being used to cross-compile for macOS. See #34923.