Qt: Add Android packaging support

icota commented at 1:41 pm on October 23, 2019: contributor

This PR is the third and final piece of the basic Android support puzzle - it depends on #16110 and is related to #16883. It introduces an android directory under qt and a simple way to build an Android package of bitcoin-qt:

Build depends for Android as described in the README
Configure with one of the resulting prefixes
Run make && make apk in src/qt

The resulting APK files will be in android/build/outputs/apk. You can install them manually or with adb. One can also open the android directory in Android Studio for that integrated development and debugging experience. BitcoinQtActivity is your starting point.

Under the hood makefile apk target:

Renames the bitcoin-qt binary to libbitcoin-qt.so and copies it over to a folder under android/libs depending on which prefix and corresponding ABI bitcoin-qt was built for
Takes libc++_shared.so from the Android NDK and puts in the same place. It must be included in the APK
Extracts Qt for Android Java support files from the qtbase archive in depends/sources to android/src

There is also just a tiny bit of ifdef’d code to make the Qt Widgets menus usable. It’s not pretty but it works and is a stepping stone towards #16883.

fanquake added the label Android on Oct 23, 2019

DrahtBot commented at 2:30 pm on October 23, 2019: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#21495 (build, qt: Fix static builds on macOS Big Sur by hebasto)
#21478 (guix: Use pigz as a faster gzip replacement by dongcarl)
#21462 (guix: Add guix-{attest,verify} scripts by dongcarl)
#21420 (build, qt: No longer need to patch translation.pro by hebasto)
#21375 (guix: Misc feedback-based fixes + hier restructuring by dongcarl)
#21304 (guix: Add guix-clean script + establish gc-root for container profiles by dongcarl)
#19817 (build: macOS toolchain bump by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

laanwj added the label GUI on Oct 24, 2019

laanwj commented at 6:37 pm on October 24, 2019: member

Concept ACK, nice !

laanwj requested review from theuni on Oct 25, 2019

laanwj requested review from dongcarl on Oct 25, 2019

icota force-pushed on Nov 4, 2019

icota commented at 3:03 pm on November 4, 2019: contributor

Rebased, squashed, added instructions under doc and updated the first comment to reflect that #16110 has been merged.

in doc/package-android.md:1 in d5f83c5cce outdated

0@@ -0,0 +1,12 @@
1+Android package guide

MarcoFalke commented at 3:08 pm on November 4, 2019:

Should this file be linked to from the doc/Readme?

Also, should it be named build-android, similar to build-unix and build-bsd?

icota commented at 3:31 pm on November 4, 2019:

I thought so but other build files seem to be about building on UNIX, BSD, etc… and this file is mostly about packaging bitcoin-qt for Android. I’m open to suggestions and whatever name it ends up being I’ll make sure to link from doc/README.md

MarcoFalke commented at 3:34 pm on November 4, 2019:

Oh, this is for cross-compilation? Well, we have build-win

icota commented at 3:46 pm on November 4, 2019:

You are right. I will change to build-android and make it a bit more generic (as in not just about packaging)

icota force-pushed on Nov 4, 2019

luke-jr commented at 4:30 pm on November 4, 2019: member

This PR is the third and final piece of the basic Android support puzzle

IMO final would be a gitian descriptor ;)

DrahtBot added the label Needs rebase on Feb 10, 2020

adamjonas commented at 1:01 am on May 1, 2020: member

@icota Friendly ping for a rebase.

icota force-pushed on May 2, 2020

icota commented at 10:36 am on May 2, 2020: contributor

Thank you @adamjonas. Rebased.

DrahtBot removed the label Needs rebase on May 2, 2020

MarcoFalke commented at 8:31 pm on May 2, 2020: member

@icota Could you pull in https://github.com/MarcoFalke/bitcoin-core/commit/a0b441baf9dbb3ce7e332b0a8e0d4d278e73f611 to shave off a few kB off of the compressed images?

icota commented at 9:15 am on May 3, 2020: contributor

Absolutely. Thanks @MarcoFalke

MarcoFalke commented at 11:05 am on May 3, 2020: member

Oh, you’ll have to squash it. Otherwise git will keep track of both versions, thus doubling the size.

icota force-pushed on May 3, 2020

icota commented at 12:24 pm on May 3, 2020: contributor

Done.

BlockMechanic commented at 10:12 pm on May 18, 2020: contributor

Awesome, will test this out.

jonasschnelli commented at 7:44 am on May 29, 2020: contributor

utACK a818b2c082e257eb67403479df61f1325231c947

MarcoFalke commented at 4:00 pm on May 29, 2020: member

Concept ACK. Not sure how this can be tested in ci. I guess it needs the android dev kit?

icota commented at 0:12 am on May 30, 2020: contributor

Yep it needs the SDK. Travis seems to support this.

MarcoFalke commented at 0:35 am on May 31, 2020: member

Unfortunately (or luckily) we can’t use travis’ packages. We use our own ci setup in ./ci.

The sdkmanager is a zip file that can be downloaded, though. And then a one-liner like

0$ANDROID_HOME/tools/bin/sdkmanager --sdk_root=$ANDROID_HOME "platform-tools" "platforms;android-28" "ndk-bundle"

should do the rest.

jonasschnelli commented at 7:47 am on June 5, 2020: contributor

I’d like to give this a try in the main branch (even without CI support). If it gets unmaintained, a revert is likely. Anyone with Android experience willing to test this @Sjors, @hebasto, @promag, @greenaddress?

MarcoFalke commented at 12:22 pm on June 5, 2020: member

To clarify. I did try to build this, but the linker step of bitcoind failed, which is why I asked for a ./ci file or at least some indication that anyone has tried compiling this current version of the pull merged into current master. I am generally not a fan of merging code and build system changes that may or may not compile.

I don’t have android experience, so maybe I did something wrong.

icota commented at 8:24 am on June 9, 2020: contributor

@MarcoFalke On Linux you might have to pull #17396. Specifically I think the linker error is due to an autoconf issue. A fix has since been merged upstream.

BlockMechanic commented at 5:40 pm on June 26, 2020: contributor

To clarify. I did try to build this, but the linker step of bitcoind failed, which is why I asked for a ./ci file or at least some indication that anyone has tried compiling this current version of the pull merged into current master. I am generally not a fan of merging code and build system changes that may or may not compile.

I don’t have android experience, so maybe I did something wrong.

I found some issues in the bitcoin_qt.m4 file. Mine now compiles, now trying to produce a apk. @icota which version of grade is best ? i see you were using 3.2, but the latest is 6.x. Should we have a recommended version ?

icota commented at 8:53 am on June 30, 2020: contributor

@icota which version of grade is best ? i see you were using 3.2, but the latest is 6.x. Should we have a recommended version ?

I don’t think so, we are not using much build system wizardry here anyway… Unless you have a specific reason to bump Gradle, I’d keep it at 3.2

MarcoFalke commented at 1:32 pm on August 24, 2020: member

Needs rebase?

icota force-pushed on Aug 25, 2020

icota commented at 7:15 am on August 25, 2020: contributor

Rebased. I’ll look into adding this to CI.

icota force-pushed on Aug 27, 2020

icota commented at 1:15 pm on August 27, 2020: contributor

@MarcoFalke Can you take a look at the CI changes? The general idea is to have the Android CLI tools fetch everything needed to build and having done that we ‘iterate’ over the 4 supported ABIs so the final APK built will have contained all the binaries.

I haven’t actually tested this to completion (it’s going to take some time on my machine) but would like some feedback on the direction.

in ci/test/06_script_a.sh:37 in 2727b4b5c0 outdated

32+
33+  DOCKER_EXEC "${BASE_ROOT_DIR}/configure" --prefix=$DEPENDS_DIR/i686-linux-android || ( (DOCKER_EXEC cat config.log) && false)
34+  DOCKER_EXEC make $MAKEJOBS $GOAL || ( echo "Build failure. Verbose build follows." && DOCKER_EXEC make $GOAL V=1 ; false )
35+  DOCKER_EXEC make -C src/qt apk
36+
37+  DOCKER_EXEC "${BASE_ROOT_DIR}/configure" --prefix=$DEPENDS_DIR/x86_64-linux-android || ( (DOCKER_EXEC cat config.log) && false)

MarcoFalke commented at 1:20 pm on August 27, 2020:

It is probably enough to keep only the aarch64-linux-android build.

icota commented at 1:32 pm on August 27, 2020:

Agreed. Okay to just comment the others out in case somebody wants them?

MarcoFalke commented at 1:38 pm on August 27, 2020:

I think it would be easy to re-create them by copy-paste and adjusting the name, etc.

MarcoFalke commented at 1:21 pm on August 27, 2020: member

ci looks ok. You can also keep the ci changes in one separate commit, so if it turns out to not work, the commit can simply be dropped/reverted on its own

icota force-pushed on Aug 28, 2020

icota commented at 9:56 am on August 28, 2020: contributor

To build with CI run: FILE_ENV="./ci/test/00_setup_env_android.sh" ./ci/test_run_all.sh

To install on your (ARM64) device: docker cp ci_android:$PWD/src/qt/android/build/outputs/apk/debug/android-debug.apk . adb install android-debug.apk

MarcoFalke commented at 10:03 am on August 28, 2020: member

Could add the ci file to travis.yml or cirrus.yml? If it takes 30-40 minutes, then travis should be sufficient. If it takes longer, then cirrus might be better. If it takes longer than 2h, maybe leave it out for now.

icota commented at 11:18 am on August 28, 2020: contributor

@MarcoFalke It’s about 30 mins to build just for ARM64 so I added it to travis.

icota force-pushed on Aug 28, 2020

in src/qt/bitcoin.cpp:473 in c74442b946 outdated

459@@ -460,6 +460,13 @@ int GuiMain(int argc, char* argv[])
460 
461     // Generate high-dpi pixmaps
462     QApplication::setAttribute(Qt::AA_UseHighDpiPixmaps);
463+
464+#if defined(QT_QPA_PLATFORM_ANDROID)
465+    QApplication::setAttribute(Qt::AA_DontUseNativeMenuBar);
466+    QApplication::setAttribute(Qt::AA_DontCreateNativeWidgetSiblings);
467+    QApplication::setAttribute(Qt::AA_DontUseNativeDialogs);
468+#endif

luke-jr commented at 11:46 pm on October 24, 2020:

Why not??

icota commented at 9:50 am on October 26, 2020:

Native menus (or Qt’s Android native implementation of QMenu) don’t support submenus. When you tap on any of the four options nothing happens.

The other two attributes are there because native dialogs are way too simple to render our modals and to make sure the dialogs don’t create another window (which is only supported on ChromeOS).

decryp2kanon approved

decryp2kanon commented at 11:35 am on November 4, 2020: contributor

Concept ACK

DrahtBot added the label Needs rebase on Nov 4, 2020

icota force-pushed on Nov 5, 2020

DrahtBot removed the label Needs rebase on Nov 5, 2020

DrahtBot added the label Needs rebase on Nov 9, 2020

in .travis.yml:100 in 1168bb92df outdated

 93@@ -94,6 +94,11 @@ jobs:
 94       env: >-
 95         FILE_ENV="./ci/test/00_setup_env_native_nowallet.sh"
 96 
 97+    - stage: test
 98+      name: 'ARM64 Android APK  [GOAL: build]'
 99+      env: >-
100+        FILE_ENV="./ci/test/00_setup_env_android.sh"

MarcoFalke commented at 2:51 pm on November 9, 2020:

We no longer use travis, so you’ll have to move those added lines over to cirrus

icota commented at 4:59 pm on November 10, 2020:

Done

icota force-pushed on Nov 10, 2020

DrahtBot removed the label Needs rebase on Nov 10, 2020

icota commented at 5:03 pm on November 10, 2020: contributor

I’m counting four concept ACKs. For anyone looking to test it’s as easy as:

To build with CI run: FILE_ENV="./ci/test/00_setup_env_android.sh" ./ci/test_run_all.sh

To install on your (ARM64) device: docker cp ci_android:$PWD/src/qt/android/build/outputs/apk/debug/android-debug.apk . adb install android-debug.apk

MarcoFalke commented at 5:06 pm on November 10, 2020: member

This hasn’t been merged because we are still in feature freeze for 0.21, but this should be good to go into 22.0

MarcoFalke added this to the milestone 22.0 on Nov 10, 2020

DrahtBot added the label Needs rebase on Dec 4, 2020

in .cirrus.yml:154 in b0debb22cc outdated

148@@ -149,3 +149,11 @@ task:
149     CI_USE_APT_INSTALL: "no"
150     PACKAGE_MANAGER_INSTALL: "echo"  # Nothing to do
151     FILE_ENV: "./ci/test/00_setup_env_mac_host.sh"
152+
153+task:
154+  name: 'ARM64 Android APK  [GOAL: build]'

MarcoFalke commented at 7:51 am on December 4, 2020:

0  name: 'ARM64 Android APK [bionic]'

nit: the “goal” is no longer part of the task name, but we mention where it was cross-compiled from

MarcoFalke commented at 9:06 am on December 4, 2020: member

Tested compilation locally (via ci script) in podman:

 0Options used to compile and link:
 1  boost process = no
 2  multiprocess  = no
 3  with wallet   = yes
 4    with sqlite = yes
 5  with gui / qt = yes
 6    with qr     = yes
 7  with zmq      = yes
 8  with test     = yes
 9    with fuzz   = no
10  with bench    = yes
11  with upnp     = yes
12  use asm       = yes
13  sanitizers    = 
14  debug enabled = no
15  gprof enabled = no
16  werror        = no
17
18  target os     = android
19  build os      = linux-gnu
20
21  CC            = /bitcoin/depends/SDKs/android/ndk/21.1.6352462/toolchains/llvm/prebuilt/linux-x86_64/bin//aarch64-linux-android28-clang
22  CFLAGS        = -pthread -g -O2
23  CPPFLAGS      =   -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -I/bitcoin/depends/aarch64-linux-android/share/../include/  -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS
24  CXX           = /bitcoin/depends/SDKs/android/ndk/21.1.6352462/toolchains/llvm/prebuilt/linux-x86_64/bin//aarch64-linux-android28-clang++ -std=c++11
25  CXXFLAGS      =   -Wstack-protector -fstack-protector-all  -Wall -Wextra -Wgnu -Wformat -Wformat-security -Wvla -Wshadow-field -Wswitch -Wthread-safety -Wrange-loop-analysis -Wredundant-decls -Wunused-variable -Wdate-time -Wconditional-uninitialized -Wsign-compare -Woverloaded-virtual -Wunreachable-code-loop-increment  -Wno-unused-parameter -Wno-self-assign -Wno-unused-local-typedef -Wno-deprecated-register -Wno-implicit-fallthrough   -g -O2
26  LDFLAGS       =   -Wl,-z,relro -Wl,-z,now -pie  -L/bitcoin/depends/aarch64-linux-android/share/../lib 
27  ARFLAGS       = cr

icota force-pushed on Dec 4, 2020

DrahtBot removed the label Needs rebase on Dec 4, 2020

MarcoFalke commented at 4:01 pm on December 10, 2020: member

ACK 11be2d282965759edd87284d111f067168d937c2

BlockMechanic commented at 4:11 am on December 25, 2020: contributor

With static depends bui;ds now a thing, i think this one just moved up the list :) Will run tests later today.

DrahtBot added the label Needs rebase on Feb 28, 2021

icota force-pushed on Feb 28, 2021

DrahtBot removed the label Needs rebase on Feb 28, 2021

ghost commented at 10:39 pm on February 28, 2021: none

Concept ACK

This looks interesting. Not sure how did I miss this. Also interested to know difference and trade-offs in this solution compared to ABCore or NayutaCore and if more projects exist to use bitcoin core on Android.

icota force-pushed on Mar 1, 2021

DrahtBot added the label Needs rebase on Mar 16, 2021

icota force-pushed on Mar 16, 2021

DrahtBot removed the label Needs rebase on Mar 16, 2021

laanwj commented at 2:13 pm on March 16, 2021: member

This looks interesting. Not sure how did I miss this.

Haha I wonder this all the time. So much going on.

Looks like the new APK build is failing on the Qt compile. Maybe the intermediate Qt version bump got in the way?

0ERROR: Feature 'precompile_header' was enabled, but the pre-condition 'tests.precompile_header' failed.
1funcs.mk:273: recipe for target '/tmp/cirrus-ci-build/depends/work/build/aarch64-linux-android/qt/5.12.10-3ab96e3dc5c/./.stamp_configured' failed
2make: *** [/tmp/cirrus-ci-build/depends/work/build/aarch64-linux-android/qt/5.12.10-3ab96e3dc5c/./.stamp_configured] Error 3
3make: Leaving directory '/tmp/cirrus-ci-build/depends'

icota force-pushed on Mar 21, 2021

icota commented at 4:23 pm on March 21, 2021: contributor

@laanwj Thanks for the heads up. I rebased on top of 5.12 and tested on an Android 11 device. @prayank23 I can’t comment on other approaches. This PR aims to bring nothing less (and nothing more) than bitcoin-qt running on Android. #16883 is an attempt to make the GUI a bit more modern and usable but it’s parked for now as I can’t find the time to work on it.

BTW everyone check out the testnet IBD: Screenshot_20210321-161502 That seems quite fast, no? And the phone’s not even warm…. This PR might make practical sense after all. :sunglasses:

Qt: add Android packaging support

Introduce an android directory under qt and allow one to package bitcoin-qt for Android by running make apk.
Add bitcoin-qt Android build instructions.

ebfb10cb75

CI: add Android APK build script 7563720e30

CI: add Android APK build to cirrus ba46adaa1a

depends: disable Qt Vulkan support on Android 8e7ad4146d

depends: fix Qt precompiled headers bug

Fixed in 5.14, see QTBUG-85214

246774e264

icota force-pushed on Mar 21, 2021

MarcoFalke deleted a comment on Mar 24, 2021

MarcoFalke commented at 7:33 am on March 24, 2021: member

cr ACK 246774e264

laanwj commented at 6:00 pm on March 24, 2021: member

Code review ACK 246774e26459cb3652e308880abdd140e8e9d204

laanwj merged this on Mar 24, 2021

laanwj closed this on Mar 24, 2021

ghost commented at 6:50 pm on March 24, 2021: none

#16883 is an attempt to make the GUI a bit more modern and usable but it’s parked for now as I can’t find the time to work on it. @icota Thanks for working on this. Its merged :) #16883 looks interesting except one comment from @promag , my younger brother is also good with Android development but Kotlin. Will share things with him after testing.

sidhujag referenced this in commit 8a273cf022 on Mar 24, 2021

ryanofsky commented at 11:27 pm on March 24, 2021: member

Seeing CI failures in ARM64 Android APK [bionic] build that might be related to this:

https://cirrus-ci.com/task/5511109198217216?command=ci#L4420 https://cirrus-ci.com/task/5967779246374912?command=ci#L4426

 0make[1]: Entering directory '/tmp/cirrus-ci-build/src'
 1tar: --exclude=*/*: Cannot open: No such file or directory
 2tar: Error is not recoverable: exiting now
 3tar: --exclude=*/*: Cannot open: No such file or directory
 4tar: Error is not recoverable: exiting now
 5tar: --exclude=*/*: Cannot open: No such file or directory
 6tar: Error is not recoverable: exiting now
 7mkdir -p qt/android/libs/arm64-v8a
 8cp /tmp/cirrus-ci-build/depends/SDKs/android/ndk/21.1.6352462/toolchains/llvm/prebuilt/linux-x86_64/bin//../sysroot/usr/lib/aarch64-linux-android/libc++_shared.so qt/android/libs/arm64-v8a
 9tar xf  -C qt/android/src/ src/android/jar/src --strip-components=5
10tar: -C: Cannot open: No such file or directory
11tar: Error is not recoverable: exiting now
12Makefile:15443: recipe for target 'bitcoin_qt_apk' failed

icota commented at 7:28 am on March 25, 2021: contributor

@ryanofsky Yes, and this line is the culprit: https://github.com/bitcoin/bitcoin/blob/9217f9fe7351d9c36803456f88ae561e23f17ac9/src/Makefile.qt.include#L382

Is Cirrus perhaps wiping depends/sources from time to time?

achow101 commented at 5:20 pm on March 25, 2021: member

Is Cirrus perhaps wiping depends/sources from time to time?

I’m able to reproduce a similar error locally so I don’t think it is a cirrus issue.

icota commented at 7:56 pm on March 25, 2021: contributor

I can’t reproduce locally (unless I manually delete sources) but I’m seeing a pattern on the latest runs:

fails when run last: https://github.com/bitcoin/bitcoin/pull/21392/checks https://github.com/bitcoin/bitcoin/pull/16545/checks https://github.com/bitcoin/bitcoin/pull/19460/checks

succeds otherwise: https://github.com/bitcoin/bitcoin/pull/21422/checks https://github.com/bitcoin/bitcoin/pull/21526/checks https://github.com/bitcoin/bitcoin/pull/17227/checks

I’m drawing a blank here and I am also fine with disabling the CI APK build for the time being.

icota commented at 8:45 am on March 28, 2021: contributor

Trying to fix the issue in https://github.com/bitcoin/bitcoin/pull/21537

icota commented at 5:42 pm on March 28, 2021: contributor

Fixed by @MarcoFalke in https://github.com/bitcoin/bitcoin/pull/21541

in src/Makefile.qt.include:381 in 246774e264

377@@ -378,6 +378,20 @@ bitcoin_qt_clean: FORCE
378 
379 bitcoin_qt : qt/bitcoin-qt$(EXEEXT)
380 
381+APK_LIB_DIR = qt/android/libs/$(ANDROID_ARCH)

hebasto commented at 7:20 pm on May 25, 2021:

Shouldn’t the content of the qt/android directory be a subject of the cleanup in make clean or make distclean?

icota commented at 7:53 am on May 26, 2021:

It should :+1:

kittywhiskers referenced this in commit bd83a80411 on Nov 24, 2021

kittywhiskers referenced this in commit 5fe57a5091 on Nov 25, 2021

in src/qt/android/src/org/bitcoincore/qt/BitcoinQtActivity.java:19 in 246774e264

14+    public void onCreate(Bundle savedInstanceState)
15+    {
16+        final File bitcoinDir = new File(getFilesDir().getAbsolutePath() + "/.bitcoin");
17+        if (!bitcoinDir.exists()) {
18+            bitcoinDir.mkdir();
19+        }

hebasto commented at 0:33 am on December 3, 2021:

This code prevents the Intro dialog from showing at the first startup. Is it intended? If yes, why?

icota commented at 8:17 am on December 4, 2021:

TBH I don’t remember why I did this. Did you try without it? Maybe the dialog was showing incorrectly?

I didn’t at the time but I realise now that this makes using external storage impossible. Possibly related to https://github.com/bitcoin-core/gui/pull/496

hebasto commented at 9:53 pm on December 5, 2021: member

After bumping Qt to 5.15.2 (#23489) this stuff seems required to be updated to work properly. Massive changes were done in Qt 5.14 (e.g., https://github.com/qt/qtbase/commit/5bb178c479a247720fbc3fbb7f06a32b725193ac). @icota Could you have a look into this please?

hebasto commented at 9:23 pm on December 6, 2021: member

After bumping Qt to 5.15.2 (#23489) this stuff seems required to be updated to work properly. Massive changes were done in Qt 5.14 (e.g., qt/qtbase@5bb178c).

@icota Could you have a look into this please?

https://www.kdab.com/qt-for-android-better-than-ever-before/ seems related.

icota commented at 8:10 am on December 12, 2021: contributor

Thanks for letting me know @hebasto. Attempting to fix the issues in https://github.com/bitcoin-core/gui/pull/504

kittywhiskers referenced this in commit a596677934 on Dec 12, 2021

kittywhiskers referenced this in commit 0217aaadc6 on Jan 12, 2022

kittywhiskers referenced this in commit b7eb580add on Feb 7, 2022

kittywhiskers referenced this in commit 73395258a5 on Feb 17, 2022

kittywhiskers referenced this in commit bf3b2c7a48 on Feb 18, 2022

kittywhiskers referenced this in commit 37a5bd02ff on Apr 25, 2022

kittywhiskers referenced this in commit 30fafdcd97 on Apr 25, 2022

kittywhiskers referenced this in commit ae45b570c2 on Apr 26, 2022

UdjinM6 referenced this in commit cb75351543 on Apr 26, 2022

kittywhiskers referenced this in commit 1e3cd312f0 on Oct 16, 2022

DrahtBot locked this on Dec 12, 2022

Qt: Add Android packaging support #17227

Conflicts