ci: error: one or more PCH files were found, but they were invalid #28365

This has been reported by multiple contributors. I assume some sort of caching issue? Seems to happen intermittently. https://cirrus-ci.com/task/5028170181640192?logs=ci#L2664:

make[4]: Entering directory '/ci_container_base/depends/work/build/i686-pc-linux-gnu/qt/5.15.5-ba80ab1130c/qtbase/src/testlib'
make[5]: Entering directory '/ci_container_base/depends/work/build/i686-pc-linux-gnu/qt/5.15.5-ba80ab1130c/qtbase/src/plugins/platforminputcontexts'
make[5]: Entering directory '/ci_container_base/depends/work/build/i686-pc-linux-gnu/qt/5.15.5-ba80ab1130c/qtbase/src/plugins/platforms'
cc1plus: error: one or more PCH files were found, but they were invalid
cc1plus: error: use -Winvalid-pch for more information
cc1plus: error: one or more PCH files were found, but they were invalid
cc1plus: error: use -Winvalid-pch for more information
cc1plus: fatal error: .pch/Qt5Test: No such file or directory
cc1plus: fatal error: .pch/Qt5Test: No such file or directory
compilation terminated.
compilation terminated.
cc1plus: error: one or more PCH files were found, but they were invalid
cc1plus: error: use -Winvalid-pch for more information
cc1plus: fatal error: .pch/Qt5Test: No such file or directory
compilation terminated.
cc1plus: error: one or more PCH files were found, but they were invalid
cc1plus: error: use -Winvalid-pch for more information
make[4]: *** [Makefile:2113: .obj/qtestcase.o] Error 1
make[4]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile:2292: .obj/qtestlog.o] Error 1
make[4]: *** [Makefile:2435: .obj/qtesttable.o] Error 1
cc1plus: fatal error: .pch/Qt5Test: No such file or directory
compilation terminated.
make[5]: Entering directory '/ci_container_base/depends/work/build/i686-pc-linux-gnu/qt/5.15.5-ba80ab1130c/qtbase/src/plugins/platformthemes'
make[4]: *** [Makefile:2578: .obj/qtestdata.o] Error 1
cc1plus: error: one or more PCH files were found, but they were invalid
cc1plus: error: use -Winvalid-pch for more information
cc1plus: fatal error: .pch/Qt5Test: No such file or directory
compilation terminated.
make[4]: *** [Makefile:2944: .obj/qtestresult.o] Error 1
make[4]: Leaving directory '/ci_container_base/depends/work/build/i686-pc-linux-gnu/qt/5.15.5-ba80ab1130c/qtbase/src/testlib'
make[3]: *** [Makefile:298: sub-testlib-make_first] Error 2
make[3]: *** Waiting for unfinished jobs....

Not sure why Qt is even trying to use pch on a Linux build, as it was meant to be disabled by #20520.

Well, Qt configure summary shows:

  Using precompiled headers .............. yes

Not sure why Qt is even trying to use pch on a Linux build, as it was meant to be disabled by #20520.

The -pch option was not forced in that PR.

Ok, I realize the cache may not be atomic. It is a container volume, and if the task is cancelled and the container is killed, it will leave the cache at the current state, which may be "broken".

Usually this shouldn't be an issue, because depends (and ccache) will use some heuristics to check if a cache is valid, but in this case, the heuristic may have failed?

Obviously it is possible to manually delete the cache on the machine with podman volume rm --all, but I wonder if there is a better solution.

I guess volumes can be removed and stored on cirrus again, when run on cirrus. See commit 5fffff54e9fcf154c722dc421025a567fa0c5c97 and commit fad006fa0a8de7b527e9f7b50306921df7a2069a

I've double checked Qt build logs (current and historical) and can confirm that Qt build scripts are correct.

Haven't seen this for a while, so closing for now. Can re-open if it reappears.

Yes, this exact issue may not appear again, but the underlying issue (https://github.com/bitcoin/bitcoin/issues/28365#issuecomment-1697837720) may appear every time the qt depends is re-built:

• https://cirrus-ci.com/task/4517098299326464

A better title would be: ci: qt depends cache inside persistent volume is non-atomically cached (or similar)

Same underlying issue?

https://cirrus-ci.com/task/5064641198424064?logs=ci#L395

I guess this can be fixed by caching the built folder not the build folder?