ci: spurious output / errors in IWYU job #35361

https://github.com/bitcoin/bitcoin/actions/runs/26292243180/job/77395542650: Looks like IWYU is trying to run on libmultiprocess related code, and failing, but we shouldn't be trying to lint a subtree in the first place:

 /home/admin/actions-runner/_work/_temp/src/ipc/libmultiprocess/test/mp/test/test.cpp:5:10: fatal error: 'mp/test/foo.capnp.h' file not found
    5 | #include <mp/test/foo.capnp.h>
      |          ^~~~~~~~~~~~~~~~~~~~~

Along with other sporadic error output:

[1](https://github.com/bitcoin/bitcoin/actions/runs/26292243180/job/77395542650#step:11:11661)
error: no such file or directory: '/home/admin/actions-runner/_work/_temp/build/src/ipc/capnp/common.capnp.c++'
error: no input files
error: unable to handle compilation, expected exactly one compiler job in ''
error: no such file or directory: '/home/admin/actions-runner/_work/_temp/build/src/ipc/capnp/common.capnp.proxy-client.c++'
error: no input files
error: unable to handle compilation, expected exactly one compiler job in ''
error: no such file or directory: '/home/admin/actions-runner/_work/_temp/build/src/ipc/capnp/common.capnp.proxy-server.c++'
error: no input files
error: unable to handle compilation, expected exactly one compiler job in ''
error: no such file or directory: '/home/admin/actions-runner/_work/_temp/build/src/ipc/capnp/common.capnp.proxy-types.c++'
error: no input files

Ideally we'd have no sporadic errors / output, and wouldn't be trying to run linters/tools over subtrees at all.

I think that any errors in CI output/tooling should also be getting turned into real errors, so that they aren't missed/ignored?

cc @hebasto @ryanofsky

https://github.com/bitcoin/bitcoin/actions/runs/26292243180/job/77395542650 seems to point to a successful CI run, so wondering if something might have fixed this, or if it succeeded because the errors are sporadic.

It's also not clear to me why these errors would be sporadic, unless maybe they happen due to build jobs running in parallel and a dependency not being specified. Would be curious for an update.

I think the link should have been https://github.com/bitcoin/bitcoin/actions/runs/26292243180/job/77395542650#step:11:7602 or https://github.com/bitcoin/bitcoin/actions/runs/26292243180/job/77395542650#step:11:11354 (with the line number in it)? (grep for fatal error)

Related: https://github.com/bitcoin-core/libmultiprocess/issues/284.

I think the link should have been https://github.com/bitcoin/bitcoin/actions/runs/26292243180/job/77395542650#step:11:7602...

Regarding the fatal error: 'node/data/ip_asn.dat.h' file not found issue, please see https://gitlab.kitware.com/cmake/cmake/-/work_items/27862.

seems to point to a successful CI run, so wondering if something might have fixed this

I'm finally understanding the problem now. The issue being reported is an error that shows up in the CI log files but does NOT cause the CI job to fail. You can see it by clicking the CI link in the issue description, clicking gear icon, choosing "View raw logs" and then searching for "fatal error." (For whatever reason searching for "fatal error" in the search box on the GitHub actions page does not seem to work for me.)

Also, the issue is not sporadic. You can see it in CI jobs running on the current master branch.

Different fixes are required for the different errors:

src/init.cpp:124:10: fatal error: 'node/data/ip_asn.dat.h' file not found
src/rpc/net.cpp:21:10: fatal error: 'node/data/ip_asn.dat.h' file not found
src/ipc/libmultiprocess/include/mp/proxy-io.h:11:10: fatal error: 'mp/proxy.capnp.h' file not found
src/ipc/libmultiprocess/test/mp/test/test.cpp:5:10: fatal error: 'mp/test/foo.capnp.h' file not found
src/ipc/capnp/protocol.cpp:7:10: fatal error: 'ipc/capnp/init.capnp.h' file not found
src/ipc/capnp/mining-types.h:9:10: fatal error: 'ipc/capnp/common.capnp.proxy-types.h' file not found
src/ipc/test/ipc_test.cpp:6:10: fatal error: 'ipc/capnp/mining.capnp.h' file not found
src/bench/asmap.cpp:8:10: fatal error: 'node/data/ip_asn.dat.h' file not found

• Fixing the ip_asn.dat.h error will require either:

  1. Not marking bitcoin_node library EXCLUDE_FROM_ALL (I'm actually not sure why it is marked EXCLUDE_FROM_ALL) or
  2. Adding an add_custom_target line to target_raw_data_sources and generating the header before invoking IWYU like #35468 or
  3. A change to cmake like ones being discussed in https://gitlab.kitware.com/cmake/cmake/-/work_items/27862.

• Fixing the libmultiprocess errors will require either

  1. Excluding entire libmultiprocess library from clang-tidy and IWYU checks with set(CMAKE_EXPORT_COMPILE_COMMANDS OFF) as suggested #35418 (review).
  2. Excluding only mptest from clang-tidy and IWYU checks with #35418 plus #35454 to fix the error when BUILD_TESTS is off, as reported in #35523.
  3. Excluding only mptest from clang-tidy and IWYU checks with an alternate approach suggested #35523 (review) which is equivalent but probably more confusing.
  4. Running IWYU normally with #35468.

• I believe the only fix that's been attempted to fix errors inside of src/ipc/ but outside of src/ipc/libmultiprocoess/ is #35468 but I think other approaches could be possible like setting EXPORT_COMPILE_COMMANDS there or filtering the compilation database with jq.

IMO, the best approach is to extend #35468 to generate the headers needed by the IWYU job before IWYU runs and stop trying to override EXPORT_COMPILE_COMMANDS and EXCLUDE_FROM_ALL settings to control what files the IWYU job sees and checks.