ci: Move Windows MSVC build from AppVeyor to Cirrus #21551

pull hebasto wants to merge 2 commits into bitcoin:master from hebasto:210330-cirrus changing 5 files +75 −95
  1. hebasto commented at 3:58 am on March 30, 2021: member

    This PR:

    • gets rid off unreliable AppVeyor CI
    • places all CI configs in one place
    • allows to enable functional tests in the future (using a persistent worker)
    no populated vcpkg cache populated vcpkg cache
    Screenshot from 2021-09-02 15-47-04 Screenshot from 2021-09-02 14-06-26

    Currently, AppVeyor builds take about 44..48 minutes.

  2. fanquake added the label Tests on Mar 30, 2021
  3. hebasto commented at 5:47 am on March 30, 2021: member
    The warmed up vcpkg cache reduces build time by about 40 minutes.
  4. MarcoFalke commented at 6:16 am on March 30, 2021: member
    Still takes longer than the appveyor 38 minutes right now?
  5. hebasto commented at 6:17 am on March 30, 2021: member

    Still takes longer than the appveyor 38 minutes right now?

    Right.

    The time is spent on installing msbuild tools, that are preinstalled on AppVeyor.

  6. DrahtBot commented at 0:49 am on April 7, 2021: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #22899 (ci: Build and cache static Qt instead of downloading a pre-built one by hebasto)
    • #22890 (doc: Replace a link to Qt precompiled binaries with compile instructions by hebasto)
    • #20744 (Use std::filesystem. Remove Boost Filesystem & System by fanquake)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  7. DrahtBot added the label Needs rebase on Apr 11, 2021
  8. hebasto commented at 8:20 am on April 11, 2021: member

    @MarcoFalke

    Still takes longer than the appveyor 38 minutes right now?

    Does it make sense to move this task into the DrahtBot infrastructure? Or just close it?

  9. DrahtBot commented at 8:32 am on April 11, 2021: member
    I don’t run Windows and macOS, so I can’t run those native tasks
  10. hebasto closed this on Apr 11, 2021

  11. hebasto reopened this on Apr 11, 2021

  12. hebasto force-pushed on Apr 11, 2021
  13. hebasto commented at 7:55 pm on April 11, 2021: member

    Still takes longer than the appveyor 38 minutes right now?

    It takes 27 minutes now.

    The time is spent on installing msbuild tools, that are preinstalled on AppVeyor.

    Fixed by using visualstudio2019 image.

  14. DrahtBot removed the label Needs rebase on Apr 11, 2021
  15. MarcoFalke referenced this in commit 30a86bb814 on Apr 20, 2021
  16. fanquake commented at 3:32 am on September 2, 2021: member

    What is the status of this? Is the intention to replace Appveyor? If not, do we need two Windows builds?

    An empty PR description is not really useful.

  17. MarcoFalke commented at 6:55 am on September 2, 2021: member

    I started this because Cirrus CI seems the most reliable CI provider, so having all configs on them seemed preferable.

    I guess the status here is: Needs review?

  18. MarcoFalke commented at 6:55 am on September 2, 2021: member
    (Also needs appveyor removed)
  19. hebasto marked this as a draft on Sep 2, 2021
  20. hebasto renamed this:
    [PoC] ci: Add Windows task to Cirrus CI
    ci: Move Windows MSVC build from AppVeyor to Cirrus
    on Sep 2, 2021
  21. hebasto force-pushed on Sep 2, 2021
  22. hebasto marked this as ready for review on Sep 2, 2021
  23. hebasto commented at 12:22 pm on September 2, 2021: member

    @sipsorcery

    It seems Qt5.12.11_x64_static_vs2019_16101.zip is not compatible with the msbuild in the cirrusci/windowsservercore:visualstudio2019 image:

    0LINK : fatal error C1900: Il mismatch between 'P1' version '20210202' and 'P2' version '20210113' [C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\build_msvc\bitcoin-qt\bitcoin-qt.vcxproj]
    1LINK : fatal error LNK1257: code generation failed [C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\build_msvc\bitcoin-qt\bitcoin-qt.vcxproj]
    2LINK : fatal error LNK1327: failure during running link.exe [C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\build_msvc\bitcoin-qt\bitcoin-qt.vcxproj]
    

    So, I switched back to Qt5.12.10_x64_static_vs2019_1694.zip for now.

    May I ask you to build Qt5.12.11_x64_static_vs2019_1694.zip – the recent Qt 5.12.11 with an older msbuild?

  24. sipsorcery commented at 12:32 pm on September 2, 2021: member

    May I ask you to build Qt5.12.11_x64_static_vs2019_1694.zip – the recent Qt 5.12.11 with an older msbuild?

    Sure, any idea where I can find the software versions installed on the cirrusci image you’re using?

  25. hebasto commented at 12:36 pm on September 2, 2021: member

    May I ask you to build Qt5.12.11_x64_static_vs2019_1694.zip – the recent Qt 5.12.11 with an older msbuild?

    Sure, any idea where I can find the software versions installed on the cirrusci image you’re using?

    https://github.com/cirruslabs/docker-images-windows

  26. hebasto commented at 12:50 pm on September 2, 2021: member

    @sipsorcery

    The msbuild -version output within the cirrusci/windowsservercore:visualstudio2019 Cirrus image:

    0Microsoft (R) Build Engine version 16.9.0+5e4b48a27 for .NET Framework
    1Copyright (C) Microsoft Corporation. All rights reserved.
    216.9.0.16703
    
  27. sipsorcery commented at 12:55 pm on September 2, 2021: member

    The msvc -version output within the cirrusci/windowsservercore:visualstudio2019 Cirrus image:

    0Microsoft (R) Build Engine version 16.9.0+5e4b48a27 for .NET Framework
    1Copyright (C) Microsoft Corporation. All rights reserved.
    216.9.0.16703
    

    Thanks for doing that. I was trying to track it down via the docker files. I already have that version of msbuild installed so will attempt the Qt build tonight (it kills my machine so overnight job).

  28. hebasto force-pushed on Sep 2, 2021
  29. hebasto force-pushed on Sep 2, 2021
  30. hebasto commented at 3:22 pm on September 2, 2021: member

    @sipsorcery

    To make debugging a bit easier in the future, I’ve added the following script:

    0  build_environment_script:
    1    - choco list --localonly
    2    - msbuild -version
    
  31. hebasto force-pushed on Sep 4, 2021
  32. hebasto commented at 10:21 am on September 4, 2021: member

    @MarcoFalke @fanquake @sipsorcery

    This PR is ready for reviewing now :tiger2:

  33. sipsorcery commented at 12:23 pm on September 4, 2021: member

    ACK ca9b9b9a2c0dd7ab767788be6512cfc34f1f6ee4.

    Makes sense to consolidate CI platforms.

    Tested that latest msbuild version 16.11.0.36601 still works with the Qt5.12.11_x64_static_vs2019_160900 binaries.

    One potential pitfall will be the vcpkg caching. We had some issues with that in the past, namely whan a PR added or changed a dependency and the vcpkg couldn’t deal with it. Since then the vcpkg manifest and binary caching features have been added. They may solve this problem.

  34. hebasto added the label Windows on Sep 4, 2021
  35. fanquake commented at 1:56 am on September 6, 2021: member
    ACK after rebase
  36. hebasto force-pushed on Sep 6, 2021
  37. hebasto commented at 2:25 am on September 6, 2021: member
    Rebased ca9b9b9a2c0dd7ab767788be6512cfc34f1f6ee4 -> da8810e4f69b43e96ddf69400bec503451a0e847 (pr21551.06 -> pr21551.07) due to the conflict with #22861.
  38. in .cirrus.yml:79 in da8810e4f6 outdated
    72@@ -89,6 +73,64 @@ task:
    73   env:
    74     << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
    75 
    76+task:
    77+  name: "Win64 [unit tests, no functional tests] [msvc]"
    78+  windows_container:
    79+    cpu: 8
    


    MarcoFalke commented at 6:45 am on September 6, 2021:
    why 8 when the average usage is 2?

    hebasto commented at 3:49 am on September 7, 2021:

    With cpu: 4:

    Screenshot from 2021-09-07 06-45-03

    And the total task time with a populated vcpkg cache:

    Screenshot from 2021-09-07 06-46-09

  39. in .cirrus.yml:77 in da8810e4f6 outdated
    72@@ -89,6 +73,64 @@ task:
    73   env:
    74     << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
    75 
    76+task:
    77+  name: "Win64 [unit tests, no functional tests] [msvc]"
    


    MarcoFalke commented at 6:45 am on September 6, 2021:
    0  name: "Win64 native [unit tests, no functional tests] [msvc]"
    

    hebasto commented at 3:27 am on September 7, 2021:
    Thanks! Updated.
  40. in .cirrus.yml:82 in da8810e4f6 outdated
    77+  name: "Win64 [unit tests, no functional tests] [msvc]"
    78+  windows_container:
    79+    cpu: 8
    80+    memory: 7424Mi
    81+    image: cirrusci/windowsservercore:visualstudio2019
    82+  << : *CREDITS_TEMPLATE
    


    MarcoFalke commented at 6:45 am on September 6, 2021:
    This looks expensive, especially with 8 CPUs and without BASE_TEMPLATE to skip

    hebasto commented at 3:28 am on September 7, 2021:
  41. MarcoFalke commented at 6:46 am on September 6, 2021: member
    Missing BASE_TEMPLATE to merge the latest branch?
  42. hebasto force-pushed on Sep 6, 2021
  43. hebasto marked this as a draft on Sep 6, 2021
  44. hebasto force-pushed on Sep 6, 2021
  45. hebasto force-pushed on Sep 6, 2021
  46. ci: Add Windows task to Cirrus CI 1fb70793b2
  47. ci: Drop AppVeyor CI integration 97292b1914
  48. hebasto force-pushed on Sep 7, 2021
  49. hebasto marked this as ready for review on Sep 7, 2021
  50. hebasto commented at 3:24 am on September 7, 2021: member

    Updated da8810e4f69b43e96ddf69400bec503451a0e847 -> 97292b19140db32c6d85d63b70382e7bf60a55c4 (pr21551.07 -> pr21551.08, diff):

  51. MarcoFalke commented at 8:01 am on September 7, 2021: member
    ack
  52. hebasto commented at 8:34 am on September 7, 2021: member

    @sipsorcery

    Another (final?) review? :smile:

  53. sipsorcery commented at 8:39 am on September 7, 2021: member
    re-ACK 97292b19140db32c6d85d63b70382e7bf60a55c4.
  54. MarcoFalke merged this on Sep 7, 2021
  55. MarcoFalke closed this on Sep 7, 2021

  56. in .cirrus.yml:101 in 97292b1914
     96+    IgnoreWarnIntDirInTempDetected: 'true'
     97+  merge_script:
     98+    - git config --global user.email "ci@ci.ci"
     99+    - git config --global user.name "ci"
    100+    - git config core.filemode false
    101+    - git reset --hard
    


    MarcoFalke commented at 9:27 am on September 7, 2021:
    maybe add a comment why this is needed?

    hebasto commented at 9:41 am on September 7, 2021:
    Sure, I’m on it.

    hebasto commented at 10:32 am on September 7, 2021:
    Done in #22909.
  57. hebasto deleted the branch on Sep 7, 2021
  58. in .cirrus.yml:141 in 97292b1914
    136+  build_script:
    137+    - cd $env:CIRRUS_WORKING_DIR
    138+    - python build_msvc\msvc-autogen.py
    139+    - msbuild build_msvc\bitcoin.sln -property:Configuration=Release -maxCpuCount -verbosity:minimal -noLogo
    140+  unit_tests_script:
    141+    - src\test_bitcoin.exe
    


    MarcoFalke commented at 9:35 am on September 7, 2021:
    why remove the -l test_suite?

    hebasto commented at 9:40 am on September 7, 2021:
    :man_shrugging: Going to re-add it right now.

    hebasto commented at 10:32 am on September 7, 2021:
    Done in #22909.
  59. MarcoFalke referenced this in commit e0ca5d0513 on Sep 7, 2021
  60. sidhujag referenced this in commit b0c6def933 on Sep 7, 2021
  61. sidhujag referenced this in commit e7a34552b4 on Sep 7, 2021
  62. sidhujag referenced this in commit 7453000c8b on Sep 7, 2021
  63. DrahtBot locked this on Sep 7, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-04 22:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me