Appveyor ci fails again

MarcoFalke commented at 5:20 pm on January 21, 2020: member

Happens on all recent builds. Seemingly unrelated to any code changes.

Last pass: https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/30240770 First fail: https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/30241275 First fail on master: https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/30241926

The first fail happened on the 0.19 branch. Is the appveyor cache shared across branches?

Otherwise it could be a msvc update or another dependency that broke it. If so, I wish we figure out a way to freeze the dependencies and compileres in some way.

 0C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(330,10): error C2976: 'std::deque': too few template arguments [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]
 1C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(330,1): error C2079: 'last_few' uses undefined class 'std::deque' [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]
 2C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(341,1): error C2672: 'begin': no matching overloaded function found [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]
 3C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(341,1): error C2893: Failed to specialize function template 'unknown-type std::begin(_Container &)' [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]
 4C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(341,1): error C2784: 'const _Elem *std::begin(std::initializer_list<_Elem>) noexcept': could not deduce template argument for 'std::initializer_list<_Elem>' from 'int' [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]
 5C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(341,1): error C2672: 'end': no matching overloaded function found [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]
 6C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(341,1): error C2893: Failed to specialize function template 'unknown-type std::end(_Container &)' [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]
 7C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(341,1): error C2784: 'const _Elem *std::end(std::initializer_list<_Elem>) noexcept': could not deduce template argument for 'std::initializer_list<_Elem>' from 'int' [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]
 8C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(341,1): error C3536: '<begin>$L0': cannot be used before it is initialized [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]
 9C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(341,1): error C3536: '<end>$L0': cannot be used before it is initialized [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]
10C:\projects\bitcoin\src\test\cuckoocache_tests.cpp(341,1): error C2100: illegal indirection [C:\projects\bitcoin\build_msvc\test_bitcoin\test_bitcoin.vcxproj]

MarcoFalke added the label Bug on Jan 21, 2020

MarcoFalke added the label Tests on Jan 21, 2020

fanquake assigned sipsorcery on Jan 21, 2020

sipsorcery commented at 10:24 pm on January 21, 2020: member

I see the vcpkg version of Boost has been bumped from 1.71.0 to 1.72.0. It’s almost certainly the culprit.

A potential solution to the version issue is to use vcpkg’s overlay ports option. It’s described in this blog post and this specification document.

In our case the mechanism would be something like:

Once off (and when we decide to update dependencies):

Create a new repo to hold vcpkg port files for packages required by Bitcoin Core,
Set all port files to desired versions

Appveyor Job:

Clone the vcpkg port files repo,
Execute the vcpkg install from the existing c:\tools\vcpkg directory using the --overlay-ports option:

Example:

0vcpkg install protobuf:x64-windows-static --overlay-ports=..\vcpkg-custom-ports\ports\protobuf

This solution will require a new git repo.

There are other options such as zipping the vcpkg install directory with the desired versions and downloading it in the same manner I’ve done for the Qt binaries.

fanquake closed this on Jan 22, 2020

sipsorcery commented at 9:34 am on January 22, 2020: member

Even though the appveyor job is fixed for now we may still want to discuss stabilising the versions of the vcpkg installed dependencies.

fanquake commented at 9:34 am on January 22, 2020: member

@sipsorcery fair call. Would like like to open a new issue containing the relevant information from here?

sipsorcery commented at 9:39 am on January 22, 2020: member

It’d be worth waiting and seeing what @MarcoFalke thinks first. The solutions I’ve suggested require a new git repo or offsite binaries. No point going down that path if it raises other concerns.

MarcoFalke commented at 2:26 pm on January 22, 2020: member

How large would the directory ..\vcpkg-custom-ports\ports\boost or similar be? From reading the docs it can even be in the main repo, mabye under build_msvc/vcpkg-custom-ports/...?

MarcoFalke reopened this on Jan 22, 2020

sipsorcery commented at 6:50 pm on January 22, 2020: member

The custom ports directory would be tiny, few hundred KB maybe up to 1MB. It only holds the port files which state how to build the dependencies AND only needs to hold the port files for the dependencies being used rather than the full 1283 ports.

The binaries would continue to be cached by appveyor in the c:\vcpkg\install. The advantage now being that the only trigger to cause a dependency to update is if a port file changes under the custom ports directory.

If the consensus is that vcpkg port files can sit underneath the build_msvc directory then I think using this option is a no-brainer. It gives complete control over vcpkg dependency versions and will alleviate a portion of the appveyor CI breakages.

MarcoFalke added the label Brainstorming on Jan 22, 2020

MarcoFalke removed the label Bug on Jan 22, 2020

MarcoFalke commented at 6:55 pm on January 22, 2020: member

1MB sounds a bit too large for the main repo, which everyone has to clone.

I’ll put this on “brainstorming” for now.

sipsorcery commented at 8:06 pm on January 22, 2020: member

I’ve done some preliminary testing and the required port files look like they will be at the lower end of my estimate and take around 160KB. I’m having difficulties with installing Boost though. Our 5 boost packages expand to 63 required packages and they don’t seem to like being installed from a custom directory.

If I do resolve the boost install then it should be less than 200KB required under build_msvc.

sipsorcery commented at 9:34 pm on January 22, 2020: member

Final figure 296KB.

MarcoFalke commented at 9:41 pm on January 22, 2020: member

Thanks for looking into this. 296KB of binary files in a git repo that is cloned 1000 times every day still seems a bit much. Also, if the ports are going to be bumped this will only increase, because git will need to serve both versions.

I think a separate repo could make sense.

sipsorcery commented at 9:49 pm on January 22, 2020: member

They are text files that have been cherry picked from the vcpkg ports based on the current Bitcoin Core dependencies.

I’m happy to create a git repo under my personal account for the port files (only text files reqd. no binaries). Given the appveyor job already pulls binaries from my qt_win_binary repo it would only a linear increase in risk.

MarcoFalke commented at 10:11 pm on January 22, 2020: member

Ah, text files compress well and git handles them better, so this might be fine.

hebasto commented at 11:35 am on January 27, 2020: member

Does #18001 close this issue?

fanquake closed this on Jan 27, 2020

PastaPastaPasta referenced this in commit 175ea9b53b on Jun 27, 2021

PastaPastaPasta referenced this in commit 488e666e0d on Jun 28, 2021

PastaPastaPasta referenced this in commit 451cc41bec on Jun 29, 2021

PastaPastaPasta referenced this in commit 9feb0f2318 on Jul 1, 2021

PastaPastaPasta referenced this in commit bc7867e520 on Jul 1, 2021

PastaPastaPasta referenced this in commit 867c46894d on Jul 14, 2021

PastaPastaPasta referenced this in commit 30a5b5fe0b on Jul 14, 2021

PastaPastaPasta referenced this in commit 25f3f66655 on Jul 15, 2021

DrahtBot locked this on Feb 15, 2022

Appveyor ci fails again #17976