Error when running gmake check on NetBSD 8.0 #14948

issue fanquake openend this issue on December 13, 2018
  1. fanquake commented at 1:58 pm on December 13, 2018: member

    After compiling Core on a fresh NetBSD 8.0 install (you can recreate using this Vagrantfile). The fsbridge_fstream test is failing:

     0vagrant ssh netbsd8
 1...
 2
 3src/test/test_bitcoin --log_level=all --run_test=fs_tests
 4
 5Running 1 test case...
 6Entering test module "Bitcoin Test Suite"
 7test/fs_tests.cpp(10): Entering test suite "fs_tests"
 8test/fs_tests.cpp(12): Entering test case "fsbridge_fstream"
 9unknown location(0): fatal error: in "fs_tests/fsbridge_fstream": boost::system::system_error: boost::filesystem::path codecvt to string: error
10test/fs_tests.cpp(12): last checkpoint: "fsbridge_fstream" test entry
11test/fs_tests.cpp(12): Leaving test case "fsbridge_fstream"; testing time: 30ms
12test/fs_tests.cpp(10): Leaving test suite "fs_tests"; testing time: 30ms
13Leaving test module "Bitcoin Test Suite"; testing time: 30ms
  2. fanquake added the label Linux/Unix on Dec 13, 2018
  3. MarcoFalke commented at 5:17 pm on December 13, 2018: member

    We assume that UTF-8 is set, e.g.

    0export LC_ALL=C.UTF-8
  4. MarcoFalke added the label Tests on Dec 13, 2018
  5. laanwj commented at 10:27 am on December 14, 2018: member

    The Python tests more or less require UTF-8 locale, though it shouldn’t be assuming that in the unit tests. Boost is supposed to convert UTF-8 filenames from/to the filesystem locale.

    Edit: right, it’s not able to represent those filenames, such as fs_tests_₿_🏃 in the filesystem locale, which defaults to ASCII? The failure makes sense.

  6. laanwj commented at 12:04 pm on December 14, 2018: member

    So as I see there’s three options here:

    • Force LC_ALL=C.UTF-8 (or more specifically): this ignores filesystem locale encoding completely and forces passing of UTF-8 to the OS for filenames. This will cause issues for OSes that truly don’t support UTF-8 filenames, or problems interacting with other software that does use the environment locale (the latter doesn’t really matter for the test).
    • Remove the funny characters from the test; drawback is that UTF-8 will no longer be tested on say, Windows, or Linux with UTF locale
    • Accept “codecvt to string: error” as a valid outcome in the test. Accept that the filesystem cannot support this character, there’s nothing to be done

    The last one is the most compatible thing to do, I think.

  7. ryanofsky commented at 8:17 pm on December 14, 2018: member

    I’m doubtful the codecvt errors come from accessing the filesystem. Unix filesystem APIs just take 8-byte character strings and then most filesystem implementations store them without additional translation.

    I would guess the error is coming from the line:

    https://github.com/bitcoin/bitcoin/blob/7a30e0f6c54443616fdc28b7080dcbc9b630ac77/src/test/fs_tests.cpp#L17

    which is attempting to append a wchar_t string to an char string and failing in the conversion. This might be caused by runtime local settings like @laanwj suggests, or it could be happening as a result of input-charset / exec-charset / wide-exec-charset preprocessor settings. But assuming this is a runtime issue and not a compile time issue, in addition to three options above we could also:

    1. Call fs::path::imbue like @ken2812221 did in #13877 so char <-> wchar_t conversions done by fs::path are always utf8 <-> utf16.

    2. Drop the failing test line, and no longer check what happens when you try to append a wchar_t string to an fs::path. Since we don’t actually use wchar_t strings in bitcoin code, this isn’t something we need to care about, even on windows where the fs::path class uses wchar_t strings, but only internally.

    I’d maybe lean to option 4, which could be implemented as easily as dropping the #ifndef WIN32 condition from ken’s #13877 fix. This is nice because it would be a simplification to current code, and it seems acceptable because on unix it is left to individual applications to decide what encodings they want to use for filenames. For example, while Qt applications use the system locale, glib applications use utf8 and ignore the system locale unless G_BROKEN_FILENAMES is set.

  8. laanwj commented at 6:26 am on December 15, 2018: member

    I’m doubtful the codecvt errors come from accessing the filesystem. Unix filesystem APIs just take 8-byte character strings and then most filesystem implementations store them without additional translation.

    The APIs probably accept everything, the problem is that the interpretation is up to the application and this is determined by the locale!

    UNIX has always had the concept of filesystem locales (in most cases set by environment variables). US/English was mostly ISO8859-1, Russian KOI8-R (or such). UTF-8 being the assumed default is very recent, and not the case for BSDs yet. BSDs likely default to ASCII because that’s the most compatible one.

    This is indeed a run-time, not a compile time issue. From what I understand from the error, Boost internally represents the paths as UTF-8 then fails to convert them to the local encoding to use with the operating system.

    Edit: But if you say “ignore the system locale” is acceptable for modern applications, that would certainly be easiest to do.

  9. dtzWill referenced this in commit 3e7d304d1d on Apr 30, 2019
  10. itoffshore commented at 7:17 pm on May 1, 2019: contributor

    I receive this same error on Alpine Linux (which does not support locale) with v0.18.0.

    Bitcoin under Alpine forces a locale of C so export LC_ALL=C.UTF-8 does not work:

     0Running tests: descriptor_tests from test/descriptor_tests.cpp
 1Running tests: fs_tests from test/fs_tests.cpp
 2Running tests: getarg_tests from test/getarg_tests.cpp
 3Running 1 test case...
 4Entering test module "Bitcoin Test Suite"
 5test/fs_tests.cpp(10): Entering test suite "fs_tests"
 6test/fs_tests.cpp(12): Entering test case "fsbridge_fstream"
 7unknown location(0): fatal error: in "fs_tests/fsbridge_fstream": boost::system::system_error: boost::filesystem::path codecvt to string: error
 8test/fs_tests.cpp(12): last checkpoint: "fsbridge_fstream" test entry
 9test/fs_tests.cpp(12): Leaving test case "fsbridge_fstream"; testing time: 48415us
10test/fs_tests.cpp(10): Leaving test suite "fs_tests"; testing time: 48455us
11Leaving test module "Bitcoin Test Suite"; testing time: 48585us
12
13*** 1 failure is detected in the test module "Bitcoin Test Suite"
14make[3]: *** [Makefile:14450: test/fs_tests.cpp.test] Error 1
15make[3]: *** Waiting for unfinished jobs....
16PASS: qt/test/test_bitcoin-qt
17============================================================================
18Testsuite summary for Bitcoin Core 0.18.0
19============================================================================
20# TOTAL: 1
21# PASS:  1
22# SKIP:  0
23# XFAIL: 0
24# FAIL:  0
25# XPASS: 0
26# ERROR: 0
27============================================================================
28make[4]: Leaving directory '/home/stuart/aports/community/bitcoin/src/bitcoin-0.18.0/src'
  11. practicalswift commented at 2:53 pm on July 5, 2019: contributor

    This seems to be causing some unnecessary confusion: perhaps we could handle this a bit more gracefully.

    If the UTF-8 locale requirement is not fulfilled then we could provide an error message giving guidance regarding the issue at hand (instead of boost::filesystem::path codecvt to string: error).

  12. meeDamian commented at 9:07 pm on August 18, 2019: contributor

    In case someone else stumbles on it on Alpine as well:

     0diff --git src/test/fs_tests.cpp src/test/fs_tests.cpp
 1index 93aee10bb..4a01e2ed8 100644
 2--- src/test/fs_tests.cpp
 3+++ src/test/fs_tests.cpp
 4@@ -14,7 +14,7 @@ BOOST_AUTO_TEST_CASE(fsbridge_fstream)
 5     fs::path tmpfolder = SetDataDir("fsbridge_fstream");
 6     // tmpfile1 should be the same as tmpfile2
 7     fs::path tmpfile1 = tmpfolder / "fs_tests_₿_🏃";
 8-    fs::path tmpfile2 = tmpfolder / L"fs_tests_₿_🏃";
 9+    fs::path tmpfile2 = tmpfolder / "fs_tests_₿_🏃";
10     {
11         fsbridge::ofstream file(tmpfile1);
12         file << "bitcoin";

    Save ^this as utf8-fs-test.patch, and:

    0patch -p0 < utf8-fs-test.patch

    Run make check again.

    Above works for automated builds. To make it pass manually, just delete that letter L from src/test/fs_tests.cpp.

  13. jonasnick referenced this in commit 4784264bb2 on Oct 5, 2019
  14. laanwj commented at 12:47 pm on October 9, 2019: member
    Huh, I don’t think the L should be there at all. We don’t use wchar_t at all in our code (except for some windows-specific API things), all internal strings in Bitcoin Core are 8-bit UTF-8, and we assume boost to handle that as UTF-8. If removing it fixes the test, let’s do that.
  15. carnhofdaki commented at 2:08 pm on October 9, 2019: contributor

    Huh, I don’t think the L should be there at all. We don’t use wchar_t at all in our code (except for some windows-specific API things), all internal strings in Bitcoin Core are 8-bit UTF-8, and we assume boost to handle that as UTF-8. If removing it fixes the test, let’s do that. @laanwj Yes, removing the L really fixes it.

  16. carnhofdaki referenced this in commit 4c58943cb3 on Oct 9, 2019
  17. carnhofdaki referenced this in commit e6208c8a34 on Oct 9, 2019
  18. carnhofdaki referenced this in commit 4375323fb2 on Oct 10, 2019
  19. laanwj referenced this in commit b67ca4ee73 on Oct 10, 2019
  20. carnhofdaki referenced this in commit 000198d248 on Oct 15, 2019
  21. carnhofdaki referenced this in commit d48f664440 on Oct 15, 2019
  22. laanwj referenced this in commit b544eb6370 on Oct 15, 2019
  23. carnhofdaki commented at 10:20 am on October 15, 2019: contributor

    @fanquake I guess this can be closed now.

    Thank you all.

  24. laanwj closed this on Oct 15, 2019
  25. carnhofdaki referenced this in commit c90a42e290 on Oct 16, 2019
  26. carnhofdaki referenced this in commit bd9d40dbbd on Oct 16, 2019
  27. laanwj referenced this in commit 3834d3d121 on Oct 16, 2019
  28. sidhujag referenced this in commit 598fed0514 on Oct 16, 2019
  29. barrystyle referenced this in commit 0ae325553f on Nov 11, 2019
  30. HashUnlimited referenced this in commit 4f27058988 on Nov 17, 2019
  31. fxtc referenced this in commit b816389b73 on Nov 25, 2019
  32. fxtc referenced this in commit 4f9cec8076 on Nov 25, 2019
  33. Fabcien referenced this in commit d7795abdf9 on Dec 24, 2020
  34. UdjinM6 referenced this in commit 144bf39039 on Jun 22, 2021
  35. kittywhiskers referenced this in commit 6a11e06372 on Jun 24, 2021
  36. PastaPastaPasta referenced this in commit 0f4d2e319c on Jun 27, 2021
  37. PastaPastaPasta referenced this in commit 7fb3cd6dc6 on Jun 28, 2021
  38. PastaPastaPasta referenced this in commit 0fd173db05 on Jun 29, 2021
  39. PastaPastaPasta referenced this in commit bfc1b562c1 on Jul 1, 2021
  40. PastaPastaPasta referenced this in commit 45405955a7 on Jul 1, 2021
  41. UdjinM6 referenced this in commit fe49674f2f on Jul 5, 2021
  42. PastaPastaPasta referenced this in commit eb1eb9d9ab on Jul 12, 2021
  43. PastaPastaPasta referenced this in commit 37ebcb1e90 on Jul 13, 2021
  44. PastaPastaPasta referenced this in commit e4457ce2ef on Jul 13, 2021
  45. MarcoFalke locked this on Dec 16, 2021


fanquake MarcoFalke laanwj ryanofsky itoffshore practicalswift meeDamian carnhofdaki

Labels
Linux/Unix Tests

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-03 15:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me