Locale should default to C / Posix Environment #4147

Busybox does not use locales - bitcoin should set a default C / Posix environment. The following locale error:

EXCEPTION: St13runtime_error       
locale::facet::_S_create_c_locale name not valid       
bitcoin in AppInit()       

terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid

Can be fixed with the following patch #4147 (comment)

The above patch is almost identical to the patch I proposed for Namecoin to fix this issue (Windows builds are unaffected):https://github.com/namecoin/namecoin/issues/72#issuecomment-42520429

Overwriting LC_ALL with a fixed value is quite an ugly solution (busybox may not use locales, but we do).

Isn't there something more specific that can be done about the boost error?

I tried catching the error & then setting the environment without success

Is this caused by the use time_input_facet in DecodeDumpTime in rpcdump.cpp?

This error is thrown in AppInit() - setting LC_ALL in AppInit2() did not fix the problem

If your system doesn't use locales, then setlocale(any, "" or NULL) should simply return "C", and setlocale(any, anything else) should return NULL. If that is causing problems in boost or bitcoind, those problems should be fixed rather than forcing the "C" locale.

this was in bitcoinrpc.cpp in the namecoin sources but it did not force a C locale

string locale(setlocale(LC_TIME, NULL)); 

You keep talking about namecoin.

Do we, in the current master version of bitcoin, have anything that generates this error?

If so, where is it raised? Where does the exception originate from? It must be some usage of boost. Can we avoid that, maybe by passing in our own locale object instead of using the system one?

Edit: I see we indeed use setlocale here: https://github.com/bitcoin/bitcoin/blob/master/src/rpcprotocol.cpp#L52 . That's not thread-safe at all. But it appears that the exception is raised at at start-up, so it cannot be caused by that.

The exact error in Bitcoin is:

terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid
Aborted

And which code throws it?

just running 'bitcoind --help' is enough to generate the error - any command will generate the same error on a machine without locales.

A default C locale should be set.

I don't want an ugly workaround hack unless we know what code throws this error, and that there is no other way to avoid the problem.

I don't understand how this happens in the first place: boost raises this error if it cannot find the selected locale. As @luke-jr says, the "C" locale is selected by default if you have not set another locale (so if LC_* is not set). Where does it get this information from? Can you show your environment?

uclibc [~/packages/testing/x86]$ printenv
PAGER=less
PS1=\[\e[1;32m\]\h [\[\e[0m\]\w\[\e[1;32m\]]$ \[\e[0m\]
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=en_US.UTF-8
SHELL=/bin/ash
CHARSET=UTF-8
EDITOR=nano

A C environment is not being set by default on bitcoin or namecoin - as already noted 'setlocale(LC_TIME, NULL)' does not force a C environment on Busybox (Alpine Linux)

LANG=en_US.UTF-8

This may be the culprit if you don't have the en_US locale. Can you try undefining it?

It seems the way most software handles unknown locales is to ignore them. We should do the same...

Removing 'LANG=en_US.UTF-8' fixed the issue - so setlocale NULL does work - just need to handle unknown locales

I managed to reproduce it by setting LC_ALL to a random value. It looks like boost::filesystem is raising this, line 975 in util.cpp is where it happens.

[#0](/bitcoin-bitcoin/0/)  0x00007ffff66c1a30 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[#1](/bitcoin-bitcoin/1/)  0x00007ffff6713677 in std::__throw_runtime_error(char const*) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[#2](/bitcoin-bitcoin/2/)  0x00007ffff66df0c4 in std::locale::facet::_S_create_c_locale(__locale_struct*&, char const*, __locale_struct*) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[#3](/bitcoin-bitcoin/3/)  0x00007ffff66d42e9 in std::locale::_Impl::_Impl(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[#4](/bitcoin-bitcoin/4/)  0x00007ffff66d530d in std::locale::locale(char const*) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[#5](/bitcoin-bitcoin/5/)  0x00007ffff79ce2c2 in boost::filesystem::path::codecvt() () from /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.54.0
[#6](/bitcoin-bitcoin/6/)  0x00005555556e5bea in path<char*> (source=<synthetic pointer>, this=0x7fffffffd750) at /usr/include/boost/filesystem/path.hpp:139
[#7](/bitcoin-bitcoin/7/)  GetDefaultDataDir () at util.cpp:975
[#8](/bitcoin-bitcoin/8/)  0x00005555556e79ed in GetDataDir (fNetSpecific=<optimized out>) at util.cpp:1014
[#9](/bitcoin-bitcoin/9/)  0x000055555558bb0f in AppInit (argc=2, argv=0x7fffffffddc8) at bitcoind.cpp:68
[#10](/bitcoin-bitcoin/10/) 0x0000555555582a9a in main (argc=2, argv=0x7fffffffddc8) at bitcoind.cpp:180

So it somehow uses the locale to determine the codec for filename conversion... you'd say it only needs the second part (UTF-8) but meh...

Are filename encodings really supposed to be based on the current locale?!

@luke-jr Yes, that's one of the things that used to be determined by the locale. Though these days, everything is UTF-8.

This seems to be the only workaround that works, up until now

#ifndef WIN32
    try
    {
        boost::filesystem::path::codecvt() // Raises runtime error if current locale is invalid
    } catch(std::runtime_error &e)
    {
        setenv("LC_ALL", "C", 1); // Force C locale
    }
#endif

I've tried various other things I found in the boost and std documentation, but to no avail:

std::setlocale(LC_ALL, "C");
std::locale::global(std::locale("C"));
boost::filesystem::path::imbue(std::locale("C"));

I tried your solution with error trapping (on both bitcoin & namecoin) but unfortunately it doesn't work. Libboost exits on locale errors:

terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid

https://svn.boost.org/trac/boost/ticket/9182 https://svn.boost.org/trac/boost/ticket/9190

Try replacing std::locale() with boost::filesystem::path::codecvt() in the above

See revised patch #4147 (comment)

Revised patch to also include bitcoin-qt:

--- ./src/bitcoin-cli.cpp
+++ ./src/bitcoin-cli.cpp
@@ -53,6 +53,16 @@

 int main(int argc, char* argv[])
 {
+    #ifndef WIN32
+    try
+    {
+        boost::filesystem::path::codecvt(); // Raises runtime error if current locale is invalid
+    } catch(std::runtime_error &e)
+    {
+        setenv("LC_ALL", "C", 1); // Force C locale
+    }
+    #endif
+
     try
     {
         if(!AppInitRPC(argc, argv))
--- ./src/bitcoind.cpp
+++ ./src/bitcoind.cpp
@@ -166,6 +166,16 @@

 int main(int argc, char* argv[])
 {
+    #ifndef WIN32
+    try
+    {
+        boost::filesystem::path::codecvt(); // Raises runtime error if current locale is invalid
+    } catch(std::runtime_error &e)
+    {
+        setenv("LC_ALL", "C", 1); // Force C locale
+    }
+    #endif
+
     bool fRet = false;

     // Connect bitcoind signal handlers
--- ./src/qt/bitcoin.cpp.orig
+++ ./src/qt/bitcoin.cpp
@@ -445,6 +445,16 @@
 #ifndef BITCOIN_QT_TEST
 int main(int argc, char *argv[])
 {
+#ifndef WIN32
+    try
+    {
+        boost::filesystem::path::codecvt(); // Raises runtime error if current locale is invalid
+    } catch(std::runtime_error &e)
+    {
+        setenv("LC_ALL", "C", 1); // Force C locale
+    }
+#endif
+
     /// 1. Parse command-line options. These take precedence over anything else.
     // Command-line options take precedence:
     ParseParameters(argc, argv);

Let's move the workaround to a function instead of repeating it three times.

SetupEnvironment() : #4174#commits-pushed-2816491

There was an idea about adding some module with several runtime self-tests (/cc @laanwj @theuni). This could include testing the availability of EC support for secp256k1 in OpenSSL.

Also, can you submit pull requests instead of patches as text in issues?

@sipa #4081 is the issue for that; but this may be too early in the start process to do the self-tests. Adding those tests in the beginning of Appinit2() and sending an InitError() would allow giving a pop-up message in the GUI.

@itoffshore I've had several fights with busybox configs over locale data. It depends largely on your toolchain and whether things get installed properly or not (locale data from toolchain's gconv or iconv, etc). strace is helpful to see where it's looking and failing. There are also fun libc bugs like Bionic's where setlocale(LC_ALL, NULL) returns NULL rather than the current locale, so if you're using some exotic libc, don't expect busybox to know all about it. It's worth keeping in mind also: if your shell specifies a locale that your c runtime can't provide, bitcoin is likely the least of your concerns.

That said, the approach here seems to line up with what I've found to be the most reliable in the past: set LC_ALL to the C locale (which is IIRC guaranteed), then use c++ locales when it's actually necessary.

Sorry, by busybox configs, I was referring to buildroot configs. I assume that busybox in this case comes from buildroot, but I suppose that's not necessarily true.

This pull request seems to fix this bug for boost filesystem 2 & 3 #4174#commits-pushed-2816491

I'm still getting this error with a latest build... was the fix ever added to the master branch?

terminate called after throwing an instance of 'std::runtime_error'
what():  locale::facet::_S_create_c_locale name not valid

It was applied at https://github.com/bitcoin/bitcoin/commit/5248ff40997c64cc0fde7aaa67cf94dd38b14899 - it fixed locales in Busybox (& should have fixed elsewhere too)

I'm on Ubuntu 12.04.1 LTS.

If setting 'export LC_ALL=C' in a terminal fixes it your old libboost library is not raising an error from std::locale(); in src/util.cpp

Maybe you need http://packages.ubuntu.com/precise/libs/libboost-filesystem1.48.0

that fixes it, but not permanently I guess.. I updated libboost-filesystem1.48.0.. now let's see..

yeah that's it. everytime i login i first have to run 'export LC_ALL=C' and than bitcoind works

you could also put the command in ~/.bashrc or in /etc/profile or /etc/default/locale

without exporting LC_ALL - what does 'echo $LC_ALL' or 'locale' give for LC_ALL ? (maybe something else on your system is setting it to an incompatible value for Bitcoin) - you would need to log off & on again to test this.

thanks.

"LC_ALL" is empty for some reason..

It's normal for LC_ALL to be unset.

ok. it is working with LC_ALL set to "C", so I guess that's ok for me.. i also don't need this node for long..

This is still an open issue in 0.10.1rc2 even though 0.10.0 was fine in the same environment:

EXCEPTION: St13runtime_error       
locale::facet::_S_create_c_locale name not valid       
bitcoin in AppInit()       

terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid
Aborted (core dumped)

Testing 0.10.1 there is no longer any environment regression.

0.10.1 should also be patched as shown here to fix a segfault introduced in 0.10

@laanwj Not sure where to post, but AFAIK some change to locale handling was revert, which reintroduces a crash when chaning -txindex and aborting or when trying to abort Qt startup during init phase in splash screen... too bad this is unfixed again :-/.

Is this issue fixed or still present in 0.12 / master?

There have been more changes to this code since, so I suppose it is, but as the locale initialization is complex voodoo code there could always be an edge case.

This issue has been fixed since 0.10.1 in Alpine Linux & also with 0.11.x too.

I will try building 0.12

Oh! If it was fixed back then I doubt it has come back now. Just wonder why the issue was never closed then. But if you want to try on 0.12 just in case, thanks :)

I no longer run a full node, so I never reported back.

0.12 is fine too - this issue can be closed