Fix Windows locale problem #13107

1. 
   ken2812221 commented at 4:08 am on April 28, 2018: contributor

   Fixes #13103 This PR make internal strings of Windows always be utf8-encoded

   • Add UTF8 <-> localized string conversion function
   • Use localized string on Berkeley DB
   • Use localized string on LevelDB
   • Use localized string on fstream::open
   • Use std::fstream instead of fs::fstream

   Patched the boost library to convert fs::path::string() to be UTF8 encoded. Add two functions that convert between UTF8 and Local encodes.

   To those who want to test this, adding -datadir=(some non-ASCII characters) option make every file that Bitcoin Core use contain non-ASCII characters

   Defect: If the user’s codepage does not support the filename, it will be encoded to ? I don’t known how to solve this. Leveldb and Berkeley DB don’t support wide char.

2. ken2812221 force-pushed on Apr 28, 2018
3. fanquake added the label Wallet on Apr 28, 2018
4. in src/util.cpp:1022 in 0678229867 outdated

   1017@@ -1018,3 +1018,49 @@ int ScheduleBatchPriority(void)
   1018     return 1;
   1019 #endif
   1020 }
   1021+
   1022+std::string Utf8ToLocal(const std::string& utf8_string)
   

   ---

   ---

   

   laanwj commented at 5:21 am on April 28, 2018:

   maybe move to utilstrencodings?
5. 
   laanwj commented at 5:25 am on April 28, 2018: member

   Thanks. Though I’m sad that this is needed. To stay sane, and keep locales and encodings out of everything as much as possible:

   • Please keep all our internal state in UTF-8 everywhere. This was always the intent, so in interaction with JSONRPC we can assume all strings are UTF-8, and matches with Qt’s view.
   • Only convert to local encoding when needed, so just before going to the file system layer, or when coming from the file system layer (or filenames, command-line arguments etc).

   BTW, there are GUIUtil::qstringToBoostPath and GUIUtil::qstringToBoostPath to convert Qt strings from/to local paths but I don’t think they are useful in this case as it doesn’t happen on the interface from Qt to core code.

6. ken2812221 renamed this:
   Fix Windows locale problem
   [WIP] Fix Windows locale problem
   on Apr 28, 2018
7. ken2812221 force-pushed on Apr 28, 2018
8. ken2812221 force-pushed on Apr 28, 2018
9. ken2812221 force-pushed on Apr 28, 2018
10. ken2812221 force-pushed on Apr 28, 2018
11. ken2812221 force-pushed on Apr 28, 2018
12. ken2812221 force-pushed on Apr 28, 2018
13. ken2812221 force-pushed on Apr 28, 2018
14. ken2812221 force-pushed on Apr 28, 2018
15. ken2812221 force-pushed on Apr 28, 2018
16. ken2812221 renamed this:
    [WIP] Fix Windows locale problem
    Fix Windows locale problem
    on Apr 28, 2018
17. ken2812221 force-pushed on Apr 29, 2018
18. ken2812221 force-pushed on Apr 29, 2018
19. ken2812221 force-pushed on Apr 29, 2018
20. ken2812221 force-pushed on Apr 29, 2018
21. ken2812221 force-pushed on Apr 29, 2018
22. ken2812221 force-pushed on Apr 29, 2018
23. MarcoFalke added the label Windows on Apr 30, 2018
24. MarcoFalke removed the label Wallet on Apr 30, 2018
25. MarcoFalke added the label Utils/log/libs on Apr 30, 2018
26. MarcoFalke removed the label Windows on Apr 30, 2018
27. MarcoFalke added the label Wallet on Apr 30, 2018
28. MarcoFalke added the label Windows on Apr 30, 2018
29. MarcoFalke added the label UTXO Db and Indexes on Apr 30, 2018
30. ken2812221 force-pushed on May 1, 2018
31. ken2812221 force-pushed on May 2, 2018
32. ken2812221 force-pushed on May 3, 2018
33. ken2812221 force-pushed on May 3, 2018
34. ken2812221 force-pushed on May 3, 2018
35. ken2812221 force-pushed on May 3, 2018
36. ken2812221 closed this on May 9, 2018

    ---

37. MarcoFalke deleted a comment on May 9, 2018
38. ken2812221 reopened this on May 9, 2018

    ---

39. ken2812221 force-pushed on May 10, 2018
40. ken2812221 force-pushed on May 10, 2018
41. ken2812221 force-pushed on May 10, 2018
42. Use utf8 encoded string on Windows 53552475d1
43. Change to local encoding filename on Berkeley DB 7f9dfdc544
44. Change to local encoding filename on Level DB 13a57b27c6
45. Add ifstream/ofstream to fsbridge b85f1b55dc
46. Move fs/std::i/ofstream to fsbridge::i/ofstream a7e55100f3
47. Use utf8 at StartupShortcutPath 3342746618
48. Parse command line argv utf8 c1f5cd3906
49. ken2812221 force-pushed on May 24, 2018
50. 
    ken2812221 commented at 4:26 pm on June 11, 2018: contributor

    prefer #13426
51. ken2812221 closed this on Jun 11, 2018

    ---

52. ken2812221 deleted the branch on Jun 11, 2018
53. MarcoFalke locked this on Sep 8, 2021

Contributors
ken2812221 laanwj

Labels
Wallet Windows UTXO Db and Indexes Utils/log/libs