Test for Windows encoding issue #13787

ken2812221 commented at 6:05 PM on July 28, 2018: contributor

This PR removes unused Windows compatible path. Encoding issues has been solved.

DrahtBot commented at 7:54 PM on July 28, 2018: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

DrahtBot cross-referenced this on Jul 28, 2018 from issue gui: Don't access static members from class instances by practicalswift

DrahtBot cross-referenced this on Jul 28, 2018 from issue Utils and libraries: Drops the boost/algorithm/string/split.hpp dependency by l2a5b1

DrahtBot cross-referenced this on Jul 28, 2018 from issue -masterdatadir for datadir bootstrapping by kallewoof

DrahtBot cross-referenced this on Jul 28, 2018 from issue PSBT key path cleanups by sipa

DrahtBot cross-referenced this on Jul 28, 2018 from issue bitcoin-cli: -stdinwalletpassphrase and non-echo stdin passwords by kallewoof

DrahtBot cross-referenced this on Jul 28, 2018 from issue Remove the boost/algorithm/string/case_conv.hpp dependency by l2a5b1

DrahtBot cross-referenced this on Jul 28, 2018 from issue wallet: Fix backupwallet for multiwallets by domob1812

DrahtBot cross-referenced this on Jul 28, 2018 from issue [refactor] Fix the chainparamsbase -> util circular dependency by Empact

ken2812221 cross-referenced this on Jul 28, 2018 from issue [bugfix] Fix encoding issue for Windows by ken2812221

DrahtBot cross-referenced this on Jul 28, 2018 from issue Check for datadir after the config files were read by Flowdalic

DrahtBot cross-referenced this on Jul 28, 2018 from issue Utils and libraries: Fix #13371 - move umask operation earlier in AppInit() by n2yen

ken2812221 force-pushed on Jul 30, 2018

ken2812221 force-pushed on Jul 31, 2018

DrahtBot cross-referenced this on Jul 31, 2018 from issue Docs: Create default bitcoin.conf file on startup by leishman

ken2812221 cross-referenced this on Aug 1, 2018 from issue NSIS: Error launching the installer when login name contains some non-ascii characters by MarcoFalke

DrahtBot cross-referenced this on Aug 1, 2018 from issue Include tinyformat as a subtree by Empact

DrahtBot cross-referenced this on Aug 1, 2018 from issue Inline i64tostr and itostr by Empact

DrahtBot cross-referenced this on Aug 2, 2018 from issue Include tinyformat as a subtree by Empact

DrahtBot cross-referenced this on Aug 2, 2018 from issue Move src/tinyformat.h to src/tinyformat/tinyformat.h by Empact

MarcoFalke added this to the milestone 0.18.0 on Aug 2, 2018

DrahtBot cross-referenced this on Aug 2, 2018 from issue util: Add [[nodiscard]] to all {Decode,Parse}[...](...) functions returning bool by practicalswift

DrahtBot cross-referenced this on Aug 3, 2018 from issue wallet: "avoid_reuse" wallet flag for improved privacy by kallewoof

DrahtBot cross-referenced this on Aug 3, 2018 from issue utils: drop boost::interprocess::file_lock by ken2812221

DrahtBot cross-referenced this on Aug 3, 2018 from issue validation: Document where we are intentionally ignoring bool return values from validation related functions by practicalswift

DrahtBot cross-referenced this on Aug 3, 2018 from issue utils: Use _wfopen and _wfreopen on Windows by ken2812221

DrahtBot cross-referenced this on Aug 3, 2018 from issue gui: Drop boost::scoped_array and use wchar_t API explicitly on Windows by ken2812221

DrahtBot cross-referenced this on Aug 4, 2018 from issue utils: Add fstream wrapper to allow to pass unicode filename on Windows by ken2812221

DrahtBot cross-referenced this on Aug 4, 2018 from issue utils: Make fs::path::string() always return utf-8 string on Windows by ken2812221

ken2812221 force-pushed on Aug 5, 2018

ken2812221 renamed this:
~~Test for #13426~~
Test for #13869
on Aug 5, 2018

DrahtBot cross-referenced this on Aug 5, 2018 from issue refactor: Replace boost::bind with std::bind by ken2812221

DrahtBot cross-referenced this on Aug 5, 2018 from issue travis: Enable qt for all jobs by ken2812221

ken2812221 force-pushed on Aug 6, 2018

ken2812221 cross-referenced this on Aug 6, 2018 from issue Filename and command line encoding issue on Windows by ken2812221

ken2812221 force-pushed on Aug 6, 2018

DrahtBot cross-referenced this on Aug 6, 2018 from issue [WIP] depends: Add native_nsis to support unicode by ken2812221

ken2812221 force-pushed on Aug 10, 2018

ken2812221 force-pushed on Aug 11, 2018

DrahtBot cross-referenced this on Aug 14, 2018 from issue ~~Correctly~~ terminate HTTP server by promag

DrahtBot cross-referenced this on Aug 15, 2018 from issue refactor: use fs:: over boost::filesystem:: by fanquake

DrahtBot added the label Needs rebase on Aug 15, 2018

jnewbery commented at 2:15 PM on August 23, 2018: member

suggest you update the title to describe what's being tested (rather than referencing a different issue)

ken2812221 renamed this:
~~Test for #13869~~
Test for Windows encoding issue
on Aug 24, 2018

ken2812221 commented at 3:22 AM on August 24, 2018: contributor

@jnewbery Thanks for your suggestion

ken2812221 force-pushed on Aug 29, 2018

DrahtBot removed the label Needs rebase on Aug 29, 2018

sipsorcery commented at 11:39 AM on August 31, 2018: member

Without this PR bitcoind does not get very far with a unicode directory (the conf file uses datadir=C:\Temp\资金钱包\Bitcoin) :

bitcoind -conf=c:\temp\bitcoin.conf Error reading configuration file: specified data directory "C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin" does not exist.

After applying this PR bitcoind gets to the point where it attempts to open the wallet file:

bitcoind -conf=c:\temp\bitcoin.conf 2018-08-31T11:22:29Z Bitcoin Core version v1.17.99.0-unk (release build) 2018-08-31T11:22:30Z InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1 2018-08-31T11:22:30Z Validating signatures for all blocks. 2018-08-31T11:22:30Z Setting nMinimumChainWork=0000000000000000000000000000000000000000000000000000000000000000 2018-08-31T11:22:30Z Using the 'standard' SHA256 implementation 2018-08-31T11:22:31Z Default data directory C:\Users\aaron\AppData\Roaming\Bitcoin 2018-08-31T11:22:31Z Using data directory C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest 2018-08-31T11:22:31Z Using config file c:\temp\bitcoin.conf .... 2018-08-31T11:22:32Z Using wallet wallet.dat 2018-08-31T11:22:32Z BerkeleyEnvironment::Open: LogDir=C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\database ErrorFile=C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\db.log 2018-08-31T11:22:32Z BerkeleyEnvironment::Open: Error -30974 opening database environment: DB_RUNRECOVERY: Fatal error, run database recovery 2018-08-31T11:22:32Z Moved old C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\database to C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\database.1535714552.bak. Retrying. 2018-08-31T11:22:32Z BerkeleyEnvironment::Open: LogDir=C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\database ErrorFile=C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\db.log 2018-08-31T11:22:32Z BerkeleyEnvironment::Open: Error -30974 opening database environment: DB_RUNRECOVERY: Fatal error, run database recovery 2018-08-31T11:22:32Z Error: Error initializing wallet database environment "C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets"! Error: Error initializing wallet database environment "C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets"!

I'll work out how to apply the leveldb patch and retry.

ken2812221 force-pushed on Aug 31, 2018

ken2812221 commented at 2:43 PM on August 31, 2018: contributor

@sipsorcery Thanks for testing. Could you test this again? This shall not contain garbled text.

However, I didn't get that bdb error locally. Did you forget to make -C depends HOST=x86_64-w64-mingw32? This should re-compile the bdb to support unicode.

DrahtBot cross-referenced this on Aug 31, 2018 from issue gui: Add GUIUtil::bringToFront by promag

sipsorcery commented at 8:09 AM on September 1, 2018: member

@ken2812221 I'm testing with a Visual Studio build (currently having difficulty running the mingw32 build to run due to something being up with libzmq; still tracking that one down).

I've added leveldb to my VS solution and patched it. Bitcoind now seems to work correctly with unicode characters in the data directory path IF the wallet is disabled. If the wallet is enabled then the BerkleyDB library is failing to initialise.

2018-09-01T07:37:21Z BerkeleyEnvironment::Open: LogDir=C:\Temp\资金钱包\Bitcoin\regtest\wallets\database ErrorFile=C:\Temp\资金钱包\Bitcoin\regtest\wallets\db.log 2018-09-01T07:37:21Z BerkeleyEnvironment::Open: Error -30974 opening database environment: DB_RUNRECOVERY: Fatal error, run database recovery 2018-09-01T07:37:21Z Moved old C:\Temp\资金钱包\Bitcoin\regtest\wallets\database to C:\Temp\资金钱包\Bitcoin\regtest\wallets\database.1535787441.bak. Retrying. 2018-09-01T07:37:21Z BerkeleyEnvironment::Open: LogDir=C:\Temp\资金钱包\Bitcoin\regtest\wallets\database ErrorFile=C:\Temp\资金钱包\Bitcoin\regtest\wallets\db.log 2018-09-01T07:37:21Z BerkeleyEnvironment::Open: Error -30974 opening database environment: DB_RUNRECOVERY: Fatal error, run database recovery 2018-09-01T07:37:23Z Error: Error initializing wallet database environment "C:\Temp\资金钱包\Bitcoin\regtest\wallets"! Error: Error initializing wallet database environment "C:\Temp\2018-09-01T07:37:44Z Interrupting HTTP RPC server 2018-09-01T07:37:44Z Interrupting RPC

Kudos for working on this! Unicode and Windows is a minefield. Have you seen http://utf8everywhere.org/ (Chapter 10 is dedicated to UTF on Windows).

ken2812221 commented at 12:51 PM on September 1, 2018: contributor

You can see #13884 , I pre-defined UNICODE and _UNICODE. You should define them while you are compiling berkeleydb. But I don't know how to do that with vcpkg.

sipsorcery commented at 7:12 PM on September 2, 2018: member

I've built this PR on Ubuntu and Windows Subsystem for Linux (Ubuntu 18) and bitcoind executes correctly with a data directory that contains unicode characters.

I am not familiar with nsis and have not tested that part of the PR. How do I use nsis to the build the installer package?

One additional task that would need to be done after this PR is merged is to update the Windows build instructions to add the new packages:

sudo apt install scons wine-binfmt wine64 g++-mingw-w64-i686

ken2812221 cross-referenced this on Sep 5, 2018 from issue tests: Run functional test on Windows and enable it on Appveyor by ken2812221

ken2812221 force-pushed on Sep 7, 2018

DrahtBot cross-referenced this on Sep 10, 2018 from issue utils: Convert fs::filesystem_error messages from local multibyte to utf-8 on Windows by ken2812221

DrahtBot cross-referenced this on Sep 12, 2018 from issue logging: Replace LogPrint macros with regular functions by MarcoFalke

ken2812221 force-pushed on Sep 12, 2018

laanwj added the label Tests on Sep 13, 2018

laanwj added the label Windows on Sep 13, 2018

DrahtBot added the label Needs rebase on Sep 13, 2018

ken2812221 force-pushed on Sep 13, 2018

DrahtBot removed the label Needs rebase on Sep 13, 2018

DrahtBot cross-referenced this on Sep 15, 2018 from issue utils: Convert Windows args to utf-8 string by ken2812221

ken2812221 force-pushed on Sep 16, 2018

DrahtBot cross-referenced this on Sep 16, 2018 from issue windows: Fix remaining compiler warnings (MSVC) by practicalswift

DrahtBot cross-referenced this on Sep 24, 2018 from issue rpc: Early call once CWallet::MarkDirty in import calls by promag

ken2812221 force-pushed on Sep 27, 2018

DrahtBot cross-referenced this on Sep 27, 2018 from issue Multiprocess bitcoin by ryanofsky

ken2812221 force-pushed on Sep 28, 2018

ken2812221 force-pushed on Sep 30, 2018

DrahtBot cross-referenced this on Sep 30, 2018 from issue appveyor: script improvement by ken2812221

in test/functional/test_runner.py:255 in 0131f56dd8 outdated

 243 | @@ -244,10 +244,6 @@ def main():
 244 |      # Create base test directory
 245 |      tmpdir = "%s/test_runner_₿_🏃_%s" % (args.tmpdirprefix, datetime.datetime.now().strftime("%Y%m%d_%H%M%S"))
 246 |  
 247 | -    # If we fixed the command-line and filename encoding issue on Windows, these two lines could be removed
 248 | -    if config["environment"]["EXEEXT"] == ".exe":
 249 | -        tmpdir = "%s/test_runner_%s" % (args.tmpdirprefix, datetime.datetime.now().strftime("%Y%m%d_%H%M%S"))

MarcoFalke commented at 12:57 AM on October 1, 2018:

Nice

DrahtBot cross-referenced this on Oct 2, 2018 from issue msvc: build secp256k1 and leveldb locally by ken2812221

DrahtBot added the label Needs rebase on Oct 8, 2018

ken2812221 force-pushed on Oct 8, 2018

DrahtBot removed the label Needs rebase on Oct 8, 2018

DrahtBot added the label Needs rebase on Nov 5, 2018

ken2812221 force-pushed on Nov 5, 2018

DrahtBot removed the label Needs rebase on Nov 5, 2018

DrahtBot added the label Needs rebase on Nov 7, 2018

ken2812221 force-pushed on Nov 8, 2018

DrahtBot removed the label Needs rebase on Nov 8, 2018

ken2812221 force-pushed on Nov 8, 2018

ken2812221 force-pushed on Jan 22, 2019

ken2812221 cross-referenced this on Jan 23, 2019 from issue Use utf-8 to decode filename by ken2812221

laanwj referenced this in commit f545dfabff on Jan 26, 2019

DrahtBot added the label Needs rebase on Jan 31, 2019

MarcoFalke removed the label Needs rebase on Jan 31, 2019

DrahtBot added the label Needs rebase on Jan 31, 2019

laanwj commented at 3:49 PM on February 6, 2019: member

Looks like most of the prerequisites have been merged now, can we make this in before the 0.18 split?

ken2812221 commented at 9:12 AM on February 7, 2019: contributor

Appveyor is going to update the images in a few days. After then, we don't have to get the latest vapkg port data to get Berkeley db package and re-build vcpkg every time (3+ minutes). I will rebase it once they update the CI image.

tests: accept unicode characters on Windows ae892adceb

ken2812221 force-pushed on Feb 11, 2019

DrahtBot removed the label Needs rebase on Feb 11, 2019

appveyor: Remove outdated libraries 15b31032a6

ken2812221 force-pushed on Feb 11, 2019

laanwj commented at 12:56 PM on February 12, 2019: member

Ok, just be aware that the split is going to be 2019-03-01 and it's going to be busy around then.

ken2812221 commented at 1:04 PM on February 12, 2019: contributor

This is ready to review/merge. I have already rebased it.

MarcoFalke merged this on Feb 12, 2019

MarcoFalke closed this on Feb 12, 2019

MarcoFalke referenced this in commit d8794a78a8 on Feb 12, 2019

laanwj cross-referenced this on Feb 13, 2019 from issue Invalid wallet path with Chinese characters in windows by promag

ken2812221 deleted the branch on Feb 13, 2019

kiminuo cross-referenced this on Jun 11, 2020 from issue Replace boost::filesystem with std::filesystem by kiminuo

jasonbcox referenced this in commit 6b6adaedc7 on Nov 10, 2020

linuxsh2 referenced this in commit 4b94a052d0 on Sep 16, 2021

Munkybooty referenced this in commit 1ac603aa07 on Oct 5, 2021

Munkybooty referenced this in commit ee875183ff on Oct 5, 2021

Munkybooty referenced this in commit 5cc78ce06b on Oct 7, 2021

ryanofsky cross-referenced this on Oct 12, 2021 from issue refactor: Forbid calling unsafe fs::path(std::string) constructor and fs::path::string() method by ryanofsky

pravblockc referenced this in commit d8b1e933c3 on Nov 18, 2021

bitcoin locked this on Dec 16, 2021