Test for Windows encoding issue

ken2812221 commented at 6:05 pm on July 28, 2018: contributor

This PR removes unused Windows compatible path. Encoding issues has been solved.

DrahtBot commented at 7:54 pm on July 28, 2018: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

ken2812221 force-pushed on Jul 30, 2018

ken2812221 force-pushed on Jul 31, 2018

MarcoFalke added this to the milestone 0.18.0 on Aug 2, 2018

ken2812221 force-pushed on Aug 5, 2018

ken2812221 renamed this:
~~Test for #13426~~
Test for #13869
on Aug 5, 2018

ken2812221 force-pushed on Aug 6, 2018

ken2812221 force-pushed on Aug 10, 2018

ken2812221 force-pushed on Aug 11, 2018

DrahtBot added the label Needs rebase on Aug 15, 2018

jnewbery commented at 2:15 pm on August 23, 2018: member

suggest you update the title to describe what’s being tested (rather than referencing a different issue)

ken2812221 renamed this:
~~Test for #13869~~
Test for Windows encoding issue
on Aug 24, 2018

ken2812221 commented at 3:22 am on August 24, 2018: contributor

@jnewbery Thanks for your suggestion

ken2812221 force-pushed on Aug 29, 2018

DrahtBot removed the label Needs rebase on Aug 29, 2018

sipsorcery commented at 11:39 am on August 31, 2018: member

Without this PR bitcoind does not get very far with a unicode directory (the conf file uses datadir=C:\Temp\资金钱包\Bitcoin) :

bitcoind -conf=c:\temp\bitcoin.conf Error reading configuration file: specified data directory “C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin” does not exist.

After applying this PR bitcoind gets to the point where it attempts to open the wallet file:

bitcoind -conf=c:\temp\bitcoin.conf 2018-08-31T11:22:29Z Bitcoin Core version v1.17.99.0-unk (release build) 2018-08-31T11:22:30Z InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1 2018-08-31T11:22:30Z Validating signatures for all blocks. 2018-08-31T11:22:30Z Setting nMinimumChainWork=0000000000000000000000000000000000000000000000000000000000000000 2018-08-31T11:22:30Z Using the ‘standard’ SHA256 implementation 2018-08-31T11:22:31Z Default data directory C:\Users\aaron\AppData\Roaming\Bitcoin 2018-08-31T11:22:31Z Using data directory C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest 2018-08-31T11:22:31Z Using config file c:\temp\bitcoin.conf …. 2018-08-31T11:22:32Z Using wallet wallet.dat 2018-08-31T11:22:32Z BerkeleyEnvironment::Open: LogDir=C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\database ErrorFile=C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\db.log 2018-08-31T11:22:32Z BerkeleyEnvironment::Open: Error -30974 opening database environment: DB_RUNRECOVERY: Fatal error, run database recovery 2018-08-31T11:22:32Z Moved old C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\database to C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\database.1535714552.bak. Retrying. 2018-08-31T11:22:32Z BerkeleyEnvironment::Open: LogDir=C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\database ErrorFile=C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets\db.log 2018-08-31T11:22:32Z BerkeleyEnvironment::Open: Error -30974 opening database environment: DB_RUNRECOVERY: Fatal error, run database recovery 2018-08-31T11:22:32Z Error: Error initializing wallet database environment “C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets”! Error: Error initializing wallet database environment “C:\Temp\ÞÁäÚçæÚÆ▒Õîà\Bitcoin\regtest\wallets”!

I’ll work out how to apply the leveldb patch and retry.

ken2812221 force-pushed on Aug 31, 2018

ken2812221 commented at 2:43 pm on August 31, 2018: contributor

@sipsorcery Thanks for testing. Could you test this again? This shall not contain garbled text.

However, I didn’t get that bdb error locally. Did you forget to make -C depends HOST=x86_64-w64-mingw32? This should re-compile the bdb to support unicode.

sipsorcery commented at 8:09 am on September 1, 2018: member

@ken2812221 I’m testing with a Visual Studio build (currently having difficulty running the mingw32 build to run due to something being up with libzmq; still tracking that one down).

I’ve added leveldb to my VS solution and patched it. Bitcoind now seems to work correctly with unicode characters in the data directory path IF the wallet is disabled. If the wallet is enabled then the BerkleyDB library is failing to initialise.

2018-09-01T07:37:21Z BerkeleyEnvironment::Open: LogDir=C:\Temp\资金钱包\Bitcoin\regtest\wallets\database ErrorFile=C:\Temp\资金钱包\Bitcoin\regtest\wallets\db.log 2018-09-01T07:37:21Z BerkeleyEnvironment::Open: Error -30974 opening database environment: DB_RUNRECOVERY: Fatal error, run database recovery 2018-09-01T07:37:21Z Moved old C:\Temp\资金钱包\Bitcoin\regtest\wallets\database to C:\Temp\资金钱包\Bitcoin\regtest\wallets\database.1535787441.bak. Retrying. 2018-09-01T07:37:21Z BerkeleyEnvironment::Open: LogDir=C:\Temp\资金钱包\Bitcoin\regtest\wallets\database ErrorFile=C:\Temp\资金钱包\Bitcoin\regtest\wallets\db.log 2018-09-01T07:37:21Z BerkeleyEnvironment::Open: Error -30974 opening database environment: DB_RUNRECOVERY: Fatal error, run database recovery 2018-09-01T07:37:23Z Error: Error initializing wallet database environment “C:\Temp\资金钱包\Bitcoin\regtest\wallets”! Error: Error initializing wallet database environment “C:\Temp\2018-09-01T07:37:44Z Interrupting HTTP RPC server 2018-09-01T07:37:44Z Interrupting RPC

Kudos for working on this! Unicode and Windows is a minefield. Have you seen http://utf8everywhere.org/ (Chapter 10 is dedicated to UTF on Windows).

ken2812221 commented at 12:51 pm on September 1, 2018: contributor

You can see #13884 , I pre-defined UNICODE and _UNICODE. You should define them while you are compiling berkeleydb. But I don’t know how to do that with vcpkg.

sipsorcery commented at 7:12 pm on September 2, 2018: member

I’ve built this PR on Ubuntu and Windows Subsystem for Linux (Ubuntu 18) and bitcoind executes correctly with a data directory that contains unicode characters.

I am not familiar with nsis and have not tested that part of the PR. How do I use nsis to the build the installer package?

One additional task that would need to be done after this PR is merged is to update the Windows build instructions to add the new packages:

sudo apt install scons wine-binfmt wine64 g++-mingw-w64-i686

ken2812221 force-pushed on Sep 7, 2018

ken2812221 force-pushed on Sep 12, 2018

laanwj added the label Tests on Sep 13, 2018

laanwj added the label Windows on Sep 13, 2018

DrahtBot added the label Needs rebase on Sep 13, 2018

ken2812221 force-pushed on Sep 13, 2018

DrahtBot removed the label Needs rebase on Sep 13, 2018

ken2812221 force-pushed on Sep 16, 2018

ken2812221 force-pushed on Sep 27, 2018

ken2812221 force-pushed on Sep 28, 2018

ken2812221 force-pushed on Sep 30, 2018

in test/functional/test_runner.py:255 in 0131f56dd8 outdated

243@@ -244,10 +244,6 @@ def main():
244     # Create base test directory
245     tmpdir = "%s/test_runner_₿_🏃_%s" % (args.tmpdirprefix, datetime.datetime.now().strftime("%Y%m%d_%H%M%S"))
246 
247-    # If we fixed the command-line and filename encoding issue on Windows, these two lines could be removed
248-    if config["environment"]["EXEEXT"] == ".exe":
249-        tmpdir = "%s/test_runner_%s" % (args.tmpdirprefix, datetime.datetime.now().strftime("%Y%m%d_%H%M%S"))

MarcoFalke commented at 0:57 am on October 1, 2018:

Nice

DrahtBot added the label Needs rebase on Oct 8, 2018

ken2812221 force-pushed on Oct 8, 2018

DrahtBot removed the label Needs rebase on Oct 8, 2018

DrahtBot added the label Needs rebase on Nov 5, 2018

ken2812221 force-pushed on Nov 5, 2018

DrahtBot removed the label Needs rebase on Nov 5, 2018

DrahtBot added the label Needs rebase on Nov 7, 2018

ken2812221 force-pushed on Nov 8, 2018

DrahtBot removed the label Needs rebase on Nov 8, 2018

ken2812221 force-pushed on Nov 8, 2018

ken2812221 force-pushed on Jan 22, 2019

laanwj referenced this in commit f545dfabff on Jan 26, 2019

DrahtBot added the label Needs rebase on Jan 31, 2019

MarcoFalke removed the label Needs rebase on Jan 31, 2019

DrahtBot added the label Needs rebase on Jan 31, 2019

laanwj commented at 3:49 pm on February 6, 2019: member

Looks like most of the prerequisites have been merged now, can we make this in before the 0.18 split?

ken2812221 commented at 9:12 am on February 7, 2019: contributor

Appveyor is going to update the images in a few days. After then, we don’t have to get the latest vapkg port data to get Berkeley db package and re-build vcpkg every time (3+ minutes). I will rebase it once they update the CI image.

tests: accept unicode characters on Windows ae892adceb

ken2812221 force-pushed on Feb 11, 2019

DrahtBot removed the label Needs rebase on Feb 11, 2019

appveyor: Remove outdated libraries 15b31032a6

ken2812221 force-pushed on Feb 11, 2019

laanwj commented at 12:56 pm on February 12, 2019: member

Ok, just be aware that the split is going to be 2019-03-01 and it’s going to be busy around then.

ken2812221 commented at 1:04 pm on February 12, 2019: contributor

This is ready to review/merge. I have already rebased it.

MarcoFalke merged this on Feb 12, 2019

MarcoFalke closed this on Feb 12, 2019

MarcoFalke referenced this in commit d8794a78a8 on Feb 12, 2019

ken2812221 deleted the branch on Feb 13, 2019

jasonbcox referenced this in commit 6b6adaedc7 on Nov 10, 2020

linuxsh2 referenced this in commit 4b94a052d0 on Sep 16, 2021

Munkybooty referenced this in commit 1ac603aa07 on Oct 5, 2021

Munkybooty referenced this in commit ee875183ff on Oct 5, 2021

Munkybooty referenced this in commit 5cc78ce06b on Oct 7, 2021

pravblockc referenced this in commit d8b1e933c3 on Nov 18, 2021

MarcoFalke locked this on Dec 16, 2021

Test for Windows encoding issue #13787

Conflicts