Add full UTF-8 support to RPC #7892

(the univalue patch should be taken upstream - the RPC test and doc change should end up here. Posting it here for visibility and because this fixes an ancient issue: #2127. I think this is important to many non-English users)

This adds full UTF-8 support for both on input and output through JSON.

bitcoin-cli usage

Before:

$ src/bitcoin-cli -datadir=/store/tmp/testbtc getnewaddress "рыба"
1HQCE3H87fZ1er1ExLiF5V4a1Kxf46r3J2
$ src/bitcoin-cli -datadir=/store/tmp/testbtc listaccounts
{
...  "\u00c3\u0083\u00c2\u0091\u00c3\u0082\u00c2\u0080\u00c3\u0083\u00c2\u0091\u00c3\u0082\u00c2\u008b\u00c3\u0083\u00c2\u0090\u00c3\u0082\u00c2\u00b1\u00c3\u0083\u00c2\u0090\u00c3\u0082\u00c2\u00b0": 0.00000000
}
$ src/bitcoin-cli -datadir=/store/tmp/testbtc getaccount 1HQCE3H87fZ1er1ExLiF5V4a1Kxf46r3J2
рыба

After:

$ src/bitcoin-cli -regtest getaccountaddress "рыба"
mrVZ8GaURJmL3LEUu6ygU2CrUrZ8979iK2
$ src/bitcoin-cli -regtest listaccounts
{
...
  "рыба": 0.00000000
}
$ src/bitcoin-cli -regtest getaccount mrVZ8GaURJmL3LEUu6ygU2CrUrZ8979iK2                                       
рыба

GUI

This also affects the GUI debug console.

Before:

When strings are passed directly (such as for getaccount's return argument) it works fine, but when they go through JSON formatting/parsing, it fails.

After:

RPC test

A test for this new functionality has been added to the wallet.py test.

See the commit message of the first commit for details on what exactly had to be changed in univalue.

Nice! Concept ACK.

Not sure how we should treat the UniValue changes. Should we try to keep the "upstream" repository bitcoin/univalue in sync?

Concept ACK. Tests look good.

"upstream" repository bitcoin/univalue in sync

The patch goes to jgarzik/univalue as bitcoin/univalue is only for bitcoin related patches. (Still in the end we need to merge jgarzik/univalue into bitcoin/univalue because we use this as subtree.)

Neat. What happens with characters like unicode direction modifiers? Can you end up with your terminal in a weird state?

Neat. What happens with characters like unicode direction modifiers? Can you end up with your terminal in a weird state?

I don't think that's possible with just unicode?

ANSI escape sequences are a whole different story. In JSON objects they'll be escaped, but when printing strings (say, for help, when there is a single string result) everything is let through. Not sure whether this should be changed, but this would be a change local to bitcoin-cli, not the unicode framework. I'd suggest doing so in a separate pull.

Nothing new here though: It's always been possible to pass any kind of crap characters from the server (just look at my OP) to bitcoin-cli, this just makes the round-trip sane and conform to UTF-8.

concept ACK, once-over utACK (did not cross-verify UTF-8 impl.).

Could probably use more tests for that JSONUTF8Writer...

Could probably use more tests for that JSONUTF8Writer... ing

There are few direct tests for it, but all the other tests that parses JSON with strings in it is testing it. What would be nice is doing some fuzzing using https://github.com/laanwj/univalue/tree/2015_11_unifuzz and see that everything is accepted is valid utf-8.

@pstratem Any interest in fuzzing this?

Needs rebase.

Rebased, done and removed the Python3 TODOs

Ok:

• Merged the unicode changes (https://github.com/jgarzik/univalue/pull/22) into https://github.com/bitcoin-core/univalue
• Replaced the direct changes to src/univalue to a subtree merge

This should be ready now.

ACK

It looks like the build system doesn't detect changes to univalue source files and will not automatically rebuild the library: if you get RPC test failures concerning unicode, please build from a clean tree

make clean
make distclean

should take care of this, I assume.

Yes, make clean should be enough.