BIP-39: say what separates two words #2237

fametrano commented at 9:55 PM on August 5, 2026: contributor

The BIP never says it. The rule exists — the Japanese wordlist page requires the ideographic space U+3000, added in #130 — but it is not in the BIP text, and the note there holds a caveat: ASCII and ideographic spaces are equivalent "as long as your code never shows the user an ASCII space separated phrase or tries to split the phrase input by the user".

The reference implementation splits the input, on U+0020, and so cannot read back the Japanese sentences it writes: to_entropy raises on all 24 Japanese vectors of its own vectors.json. The fix of the reference implementation is open there: https://github.com/trezor/python-mnemonic/pull/145

This adds three sentences to "From mnemonic to seed": the separator, the Japanese exception, and what it means for software that splits a sentence before normalizing it. No test vector changes, and nothing that is correct today becomes incorrect.

Found downstream in btclib: btclib-org/btclib#258.

🤖 Generated with Claude Code

murchandamus added the label Proposed BIP modification on Aug 5, 2026

murchandamus added the label Pending acceptance on Aug 5, 2026

fametrano commented at 10:07 PM on August 5, 2026: contributor

If this lands and there is interest, I am happy to write the follow-up on the other half of the question: what a reader should do with whitespace that is not the separator — a leading or trailing space, the line break of a paper backup wrapped over two lines, a doubled space from a copy-paste.

The BIP mandates NFKD and says nothing more, so implementations differ and the failure mode is quiet: to_seed hashes whatever it is given, so a sentence that a checksum check would reject still derives a seed — a valid-looking wallet that is not the user's. Two answers look defensible: refuse anything but the canonical single-space form, or collapse any run of whitespace after NFKD. No test vector changes under either; btclib, which I maintain, collapses.

I left it out of this PR because it is new normative text rather than a clarification of what the wordlist page already requires, and it should not hold up the separator sentence. Glad to open it separately if the authors want it, or to drop it if the view is that a Deployed BIP should not grow new requirements.

murchandamus commented at 10:08 PM on August 5, 2026: member

cc authors: @prusnak, @ebfull, @slush0, @voisine

prusnak commented at 3:07 PM on August 7, 2026: contributor

On several occasions I raised my will that we should drop non-English wordlists from BIP39, but I was always blocked by editors.

Since I am not able to edit my own standard, I don't really care about issues related to non-English use.

fametrano commented at 3:42 PM on August 7, 2026: contributor

prusnak I do understand your position, and I agree with you about English only.

That said, I do care about fixing inconsistencies, and I would love it if your "I don't really care" could become an ACK.

Also, the possible follow-up — what to do with whitespace that is not the separator, a leading or trailing space and so on — concerns English too, not just Japanese.

fametrano commented at 9:25 AM on September 7, 2026: contributor

@murchandamus a month has passed since the authors were cc'd. One of them, @prusnak, has answered: not an objection, but not an ACK either, and scoped to non-English use; the other three have not replied. Is a non-objection from an author enough to move a clarification like this forward under the "Pending acceptance" label, or does it need an explicit ACK? Happy to wait longer if that is the process, I just want to know what the thread is waiting on.

fametrano force-pushed on Sep 12, 2026

optout21 commented at 7:17 AM on September 14, 2026: none

A few shallow thoughts on this:

I'm not in favor of any change to this BIP related to non-English languages, as the industry consensus seems to be that the spec is sub-optimal and it's not worth supporting.
In a comment you mention trailing white space. I think this is an important point, as any white space issue (leading/trailing white space, duplicate separators, non-space white space) can have significant effect -- a different wallet derived depending on the white space handling!
As generally BIP's are not versioned, I find it dangerous modifying in a way which can make a compliant implementation non-compliant, or, if an implementation is changed to be compliant and that results in a different wallet for the same input. I think such changes should be addressed only a follow-up BIP.
What I can imagine to be changed in this BIP is to document the deficiency, and add a clarification that the current form of the BIP does not specify the handling of the mentioned white space issues. A clarification only, not a change to the mandated rules.

Nonetheless, if the industry standard is white space normalization (before hashing), then this could be reflected in the spec, but this is non-trivial to assess.

fametrano commented at 7:38 AM on September 16, 2026: contributor

We agree on the principle, and I want to say that first. A BIP that is already widely implemented should not be changed in a way that makes a conforming implementation non-conforming, and much less in a way that derives a different wallet from the same input. If this PR did either, I would withdraw it. I think it does neither, and the PR description did not make that clear enough.

No input derives a different wallet. The seed is PBKDF2 over the NFKD of the sentence, and NFKD maps U+3000 to U+0020, so the two ways of writing a Japanese sentence already agree today. Taking the standard all-zero-entropy Japanese vector:

あいこくしん ×11, then あおぞら     (12-word all-zero-entropy Japanese vector, U+3000 separators)

seed, U+3000 separators: 646f1a38134c556e948e6daef213609a62915ef5…
seed, U+0020 separators: 646f1a38134c556e948e6daef213609a62915ef5…

Both forms also pass the checksum. This PR does not touch derivation at all.

What the paragraph is about is splitting a sentence into words, which is what an implementation does to look the words up and to verify the checksum. Split on U+0020 before normalizing and that sentence is 1 word; split after normalizing and it is 12. So the text can change whether an implementation accepts a sentence, never which wallet it derives.

And it is not a new rule. bip-0039-wordlists.md, linked from this BIP, already states it in bold:

Developers implementing phrase generation or checksum verification must separate words using ideographic spaces / accommodate users inputting ideographic spaces.

and then explains that after normalization "dealing with ASCII or Ideographic space is the same". The body of BIP-39 never mentions the separator at all. So the paragraph lifts an existing "must" out of a linked file and into the text implementers actually read.

On your first point, the paragraph does not change anything about non-English wordlists either — it repeats the rule the wordlists file already sets.

So the question I would put back to you: with the above, does this fall within the clarification you say in your fourth point you could accept — the deficiency documented, no mandated rule changed? If not, I would rather fix the wording than argue the point: what would you need it to say?

optout21 commented at 10:07 AM on September 16, 2026: none

You are correct, the change in its current form does not affect wallet derivation. I was carried away by the mention of trailing spaces, but that's not addressed in the current PR. My comments were general, but not valid here, my apologies!

The proposed clarification is being added to the "From mnemonic to seed" section, but I don't think it belongs there. The section describes the NFKD Unicode normalization before PBKDF2. Also, I don't think the spec document itself is the right place for a rule specific to single language, the bip-0039/bip-0039-wordlists.md seems better suited for that.

Looking at the "Japanese" section of bip-0039/bip-0039-wordlists.md, it mentions about separators, the ideographic space, and how it is converted to ASCII space through NFKD. The description there (although not as clearly written as possible) implies the rules for separation, verification, and generation. What extra detail is being added by the proposed change? I see at most the simple rule that words are joined using a space character, the rest is all present already or implied.

I think a better place to mention the space separator would be after the sentence: "Finally, we convert these numbers into words and use the joined words as a mnemonic sentence." "Words are joined using a single space character (for non-English language exceptions see [[bip-0039/bip-0039-wordlists.md]] Wordlists (Special Considerations))."

BIP-39: say what separates two words

The BIP never says it. The rule is on the wordlist page, which requires
the ideographic space for Japanese, and it is not quoted here; the note
there also holds a caveat that ASCII and ideographic spaces are the same
"as long as your code ... [does not try] to split the phrase input by
the user". The reference implementation's to_entropy() does split it, on
U+0020 and with no prior normalization, and so cannot read back the
Japanese sentences to_mnemonic() writes; check() and to_seed() normalize
first and are unaffected.

No test vector changes.

f84880db87

fametrano force-pushed on Sep 16, 2026

fametrano commented at 2:55 PM on September 16, 2026: contributor

Thank you for the detailed follow-up — the placement note is fair, and I've moved the paragraph. The seed section covers NFKD before PBKDF2; this is about how a sentence is written and read, so it now sits right after the sentence you named.

On content, I'd keep one clause beyond your version, and the reason is in the file you linked. It does address splitting, but only as a precondition:

as long as your code never shows the user an ASCII space separated phrase or tries to split the phrase input by the user, dealing with ASCII or Ideographic space is the same

Verifying the checksum requires recovering the individual words, and in practice that means splitting the sentence — so that precondition does not hold for an implementation that checks a mnemonic it was given. What is left unsaid is the order splitting happens in. Split a Japanese sentence on U+0020 before normalizing and the whole sentence reads as one token; normalize first and it splits into the individual words, since NFKD maps U+3000 to U+0020. That order is the detail I think is missing rather than implied.

What I've pushed is your sentence plus that clarification:

Words are joined using a single space character (U+0020), except in Japanese, where the ideographic space (U+3000) is used; see [[bip-0039/bip-0039-wordlists.md|Wordlists (Special Considerations)]]. A sentence is split back into words after NFKD normalization, which maps U+3000 to U+0020; splitting on U+0020 before normalization reads a Japanese sentence as a single word.

Two sentences, in the section you chose. If you'd rather the second lived in the wordlists file instead of the BIP, I'm happy to move it — the order is what matters to me, not where it's written.