Lost characters with complex emoji in command history (Grapheme Clustering)

[peterjc]

What Operating System(s) are you seeing this problem on?

macOS

Which Wayland compositor or X11 Window manager(s) are you using?

No response

WezTerm version

20240203-110809-5046fc22

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

No, and I'll explain why below

Describe the bug

Using most emoji and a sample of double-width Asian characters in commands does not seem to be a problem for the history in WezTerm.

However, composite emoji like "pirate flag" ("🏴‍☠️") or "farmer" ("🧑‍🌾") using a zero-width-joiner (ZWJ) have a display width (2) which is smaller than the sum of their parts (2 + 0 + 2 = 4) and is likely the cause of the glitch.

The pirate flag is black-flag, ZWJ, skull-and-crossbones. See https://en.wikipedia.org/wiki/Zero-width_joiner for background on ZWJ, including non-emoji real language examples.

See also https://mitchellh.com/writing/grapheme-clusters-in-terminals and https://contour-terminal.org/internals/text-stack/#other-terminal-emulator-related-challenges which notes mismatches in wcwidth() and utf8proc_charwidth.

See also https://github.com/contour-terminal/terminal-unicode-core for the "Terminal Unicode Core Specification" which is truing to standardize grapheme cluster behaviour. I see from #4320 that WezTerm should be following that.

To Reproduce

Running with no user configuration on

1. start wezterm from finder.
2. press Apple+EQUALS a dozen times to enlarge the font, and resize window accordingly
3. Run the following:

(original prompt) $ export PS1="$ "
$ clear
$ uname -rsm
Darwin 21.6.0 x86_64
$ echo "[AB]"
[AB]
$ echo "[🏴‍☠️]"
[🏴‍☠️]
$

In the above the final command was typed using e, c, h, o, space, double-quote, open-square-bracket, ctrl+apple+space, search for pirate, select pirate flag, close-square-bracket, double-quote, enter.

[You can also use paste, or the system menu "Show emojis & symbols" but the above is simpler to describe reproducibly.]

So far so good, the pirate flag printed as expected with a width of two characters:

Now press up to pull back the previous command echo "[🏴‍☠️]" from the history, again so far so good, the cursor is after the closing quote around the flag example:

Now press up again to pull back the command echo "[AB]", and this time you should see:

$ echoAB]"

Again the cursor is after the closing quote - but is it as if three characters are missing. The expected output is:

$ echo "[AB]"

I presume there is some cleverness not to redraw the first unchanged 7 characters echo "[ and only overwrite the changed part from 🏴‍☠️]" to AB]". The off by three then fits with wrongly using an alternative length of 5 for the pirate flag?

With a longer prompt, and a longer command, it is often nigh impossible to edit the command as one might expect.

Configuration

no config

Expected Behavior

I would expect even when using unicode characters with ZWJ (grapheme clustering), moving through the command history would work without mispositioning.

Logs

No response

Anything else?

No response

[peterjc]

I should clarify this was using the Apple provided elderly bash, and also happens on ARM.

[peterjc]

Using the Apple provided zsh, this does not happen - while composing the command you don't see the composite character, but rather the three elements which make it up. That feels like a different issue.