I stumbled across the problem of bidirectional text in terminals while trying to test a variant of the Unicode Bidirectional Algorithm. The Unicode consortium publishes a set of bidi test-cases, which suffer from being somewhat “non-visual”. At the end of the day you want to deal with real sentences in real languages and scripts. Preparing that, you face a peculiar problem: how do you display your test output? After all, UAX#9 is about visual ordering of characters. So we just print it to the terminal, do we? But things get confused quickly.

How Hard can It Be?

Sure, terminal applications have to manage some of the messy parts of type, just as browsers do: They need fonts with glyphs for almost any foreseeable script, but restricted to monospace. Text shaping is hard, but probably less so for monospaced fonts. And at least there are no difficulties with sophisticated line breaking, complicated layout with different font sizes, graphics, etc. And bidirectional text should be a solved problem. Or is it?

Turns out for terminal emulators this is an especially tricky topic!

A Conspiracy to Not Let You Bidi

I started my tests from the kitty terminal* I use on MacOS. Let’s go small first, with just a single phrase in Hebrew:

This is the output of the original string, prior to any reordering logic from my test. I’ve not been surprised by the output of MacOS Terminal: obviously it recognizes Hebrew script as right-to-left (RTL) and the errorneous positioning of the ‘!’ is a result of me not specifying overall RTL context. But what about kitty? Perhaps I did get the logical (memory) representation of the phrase wrong, so let’s check by opening it in vim. Surprise: It looks different, depending on the terminal vim runs in. Strange…!

When working on bidirectional text there is a requirement for text-editors to help developers enter the logical (memory) order of characters, instead of the visual order. Unfortunately only Emacs has an option to turn off interpreting bidi (these days I program in Go on VSCode, and I’m not alone in my struggles). But shouldn’t an editor like vim, running in a terminal’s raw mode, operate independent of what the terminal emulator considers clever behaviour for bidi? Obviously no.

The Brave Gnome

Diving deeper, I read up on attempts to standardize bidi handling, including in terminal emulators. After all, the majority of the world’s population use non-Latin scripts, many of them with non-LTR writing direction. But obviously, as of today the behavious of Mac OS Terminal is the best the industry could come up with. And for good reasons: A terminal is not primarily concerned with paragraphs of text, but rather with (from a typesetting perspective) funny combinations of characters, where application of Unicode recommendations simply falls flat.

A smart guy named Egmont Koblinger, contributor to a lot of stuff around Linux terminals (joe  editor, mc  file manager, etc.), came up with a proposal for an updated standard for bidi handling in terminals. It recommends letting applications decide whether they want implicit or explicit bidi handling, the former more or less meaning the Unicode Bidi Algorithm applied by the terminal. When writing tests for bidi yourself, you most certainly prefer explicit mode, i.e., short-curcuiting the terminal’s cleverness.

What’s best: these ideas have already been implemented in VTE, which underlies the Gnome Terminal window. I installed VTE3 on my Mac, included some lines of code in my test to force VTE in explicit mode, and it gave me this:

Now this made me happy 😄! VTE shows me the real thing, the characters in logical (memory) order. While this is of no use to an end-user of terminals, I consider it very valuable for application developers, at least those who have to work on the fine points of international text handling. (VTE even has you covered if your intuition about logical order is from right to left.)

Wrap Up

Terminals, text-editors, IDEs and web browsers all conspire to make it hard to develop and test algorithms for handling of international text. But a brave Gnome offers the best of two worlds: a level-headed implicit mode for day-to-day usage, and an explicit mode for bidi-nerds. Thank you!

I will try to wrap my learnings into a Go package, but at the same time I am aware that this is a tiny niche in a programming universe dominated by web applications, where the browser is expected to do all the heavy lifting. But that’s a different story.

*) Despite the peculiarities described here, switching to kitty is a choice I never came to regret and I recommend kitty without reserve.