Unicode for working developers

var preview = displayName.Length > 24
    ? displayName.Substring(0, 24) + "…"
    : displayName;

That’s more or less the line that caused a support ticket last month. A customer’s push notifications showed her name with a black diamond and a question mark at the end. Her display name ended in an emoji, the 24th char was the first half of it, and Substring happily cut it in two. When the notification service encoded the string to UTF-8, the orphaned half got replaced with U+FFFD, the replacement character, which is the black diamond.

I’ve known for twenty years that you’re not supposed to do that, and it’s in our codebase anyway, written by someone who also knew. The same week, Nikita Prokopov published The Absolute Minimum Every Software Developer Must Know About Unicode in 2023, a deliberate update of Joel Spolsky’s 2003 essay, and it was all over Hacker News. It’s very good, and you should read it. This is my version, with the parts I’ve actually been bitten by, and a bit more on normalization, which he covers more briefly.

Code points

Unicode is, first, a numbering. Every character gets a number called a code point, written U+ and then hex: A is U+0041, é is U+00E9, € is U+20AC, न (the Devanagari letter NA) is U+0928, and 😀 is U+1F600.

The space goes from U+0000 to U+10FFFF, which is 1,114,112 possible code points, split into 17 “planes” of 65,536 each. Almost everything you’ll meet in ordinary text, including nearly all living scripts, is in plane 0, the Basic Multilingual Plane, U+0000 to U+FFFF. Emoji, a lot of historic scripts and the rarer CJK characters live above it. Unicode 15.1, released last month, has about 150,000 assigned characters, so most of the space is still empty.

A code point isn’t a byte, and it isn’t a “character” in the sense a user means, either. Hold on to both of those, because every bug in this post comes from forgetting one or the other.

UTF-8

To store code points you need an encoding. UTF-8 is the one that won: it’s on about 98% of web pages now, according to the surveys Prokopov cites, and if you’re choosing an encoding for anything new in 2023 there’s nothing else to consider.

It’s variable-length, from one to four bytes per code point, and the top bits of each byte tell you where you are:

Code points	Bytes	Bit pattern
U+0000 to U+007F	1	0xxxxxxx
U+0080 to U+07FF	2	110xxxxx 10xxxxxx
U+0800 to U+FFFF	3	1110xxxx 10xxxxxx 10xxxxxx
U+10000 to U+10FFFF	4	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Take €, U+20AC. In binary that’s 0010 0000 1010 1100, sixteen bits, so it needs the three-byte form, which has room for 4 + 6 + 6 = 16 payload bits. Split them up as 0010 000010 101100, drop them into the slots, and you get 11100010 10000010 10101100, which is E2 82 AC. If you’ve ever seen € on a web page where a euro sign should be, that’s those three bytes decoded as Windows-1252 one at a time.

The design has some properties I didn’t appreciate until I’d written a parser. ASCII is unchanged, so any ASCII file is already valid UTF-8. A continuation byte always starts with 10, so if you land in the middle of a string you can always find the start of the next code point by skipping at most three bytes. And no byte of a multi-byte sequence is ever below 0x80, so a UTF-8 string never contains a stray / or a NUL byte that isn’t really one, which is why it slid into Unix and C APIs with so little fuss.

UTF-16, or why .NET has this problem

My native tongue as a programmer is .NET, and .NET strings, like Java’s and JavaScript’s, are sequences of 16-bit code units. That made sense in the early 1990s, when Unicode was going to fit in 16 bits and UCS-2 was going to be one unit per character forever. Then Unicode outgrew 65,536, and UCS-2 became UTF-16, which represents anything above U+FFFF as a surrogate pair: two 16-bit units from a reserved range (U+D800 to U+DFFF) that together encode one code point. 😀 is D83D DE00.

So in C#, "😀".Length is 2. Substring, indexing and Length all count UTF-16 code units, and nothing stops you cutting a surrogate pair in half, which is exactly what our notification code did. A lone surrogate isn’t a valid code point, so when you encode it to UTF-8, .NET’s default is to substitute U+FFFD, the black diamond in the support ticket. Other runtimes throw an exception, which I’d honestly prefer.

When I was answering .NET questions on newsgroups in my MVP years, the standard advice was “use char for characters”. It was wrong then for anything outside the BMP, and it’s more wrong now that emoji are in everyone’s names.

Grapheme clusters

Even counting code points isn’t enough, because what a person calls one character can be several code points. Unicode’s name for the user-perceived character is an extended grapheme cluster, and the rules for where one ends are in UAX #29.

Prokopov’s example is the best one I’ve seen, so I’m borrowing it: 🤦🏼‍♂️, the man facepalming with a medium-light skin tone.

It’s five code points: the facepalm, a skin-tone modifier, a zero-width joiner (ZWJ), the male sign and a variation selector that asks for the emoji rather than the text presentation. Two of the five, the facepalm and the skin tone, are outside the BMP and take a surrogate pair each in UTF-16, and the other three are one unit each, so it’s 7 UTF-16 units. In UTF-8 it’s 4 + 4 + 3 + 3 + 3 = 17 bytes. So depending on which language you ask, the “length” of this one character is 7 (JavaScript), 5 (Python, which counts code points) or 17 (Rust’s len(), which counts bytes). None of them is 1.

Emoji make it obvious, but they’re not the main reason grapheme clusters exist. Most of the world’s scripts need them. Here’s “namaste” in Hindi, नमस्ते, which is six code points:

• न NA
• म MA
• स SA
• ् VIRAMA (the sign that kills the inherent vowel of SA, so it joins the next consonant)
• त TA
• े VOWEL SIGN E

A Hindi reader sees three units: न, म and the conjunct स्ते. Under the grapheme rules up to Unicode 15.0, a segmenter gives you four clusters (न, म, स्, ते), because the virama attached to SA but nothing joined it to TA. Unicode 15.1 added a new rule specifically for conjuncts in several Indic scripts, including Devanagari, so a 15.1 segmenter gives three. The Intl.Segmenter in the Node 20 on my laptop, which is on an ICU release that predates 15.1, still says four.

That’s the uncomfortable part. The answer to “how many characters is this?” depends on the version of Unicode your library implements, and it changes. Prokopov makes the same point, and it’s the strongest argument I know for never writing your own segmentation code.

Growing up in India, I used computers in English, on DOS, with code page 437 and its box-drawing characters doing all the work in my Turbo C screens. Indian scripts had their own 8-bit standard, ISCII, and in practice most software just didn’t support them. It’s genuinely nice that a Hindi name now works on every phone, but it means the bugs have moved from “the text can’t be shown” to “the text is shown and we count it wrong”.

What to use instead of Substring: in .NET, System.Globalization.StringInfo (and since .NET 5 it follows UAX #29 properly; before that it didn’t handle emoji sequences), or the newer Rune type if you only need code points. In JavaScript, Intl.Segmenter. In Swift, String is grapheme-based already, which is why Swift strings are annoying to index and why they’re right. Our fix was four lines: truncate by text elements with StringInfo, and only then add the ellipsis.

Normalization

There’s one more trap. é can be written as one code point, U+00E9 (LATIN SMALL LETTER E WITH ACUTE), or as two, U+0065 e followed by U+0301 COMBINING ACUTE ACCENT. They look identical, they’re the same grapheme, and they’re different strings. "José" == "José" can be false in any language if one came from a filename on an older Mac (HFS+ stored names decomposed) and the other from a web form.

Unicode defines normal forms to deal with this. NFC composes everything that can be composed (so you get U+00E9), NFD decomposes everything (so you get e plus the accent). The two “K” forms, NFKC and NFKD, also fold compatibility characters: the ligature fi (U+FB01) becomes the two letters fi, a superscript ² becomes 2, full-width letters become ordinary ones. That folding loses information, so you wouldn’t apply it to text you’re storing for display, but it’s what you want for things like usernames and search keys.

My rules, which aren’t original:

1. Normalize to NFC at the boundary, when text comes into your system, and store it that way. Most input is NFC already, and filenames are the classic exception.
2. For identifiers that people type and compare, usernames, tags, email local parts if you dare, use NFKC plus case folding, and do it in one place.
3. Lowercase with an invariant culture when the result is for a machine. The classic .NET bug is ToLower() on a server whose culture is Turkish, where I lowercases to a dotless ı and your "ID" comparison stops matching. ToLowerInvariant(), or better, an ordinal case-insensitive comparison.

There’s a security side to this that I’ll only gesture at. Different code points that look alike (a Cyrillic а and a Latin a, say) make it possible to register a username or a domain that looks like someone else’s, and normalization alone doesn’t fix it, since those are different letters, not different encodings of the same one. Unicode publishes a confusables list and a “skeleton” algorithm in UTS #39 for exactly this, and browsers use similar rules to decide when to show a domain in punycode instead. If you run a system where people pick names that other people see, it’s worth an afternoon.

Back to the ticket

I grepped our main repo for Substring(0, after the ticket. There were 31 hits. Most are on things like hex strings and ID prefixes, where it’s fine. Six were on text a person wrote, and we’ve fixed four of them so far.*

* The other two are in a PDF export that uses a font without emoji anyway, so the customer sees a box either way. That’s a different bug, and it’s on the list.