Falsehoods programmers believe about unicode

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Did you know about the "falsehoods programmers believe about ..."

I figured there need to be one on unicode.

You may also care about:

made a good start, and I started by copying most of that.

What unicode is and does

"Unicode is all about displaying characters"

It does not describe how to draw characters at all.

It does however deal with some typesetting semantics that would change change what would be drawn.

Note the 'some'.

"Unicode is all about characters"

Unicode includes various characters that only have semantic function, and cannot be drawn, but should be interpreted.

Consider, for example, a⁢b. That means 'a times b' to anything paying attention, because there is an TIMES U+INVISIBLE TIMES (&#xINVISIBLE TIMES;) inbetween.

That also means that most comparisons not paying attention will not see a⁢b as equivalent to ab.

And codepoints that have only internal function, such as ZWJ.

And codepoints that you could say have function only handled externally to it, such as flags.

"Every graphical thing has a single unicode representation"

Unicode ends up being one decent approach towards that goal.

But not the only, and not necessarily the best.

See e.g. the woes of UniHan.

"Unicode is all about semantics"

In the end, it cares mainly/only about semantics that would mess with its design goals, which are roughly to make a singular good set of characters.

Any other semantics? It mostly stays away from that pit of endless ambiguity.

"Every logicaly equivalent thing has a single unicode representation"

Exposé Exposé These are logically equivalent, but different codepoint sequences.

"Unicode is a table of code point numbers to glyphs."

A subset works out like that, sure.

"Unicode is a table of code point numbers to characters."

"Comparing two strings for equivalence is easy"

Ha, no.

Especially not if you want equivalence in human 'I would accept that as the same' terms.

"We can sort unicode by codepoint numbers"

Sorting characters is made impossible by people shenanigans - e.g. Aarhus.

"Uppercasing is the opposite of lowercasing."

"To check if string1 and string2 are equal case-insensitively, I can use lowercase(string1) == lowercase(string2), or uppercase(string1) == uppercase(string2)"

That works for many things, yes. But not all.

Encoding stuff

"UTF-8 is worse than UTF-16 in any place that uses CJK"

A while ago there was this idea that UTF-8 gets very chunky for all CJK, and UTF-16 is smaller or simpler.

https://web.archive.org/web/20250422221838/https://utf8everywhere.org/

"Windows is UCS2" / "Windows is UTF-16"

Parts of it are UTF-16, but data that comes from it cannot be strongly assumed to be valid UTF-16.

(also parts of it are ANSI, basically meaning codepage. It get remapped to wide on rendering, so that should only mean a (severe) character limit and cases where you didn't specify the correct encoding to convert from)

Many of those functions were, a long time ago, only UCS2, meaning that old versions of windows cannot show all of unicode. Also, some libraries that have origins that old also have issues.

There are some rough edges left over from this transition(verify).

"UTF-16 interfaces never not give you malformed UTF-16"

Windows's UTF-16 does not enforce validity, so happily gives you malformed UTF-16.

Javascript UTF-16 does not enforce validity.

There are certainly others.

When many functions pay attention, it may not be easy to get malformed UTF-16 in there, but once it's there it may be a right pain.

Strings become simpler yea?

The fact that we are trying to do everything, well, means more complexity.

Given that new compexity, it's tamed quite well, actually.

But you can't take away the fact we now need to think a little harder about all that everything.

"The length of a Unicode string is well-defined."

Consider U+1F4A3 (💣)

Let's ask Javascript to put that into a string:

String.fromCodePoint(0x1F4A3)

Single-character string, right?

Well, it has .length 2 actually.

Because JS is a UTF-16 implementation.

And this isn't in BMP so it's actually two surrogate codepoints, U+d83d and U+dca3

"Okay, then the codepoint length is best"=

Exposé is 7 codepoints, Exposé is 6 codepoints.

"Okay, in NFC each code point corresponds to one user-perceived character."

That solves the just-mentioned issue, but we weren't even getting started.

👨🏻‍👩🏽‍👧🏿‍👦🏾 has eleven codepoints.

﷽ has one codepoint

"Okay, then the grapheme cluster count is best"

It improves some cases, absolutely, because it involves the segmentation algorithm.

For estimates, anyway.

"Grapheme clusters are fixed"

The segmentation algorithm relies on the unicode database, it will give slightly different answers between unicode versions.

Sure the characters we use most are not likely to change anymore, but still.

"Grapheme clusters are fixed for a unicode version"

Mostly true, in that grapheme clustering is defined in a way where the default behaviour is locale-independent. But not required to be (verify)

https://www.reddit.com/r/ProgrammingLanguages/comments/1e5dapz/unicode_grapheme_clusters_and_parsing/

"None of this length stuff affects me"

If you limit the length of a text field, what do you use?

Consider

limiting the size of text fields.

...length of what?

substring

not even cursor/selection, those ought to be handled for you.

Grapheme clusters seems to be a much better idea.

"Paths are just strings, so file paths can be considered unicode / can be read into Unicode"

In unices, they should be assumed to be bytes. Maybe those bytes represent unicode, maybe it's a codepage thing.

In windows, they should be assumed to be UCS2 or UTF-16 but cannot be assumed to be valid. And you may need to keep the invalid string if you want to save back to the same file.

The path libraries in cross-platform languages do some dancing around to deal with this.

python basically has two distinct implementations between windows and posix.

Rust internally defines wtf8 to deal with some windows nonsense, though (luckily) shields you from that implementation.

"Changes between normal forms is two-way"

Only if they are canonically equivalent, yes. For example:

U+E9 (é) compatibility-decomposes into U+0045 U+0301,

U+0045 U+0301 compatibility-composes into into U+E9

If they are compatibility equivalent, no. For example

U+2163 (Ⅳ) decomposes into U+0049 U+0056

nothing composes into U+2163

...and there are a lot of cases like that.

"Programming languages thing in terms of codepoints, or UTF-32"

Except in systems where functions are UTF-16 in, UTF-16 out.

Except where functions are UTF-8 in, UTF-8 out.

"There is a name for every Unicode code point."

"Okay, but there is a name for every assigned/usable Unicode code point."

Still nope.

See PUAs.

"UTF-8 encodes all of Unicode"

Oh yeah absolutely.

...I mean, except when someone implements it sorta-wrong, as a feature to save space.

Consider MySQL, which defaulted encoding to 'utf8' for a long time, which which they mean utf8mb3, which is a sort of custom implementation that can only store up to U+FFFF.

They never changed the meaning of 'utf8' either, they just say it's deprecated. Databases default to utf8mb4 since 8.0, so it's correct by default - but if you think 'utf8' means UTF-8, or don't notice that you selected utf8mb3 rather than utf8mb4, you will break your database's text.

Rendering

"Unicode also automatically solves rendering and fonts"

It makes things less terrible, but not simple.

"Fonts can show all of unicode"

Aside from the fact that most fonts can only contain sixty-five thousand glyphs, and unicode is larger?

"I can get a quick estimate of visual length"

Depends on how quick and how accurate you want it.

"I can at least make this simpler in cases of fixed-with drawing"

Fixed-width is another lie we tell ourselves to keep from despair.

In particular, CJK characters are often made two columns wide, because they would not be readable in one.

"A string has a singular correct rendering."

Define correct.

Again, consider UniHan.

This meant some codepoints should get displayed differently depending on the language.

The differences are usually subtle, but they are important.

This is one of the core reasons UniHan annoyed a lot of people.

"Fonts should fall back to something for every codepoint"

Technically yes, but 'something' does not need to resemble anything useful.

"Unicode can't cause stability issues."

See unicode crashed iphone.

"I can give my OS some UTF-8 and it'll just render it"

There are layers to that working correctly.

And there are cases it works, but not correctly.

And there are cases it does not work at all.

There will be a reason for it, but whether it is reasonable, well...

"The same character always has the same width contribution"

No.

It should be obvious that it can vary between fonts, so let's assume we are talking within the same font.

consider:

ligatures.
unicode grapheme clusters.
unicode combining characters - with different others.
contextual glyph substitution.
kerning
script shaping rules.
hinting and subpixel rendering.
different rendering engines; e.g. CSS adds some specific rules, that are not followed identically.
switches between RTL and LTR
CJK characters define different widths (e.g. fullwidth, halfwidth, and more) and how that gets used may depend on context.
font fallbacks
emoji, many of which have two unicode representations
emoji, which may have additional app/OS representations
a renderer simulating bold and italic variations
scaling can lead to rounding details

"You can associate any glyph on screen with a character in the source text" / "Every character is represented by one glyph"