Unicode, Encodings, and Endianness
How text becomes bytes: code points, UTF-8, why length depends on what you count, mojibake, and how byte order (endianness) changes the layout of numbers.
intermediate40 min#unicode#utf-8#encoding#endianness#strings#bytes
Prerequisites: Bitwise Operations
Why it matters
Text looks simple until a name with an accent turns into garbage characters, a database column rejects an emoji, a string "of length 5" takes 11 bytes, or a file written on one machine reads as nonsense on another. Every one of those is the same confusion: a string is not bytes, and a character is not one byte. Anyone who handles user input, files, networks, or APIs needs a correct model of text, and the same byte-level thinking explains why numbers are laid out differently across machines.
Intuition
Computers only store numbers, so text needs two translations.
- Character set (Unicode): gives every character in every writing system a unique number called a code point. "A" is 65 (written
U+0041), "é" isU+00E9, "€" isU+20AC, and an emoji such as the grinning face isU+1F600. Unicode defines over a hundred thousand of them. It says nothing about bytes. - Encoding (UTF-8, UTF-16, UTF-32): a rule for turning those numbers into bytes for storage or transmission.
It is like names and spellings. A person (the code point) exists independently of how you write their name in different alphabets (the encodings). To read a file correctly you must know which encoding wrote it.
How it works
UTF-8
UTF-8 is the dominant encoding on the web. It uses 1 to 4 bytes per code point, and the first byte tells you how many:
code point range bytes bit pattern
U+0000 to U+007F 1 0xxxxxxx
U+0080 to U+07FF 2 110xxxxx 10xxxxxx
U+0800 to U+FFFF 3 1110xxxx 10xxxxxx 10xxxxxx
U+10000 to U+10FFFF 4 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
The x positions carry the bits of the code point. Useful properties follow.
How the euro sign U+20AC is packed into the 3-byte pattern. Only the 16 bits of the code point are reused; the 1110 and 10 prefixes tell a reader how long the character is.
- ASCII compatible: the first 128 code points are single bytes identical to ASCII, so plain English text is the same in both.
- Self-synchronizing: continuation bytes always start with
10, so after corruption you can find the next character start. - No zero bytes inside characters (other than U+0000), so it works with C-style strings.
- No byte-order problem: it is defined as a byte sequence.
Other encodings
- UTF-16: 2 bytes for most characters and 4 bytes (a surrogate pair) for code points above U+FFFF. JavaScript strings and Windows APIs use it internally.
- UTF-32: always 4 bytes per code point. Simple but wasteful.
- Latin-1 (ISO-8859-1) and legacy code pages: 1 byte per character, covering only some languages. Decoding UTF-8 bytes as Latin-1 yields mojibake: "é" becomes "é".
What is the length of a string?
It depends on what you count.
- Bytes in some encoding: what storage and network limits care about.
- Code units: UTF-16 units in JavaScript, where
"😀".lengthis 2. - Code points: what Python's
len()counts, and whatArray.from(str).lengthcounts in JavaScript. - Grapheme clusters: what a user sees as one character. A flag, a family emoji, or "e" plus a combining accent is several code points but one visible character. Also, "é" can be one code point (
U+00E9) or two (eplus a combining acute). Normalization (NFC and NFD) makes such strings comparable.
Endianness
A 32-bit number takes four bytes, so which comes first in memory? Big-endian stores the most significant byte first, like writing a number left to right. Little-endian stores the least significant byte first. Most x86 and ARM machines run little-endian; network protocols traditionally use big-endian ("network byte order").
value 0x12345678 at addresses 0..3
big-endian: 12 34 56 78
little-endian: 78 56 34 12
Endianness matters when you read binary files, parse network packets, or share memory across systems. It does not affect UTF-8 (a byte stream), but it does affect UTF-16 and UTF-32. That is why those files often start with a byte order mark (BOM, U+FEFF), so a reader can tell which order was used.
| Operation | Time | Notes |
|---|---|---|
| Encode or decode a string of n characters | O(n) | One pass over the bytes |
| Index the i-th code point in UTF-8 | O(n) | Variable width, must scan |
| Index the i-th code point in UTF-32 | O(1) | Fixed width, at 4x the memory |
| Compare two normalized strings | O(n) | Normalize first if forms may differ |
Code: encode by hand, compare lengths, flip byte order
The function below encodes one code point as UTF-8 using shifts and masks from the previous lesson, and the program checks it against the language's built-in encoder.
def utf8_encode(cp):
"""Encode one code point to UTF-8 bytes by hand."""
if cp < 0x80:
return bytes([cp])
if cp < 0x800:
return bytes([0xC0 | (cp >> 6), 0x80 | (cp & 0x3F)])
if cp < 0x10000:
return bytes([0xE0 | (cp >> 12), 0x80 | ((cp >> 6) & 0x3F), 0x80 | (cp & 0x3F)])
return bytes([0xF0 | (cp >> 18), 0x80 | ((cp >> 12) & 0x3F),
0x80 | ((cp >> 6) & 0x3F), 0x80 | (cp & 0x3F)])
for ch in ["A", "é", "€", "😀"]:
mine = utf8_encode(ord(ch))
assert mine == ch.encode("utf-8")
print(f"U+{ord(ch):04X} -> {mine.hex(' ')} ({len(mine)} bytes)")
text = "naïve 😀"
print(len(text), len(text.encode("utf-8")))
# the same decoded wrongly: UTF-8 bytes read as Latin-1
print("é".encode("utf-8").decode("latin-1"))
# endianness: the same 32-bit integer as bytes
n = 0x12345678
print(n.to_bytes(4, "big").hex(" "), "|", n.to_bytes(4, "little").hex(" "))const hex = (bytes) => Array.from(bytes, (b) => b.toString(16).padStart(2, "0")).join(" ");
function utf8Encode(cp) {
// Encode one code point to UTF-8 bytes by hand.
if (cp < 0x80) return [cp];
if (cp < 0x800) return [0xc0 | (cp >> 6), 0x80 | (cp & 0x3f)];
if (cp < 0x10000) return [0xe0 | (cp >> 12), 0x80 | ((cp >> 6) & 0x3f), 0x80 | (cp & 0x3f)];
return [0xf0 | (cp >> 18), 0x80 | ((cp >> 12) & 0x3f), 0x80 | ((cp >> 6) & 0x3f), 0x80 | (cp & 0x3f)];
}
const enc = new TextEncoder();
for (const ch of ["A", "é", "€", "😀"]) {
const cp = ch.codePointAt(0);
const mine = utf8Encode(cp);
if (hex(mine) !== hex(enc.encode(ch))) throw new Error("mismatch");
console.log(`U+${cp.toString(16).toUpperCase().padStart(4, "0")} -> ${hex(mine)} (${mine.length} bytes)`);
}
const text = "naïve 😀";
console.log(Array.from(text).length, enc.encode(text).length);
// the same decoded wrongly: UTF-8 bytes read as Latin-1
console.log(new TextDecoder("latin1").decode(enc.encode("é")));
// endianness: the same 32-bit integer as bytes
const view = new DataView(new ArrayBuffer(4));
view.setUint32(0, 0x12345678, false);
const big = hex(new Uint8Array(view.buffer));
view.setUint32(0, 0x12345678, true);
const little = hex(new Uint8Array(view.buffer));
console.log(big, "|", little);Both versions print:
U+0041 -> 41 (1 bytes)
U+00E9 -> c3 a9 (2 bytes)
U+20AC -> e2 82 ac (3 bytes)
U+1F600 -> f0 9f 98 80 (4 bytes)
7 11
é
12 34 56 78 | 78 56 34 12
The string "naïve 😀" has 7 code points but takes 11 bytes: n, a, v, e, and the space are 5 one-byte characters, "ï" is 2 bytes, and the emoji is 4, so 5 + 2 + 4 = 11. The last two lines show mojibake and the two byte orders of the same integer.
Common pitfalls
- Treating
len()as the visible character count: it counts code points in Python and UTF-16 units in JavaScript. Emoji, flags, and combined accents break assumptions. Use a grapheme-aware library for user-facing truncation. - Slicing or reversing in the middle of a character: cutting UTF-8 bytes at a fixed offset can split a multi-byte character. Cut on code point or grapheme boundaries.
- Forgetting normalization:
"é"as one code point and aseplus an accent look identical but compare unequal. Normalize to NFC before comparing or hashing. - Wrong or implicit encoding: opening a file without
encoding="utf-8"may use a platform default. If you see "é" or question marks, the encoding was misread. - Byte limits versus character limits: a database field of 255 bytes holds fewer characters if they are non-ASCII.
- Assuming one endianness: when reading binary formats, specify the byte order explicitly (
int.from_bytes(..., "big"), orDataViewflags).
Practice
How many bytes in UTF-8? easy
Hint
Use the table of ranges: below U+0080, below U+0800, below U+10000, and the rest.
Show solution
"A" is 1 byte, "é" (U+00E9) is 2, "€" (U+20AC) is 3, and an emoji such as U+1F600 is 4. The string "héllo" is 6 bytes: four ASCII letters plus 2 for "é".
Write a UTF-8 decoder for one character medium
Hint
The first byte's leading ones tell you the length; mask off the marker bits and shift in the continuation bits.
Show solution
Read byte b0. If b0 < 0x80, the code point is b0. If (b0 >> 5) == 0b110, it is ((b0 & 0x1F) << 6) | (b1 & 0x3F). For three bytes, the lead mask is 0x0F and you add one more continuation byte; for four, the lead mask is 0x07. Each continuation contributes its low six bits. Validate that continuations start with 10.
Count characters correctly in JavaScript medium
Hint
What does for...of iterate over in JavaScript, UTF-16 units or code points?
Show solution
"😀".length is 2 because the length counts UTF-16 code units. Array.from(s).length or [...s].length iterates by code point and gives 1. For user-visible characters (a family emoji is several code points joined), use Intl.Segmenter with granularity "grapheme".
Detect the machine's endianness hard
Hint
Store a multi-byte integer, then inspect its first byte.
Show solution
Write the 16-bit value 1 into a buffer in the machine's native order and read the first byte. If it is 1, the machine is little-endian; if 0, big-endian. In Python, sys.byteorder reports it directly. In JavaScript, create new Uint16Array([1]), view the same buffer with a Uint8Array, and check bytes[0] === 1. This works because typed arrays use the platform order, while DataView lets you choose.
Interview questions
- What is the difference between Unicode and UTF-8? Unicode assigns code points; UTF-8 is one way to encode those numbers as bytes.
- Why did UTF-8 win? It is ASCII compatible, compact for Latin text, has no byte-order issues, and is self-synchronizing.
- Why is
"😀".length2 in JavaScript? Strings are UTF-16, and that code point needs a surrogate pair (two units). - What is endianness and when does it matter? The byte order of multi-byte numbers in memory or on the wire; it matters for binary file formats, network protocols, and UTF-16 or UTF-32.
Interviewers look for: separating characters, code points, and bytes, naming the encoding explicitly, and mentioning normalization for comparisons.
Quick check
You decode the UTF-8 bytes of the word cafe with an accent on the final e, using Latin-1 instead of UTF-8. What do you most likely see?
Finished this lesson?
Further resources
- Unicode (Wikipedia) (article)
- UTF-8 (Wikipedia) (article)
- Endianness (Wikipedia) (article)
- Python Unicode HOWTO (docs)