DevPath
Tracks

Unicode, Encodings, and Endianness

How text becomes bytes: code points, UTF-8, why length depends on what you count, mojibake, and how byte order (endianness) changes the layout of numbers.

intermediate40 min#unicode#utf-8#encoding#endianness#strings#bytes

Prerequisites: Bitwise Operations

Why it matters

Text looks simple until a name with an accent turns into garbage characters, a database column rejects an emoji, a string "of length 5" takes 11 bytes, or a file written on one machine reads as nonsense on another. Every one of those is the same confusion: a string is not bytes, and a character is not one byte. Anyone who handles user input, files, networks, or APIs needs a correct model of text, and the same byte-level thinking explains why numbers are laid out differently across machines.

Intuition

Computers only store numbers, so text needs two translations.

  1. Character set (Unicode): gives every character in every writing system a unique number called a code point. "A" is 65 (written U+0041), "é" is U+00E9, "€" is U+20AC, and an emoji such as the grinning face is U+1F600. Unicode defines over a hundred thousand of them. It says nothing about bytes.
  2. Encoding (UTF-8, UTF-16, UTF-32): a rule for turning those numbers into bytes for storage or transmission.

It is like names and spellings. A person (the code point) exists independently of how you write their name in different alphabets (the encodings). To read a file correctly you must know which encoding wrote it.

Texté
Code pointU+00E9 (233)
UTF-8 bytesc3 a9
Two separate steps. Unicode maps the character to a number, then an encoding maps that number to bytes. Read the bytes with the wrong encoding and you get mojibake.

How it works

UTF-8

UTF-8 is the dominant encoding on the web. It uses 1 to 4 bytes per code point, and the first byte tells you how many:

code point range       bytes   bit pattern
U+0000   to U+007F      1      0xxxxxxx
U+0080   to U+07FF      2      110xxxxx 10xxxxxx
U+0800   to U+FFFF      3      1110xxxx 10xxxxxx 10xxxxxx
U+10000  to U+10FFFF    4      11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

The x positions carry the bits of the code point. Useful properties follow.

0010
bits 15-12
0000
bits 11-8
1010
bits 7-4
1100
bits 3-0
U+20AC is 0x20AC, which is 0010 0000 1010 1100 in binary (16 bits). It falls in the 3-byte range U+0800 to U+FFFF.
step 1 / 4

How the euro sign U+20AC is packed into the 3-byte pattern. Only the 16 bits of the code point are reused; the 1110 and 10 prefixes tell a reader how long the character is.

  • ASCII compatible: the first 128 code points are single bytes identical to ASCII, so plain English text is the same in both.
  • Self-synchronizing: continuation bytes always start with 10, so after corruption you can find the next character start.
  • No zero bytes inside characters (other than U+0000), so it works with C-style strings.
  • No byte-order problem: it is defined as a byte sequence.
41
A
c3
é
a9
e2
€
82
ac
f0
😀
9f
98
80
The UTF-8 bytes of A, é, € and 😀 laid end to end (hex). Green is the first byte of each character, amber is a continuation byte. They take 1, 2, 3 and 4 bytes, and the first byte tells you how many follow.

Other encodings

  • UTF-16: 2 bytes for most characters and 4 bytes (a surrogate pair) for code points above U+FFFF. JavaScript strings and Windows APIs use it internally.
  • UTF-32: always 4 bytes per code point. Simple but wasteful.
  • Latin-1 (ISO-8859-1) and legacy code pages: 1 byte per character, covering only some languages. Decoding UTF-8 bytes as Latin-1 yields mojibake: "é" becomes "é".

What is the length of a string?

It depends on what you count.

  • Bytes in some encoding: what storage and network limits care about.
  • Code units: UTF-16 units in JavaScript, where "😀".length is 2.
  • Code points: what Python's len() counts, and what Array.from(str).length counts in JavaScript.
  • Grapheme clusters: what a user sees as one character. A flag, a family emoji, or "e" plus a combining accent is several code points but one visible character. Also, "é" can be one code point (U+00E9) or two (e plus a combining acute). Normalization (NFC and NFD) makes such strings comparable.

Endianness

A 32-bit number takes four bytes, so which comes first in memory? Big-endian stores the most significant byte first, like writing a number left to right. Little-endian stores the least significant byte first. Most x86 and ARM machines run little-endian; network protocols traditionally use big-endian ("network byte order").

value 0x12345678 at addresses 0..3

big-endian:     12 34 56 78
little-endian:  78 56 34 12

Endianness matters when you read binary files, parse network packets, or share memory across systems. It does not affect UTF-8 (a byte stream), but it does affect UTF-16 and UTF-32. That is why those files often start with a byte order mark (BOM, U+FEFF), so a reader can tell which order was used.

address 0address 1address 2address 3
big-endian
12
34
56
78
little-endian
78
56
34
12
The 32-bit value 0x12345678 as it sits in memory (hex bytes). Green is the most significant byte (12), amber the least significant (78). Same four bytes, opposite order.
OperationTimeNotes
Encode or decode a string of n charactersO(n)One pass over the bytes
Index the i-th code point in UTF-8O(n)Variable width, must scan
Index the i-th code point in UTF-32O(1)Fixed width, at 4x the memory
Compare two normalized stringsO(n)Normalize first if forms may differ

Code: encode by hand, compare lengths, flip byte order

The function below encodes one code point as UTF-8 using shifts and masks from the previous lesson, and the program checks it against the language's built-in encoder.

def utf8_encode(cp):
    """Encode one code point to UTF-8 bytes by hand."""
    if cp < 0x80:
        return bytes([cp])
    if cp < 0x800:
        return bytes([0xC0 | (cp >> 6), 0x80 | (cp & 0x3F)])
    if cp < 0x10000:
        return bytes([0xE0 | (cp >> 12), 0x80 | ((cp >> 6) & 0x3F), 0x80 | (cp & 0x3F)])
    return bytes([0xF0 | (cp >> 18), 0x80 | ((cp >> 12) & 0x3F),
                  0x80 | ((cp >> 6) & 0x3F), 0x80 | (cp & 0x3F)])
 
for ch in ["A", "é", "€", "😀"]:
    mine = utf8_encode(ord(ch))
    assert mine == ch.encode("utf-8")
    print(f"U+{ord(ch):04X} -> {mine.hex(' ')} ({len(mine)} bytes)")
 
text = "naïve 😀"
print(len(text), len(text.encode("utf-8")))
 
# the same decoded wrongly: UTF-8 bytes read as Latin-1
print("é".encode("utf-8").decode("latin-1"))
 
# endianness: the same 32-bit integer as bytes
n = 0x12345678
print(n.to_bytes(4, "big").hex(" "), "|", n.to_bytes(4, "little").hex(" "))

Both versions print:

U+0041 -> 41 (1 bytes)
U+00E9 -> c3 a9 (2 bytes)
U+20AC -> e2 82 ac (3 bytes)
U+1F600 -> f0 9f 98 80 (4 bytes)
7 11
é
12 34 56 78 | 78 56 34 12

The string "naïve 😀" has 7 code points but takes 11 bytes: n, a, v, e, and the space are 5 one-byte characters, "ï" is 2 bytes, and the emoji is 4, so 5 + 2 + 4 = 11. The last two lines show mojibake and the two byte orders of the same integer.

Common pitfalls

  • Treating len() as the visible character count: it counts code points in Python and UTF-16 units in JavaScript. Emoji, flags, and combined accents break assumptions. Use a grapheme-aware library for user-facing truncation.
  • Slicing or reversing in the middle of a character: cutting UTF-8 bytes at a fixed offset can split a multi-byte character. Cut on code point or grapheme boundaries.
  • Forgetting normalization: "é" as one code point and as e plus an accent look identical but compare unequal. Normalize to NFC before comparing or hashing.
  • Wrong or implicit encoding: opening a file without encoding="utf-8" may use a platform default. If you see "é" or question marks, the encoding was misread.
  • Byte limits versus character limits: a database field of 255 bytes holds fewer characters if they are non-ASCII.
  • Assuming one endianness: when reading binary formats, specify the byte order explicitly (int.from_bytes(..., "big"), or DataView flags).

Practice

How many bytes in UTF-8? easy

Hint

Use the table of ranges: below U+0080, below U+0800, below U+10000, and the rest.

Show solution

"A" is 1 byte, "é" (U+00E9) is 2, "€" (U+20AC) is 3, and an emoji such as U+1F600 is 4. The string "héllo" is 6 bytes: four ASCII letters plus 2 for "é".

Write a UTF-8 decoder for one character medium

Hint

The first byte's leading ones tell you the length; mask off the marker bits and shift in the continuation bits.

Show solution

Read byte b0. If b0 < 0x80, the code point is b0. If (b0 >> 5) == 0b110, it is ((b0 & 0x1F) << 6) | (b1 & 0x3F). For three bytes, the lead mask is 0x0F and you add one more continuation byte; for four, the lead mask is 0x07. Each continuation contributes its low six bits. Validate that continuations start with 10.

Count characters correctly in JavaScript medium

Hint

What does for...of iterate over in JavaScript, UTF-16 units or code points?

Show solution

"😀".length is 2 because the length counts UTF-16 code units. Array.from(s).length or [...s].length iterates by code point and gives 1. For user-visible characters (a family emoji is several code points joined), use Intl.Segmenter with granularity "grapheme".

Detect the machine's endianness hard

Hint

Store a multi-byte integer, then inspect its first byte.

Show solution

Write the 16-bit value 1 into a buffer in the machine's native order and read the first byte. If it is 1, the machine is little-endian; if 0, big-endian. In Python, sys.byteorder reports it directly. In JavaScript, create new Uint16Array([1]), view the same buffer with a Uint8Array, and check bytes[0] === 1. This works because typed arrays use the platform order, while DataView lets you choose.

Interview questions

  1. What is the difference between Unicode and UTF-8? Unicode assigns code points; UTF-8 is one way to encode those numbers as bytes.
  2. Why did UTF-8 win? It is ASCII compatible, compact for Latin text, has no byte-order issues, and is self-synchronizing.
  3. Why is "😀".length 2 in JavaScript? Strings are UTF-16, and that code point needs a surrogate pair (two units).
  4. What is endianness and when does it matter? The byte order of multi-byte numbers in memory or on the wire; it matters for binary file formats, network protocols, and UTF-16 or UTF-32.

Interviewers look for: separating characters, code points, and bytes, naming the encoding explicitly, and mentioning normalization for comparisons.

Quick check

You decode the UTF-8 bytes of the word cafe with an accent on the final e, using Latin-1 instead of UTF-8. What do you most likely see?

Finished this lesson?

Further resources

Related lessons