Basics Catch-Up · One Character, Five Byte Strings: Charsets Number, Encodings Store

Part 23 gave this chain:

text  →  [tokenizer]  →  integers  →  [model]  →  integer  →  [decode]  →  text

There the vocabulary was 65 characters, one ID each, looked up straight through stoi. Real models do not work that way — their vocabulary is derived from a corpus, and what gets counted is bytes. The first line of BPE in the next section is:

tokens = list(text.encode("utf-8"))

That text.encode("utf-8") step takes a few minutes in the video; this part finishes it off. And it is needed beyond tokenizing: cleaning a Chinese corpus, de-duplicating it, or working out why an old file opens as garbage all run into it.

The one-line version: a charset governs the numbering (which character this is), an encoding governs the storage (which bytes that number becomes). Unicode only numbers, UTF-8 / 16 / 32 only store, and ASCII and the GB standards do both at once.

One byte cannot hold one Hanzi

A computer stores nothing but 0s and 1s. 8 bits = 1 byte, which can represent 0—255, 256 numbers in all. To store text, you first have to agree on which character corresponds to which number. ASCII (1963) is exactly that agreement: 128 characters, numbered 0—127, and one byte holds them.

'A' = 65    'a' = 97    '0' = 48    space = 32    newline = 10

Part 23’s remark that “the vocabulary is sorted by ASCII: 0 is newline, 1 is space, then punctuation, upper case, lower case” is sorting this very table — newline 10 < space 32 < punctuation < upper case from 65 < lower case from 97.

128 is only enough for English, so each country extended it on its own: mainland China’s GB2312 (1980, 6,763 common Hanzi) then GBK (1995, over twenty thousand), Taiwan’s Big5, Japan’s Shift_JIS. None of them are compatible with each other, and each numbers characters its own way.

🔑 Mojibake: the same bytes, read under a different agreement

Both of these lines can be run as they stand:

"你好".encode("utf-8").decode("gbk")            # '浣犲ソ'
bytes.fromhex("efbfbd efbfbd").decode("gbk")   # '锟斤拷'

The first: the 6 bytes UTF-8 produced get read two at a time as GBK, and come out as three other characters. The second is more roundabout — EF BF BD is the byte sequence of UTF-8’s replacement character �, and two �, 6 bytes in all, read as GBK are exactly 锟斤拷. So 锟斤拷 is mojibake of mojibake: first bytes broke and became �, then those � were read under the wrong encoding.

🔑 Bytes carry no information about which encoding they are. Mojibake is never broken bytes; it is a reader using the wrong agreement.

Two layers: charsets number, encodings store

Four rows on the single character 你 are enough to separate the two layers:

Python has one type for each — str is a sequence of code points, bytes a sequence of bytes:

s = "你"
ord(s)                           # 20320        — the number
s.encode("utf-8").hex(" ")       # 'e4 bd a0'   — storage (UTF-8)
s.encode("utf-16-be").hex(" ")   # '4f 60'      — storage (UTF-16)
b"\xe4\xbd\xa0".decode("utf-8")  # '你'         — bytes back to a code point

encode = code point → bytes, decode = bytes → code point. Get the direction backwards and the error you get is far from obvious.

Why the ASCII era did not need the distinction

Think of a byte as a small box that holds only 0—255:

• ASCII: the largest number is 127, so one box holds it. “A” is number 65 and stores as one byte, 65 — the number is the byte, the two layers coincide, and there is nothing to separate
• Unicode: numbers run all the way to 1,114,111 (0x10FFFF), 你 is 20320, and one box will not hold it, so “how to split it across boxes” becomes a separate question — which is what the second layer is for

The three UTFs are three ways of splitting: UTF-32 always uses 4 boxes with leading zeros, UTF-16 generally 2, and UTF-8 one for small numbers and 2—4 for large ones, with each box also carrying a mark saying which position it holds.

❗ Not parallel: it depends what you are comparing

ASCII, GBK, UTF-8, UTF-16 and UTF-32 are often listed side by side, but they are not on one layer. What matters is what you are comparing:

So “Unicode” is not a sixth way to store; it is the first step shared by the three UTFs. That also settles two asymmetries:

• UTF-8 ↔ UTF-16 ↔ UTF-32 convert by formula, because all three use the same number
• GBK ↔ UTF-8 needs a lookup table, because the numbers on each side come from different tables entirely

GBK has two steps too, written into one standard

GBK is not unlayered; it writes both layers into one standard. In the GB table 你 sits at zone 36, position 67, and adding 160 to each gives 196, 227 — that is, C4 E3:

b = "你".encode("gbk")      # b'\xc4\xe3'
b[0] - 160, b[1] - 160      # (36, 67)  — zone, position

That “zone 36, position 67” has nothing to do with Unicode’s 20320, which makes the “lookup table only” point above a necessity rather than an implementation shortcut.

💡 GB18030 (2000) covers every Unicode character and can be read as “another way of storing Unicode that stays GBK-compatible” — "你".encode("gb18030") is also C4 E3, but it can store characters GBK cannot, such as emoji.

UTF-8’s templates, and working one out by hand

Each row’s x bits are exactly enough for that row’s upper bound: 7 bits reach 127, 11 reach 2,047, 16 reach 65,535, not one more and not one fewer. Only the fourth row does not match — 21 bits reach 2,097,151, yet Unicode stops at 1,114,111. The excess is not held in reserve; it was cut off: the ceiling is fixed by UTF-16 (the section on surrogate pairs below works the number out), and although UTF-8 could originally run to 6 bytes, RFC 3629 in 2003 trimmed it to 4 and to “the UTF-16 accessible range” only — anything beyond is invalid.

Working 你 out by hand, in four steps

① ord('你') = 20320, which falls in 2,048—65,535  →  the 3-byte template, 16 x bits in all
② write 20320 as 16 binary digits                →  0100111101100000
③ split it 4 | 6 | 6 as the template needs       →  0100  111101  100000
④ fill the x bits in order                       →  11100100  10111101  10100000
                                                     228       189       160

Check it in Python and list("你".encode("utf-8")) is [228, 189, 160]. Work out 好 (22909) yourself for practice; the answer is [229, 165, 189] (E5 A5 BD).

Reading the bytes back

No separate length information is needed — the length is written into the first byte’s opening bits:

For [228, 189, 160]: 228 = 1110 0100, three 1s then a 0 → the start of a 3-byte character, so take the two following bytes that open with 10, reassemble the coloured bits into 0100 111101 100000 = 20320 → 你.

❗ Fitting in one byte does not mean UTF-8 lets you store it in one

“If it fits, store it directly” only holds for 0—127. é is 233, which fits in a byte numerically, but UTF-8 will not allow it: 233 = 1110 1001 opens with 1110, and a reader would take it for the start of a 3-byte character. So it has to use the 2-byte template:

é = 233 = 00011 101001 (11 bits)
  → 110 00011 | 10 101001 = C3 A9 → [195, 169]

The old Western European encoding Latin-1 stores é as exactly that single byte 233, so mixing the two gives another very common kind of mojibake:

"é".encode("utf-8").decode("latin-1")    # 'é'  two bytes read as two characters

🔑 UTF-8 hands the whole of 128—255 over to marker bits, leaving a lone byte only 0—127 to store as is — that is the price of ASCII compatibility. The same design (every byte spends bits on markers) is also why a Hanzi takes 3 bytes: the 2-byte template has only 11 bits left for the code point, too few for 20320.

❗ Why half a Hanzi is �

For [228, 189]: the opening declares 3 bytes and only 2 arrived. Declaration and fact contradict each other, and the decoder has no other option — it reports the replacement character U+FFFD, the �:

bytes([228, 189]).decode("utf-8", errors="replace")   # '�'

Any truncated byte stream runs into it: reading a file in fixed-size chunks, a network packet cut in half, a log truncated by byte count. BPE’s “byte fragments” are the same thing — when a Hanzi’s 3 bytes have not been merged into one block, none of them alone is valid UTF-8.

Why UTF-8 became the default

All three reasons come straight off that template table:

• ASCII-compatible: the 1-byte template 0xxxxxxx is ASCII, so a pure-English file is byte for byte identical under both
• cheapest for English: one byte per letter, against 2 for UTF-16 and 4 for UTF-32
• fault-tolerant: anything opening with 10 is a continuation byte, so from any position you can skip forward over 10 openings and find the next character’s start — one broken byte damages one character instead of throwing off everything after it

UTF-16, UTF-32, and three kinds of “length”

The number in each name is the size of the unit, not how many bytes a character takes: the “8” in UTF-8 means 8 bits = 1 byte, and a character uses 1—4 of those units depending on its code point (“A” one, 你 three); UTF-16’s unit is 2 bytes and UTF-32’s is 4.

• UTF-32: 4 bytes per character, storing the code point’s value directly. The simplest, but three times the size for English
• UTF-16: units of 2 bytes. Code points ≤ 65,535 (common Hanzi included) take 2 bytes, larger ones take 4

Chinese is actually shorter in UTF-16

你 is 4F 60, 2 bytes, one fewer than UTF-8’s 3. The same passages measured across all three:

Pure Chinese really is cheaper in UTF-16. Yet files, web pages and Linux are essentially all UTF-8 — because real text is laced with tags, punctuation, code and English, and those take 1 byte in UTF-8 against 2 in UTF-16. Being ASCII-compatible counts for far more than saving one byte per Hanzi.

Byte order and the BOM

A two-byte number can lead with its high byte (big-endian) or its low byte (little-endian) — even UTF-16 has two byte sequences of its own, 4F 60 and 60 4F. To say which, a file may open with two extra bytes:

list("hi".encode("utf-16"))[:2]      # [255, 254]  — a BOM, marking little-endian

So take care when counting UTF-16 bytes: encode("utf-16") is 2 bytes longer than encode("utf-16-le"), and the table above uses the -le form, without the BOM.

❗ One string, three kinds of “length”

Java / Kotlin and JavaScript hold strings as UTF-16 internally, Python 3’s str is a sequence of code points, and what lands in a file is UTF-8 bytes. All three numbers can differ:

That emoji row is the classic trap in Java and JS: "👋".length is 2, and substring(0, 1) cuts out half a character. “One character” is an imprecise phrase in itself — you have to say whether you mean a code point, a UTF-16 unit, or a byte.

Why 👋 takes 2 units in UTF-16: surrogate pairs

A UTF-16 unit is 16 bits, and a code point ≤ 65,535 goes in as is, with no markers at all (你 is stored straight as 4F 60). Anything above 65,535 does not fit, so it is split across two units, both marked — a surrogate pair. Take 𠮷 (a variant of 吉, code point 134071):

① subtract 65,536 first:     134071 − 65536 = 68535, exactly 20 bits
                             0001000010 | 1110110111      ← first 10 bits | last 10 bits
② mark the first 10, 110110:  110110 0001000010 = D8 42
③ mark the last 10, 110111:   110111 1110110111 = DF B7
→ UTF-16 = D8 42 DF B7 (2 units, 4 bytes)

How does a reader tell the two units belong together? Unicode sets aside the range D800—DFFF and never assigns it to any character: a unit in D800—DBFF is the first half of a pair, DC00—DFFF the second, and any other value is a whole character. UTF-8 finds boundaries by the marker bits opening each byte; UTF-16 by a block of numbers held permanently unused.

🔑 Unicode’s ceiling is worked out right here: 20 bits can represent 1,048,576 values, and with the 0—65,535 stored directly on top, the largest code point is exactly 1,114,111. The fourth row of the template table above could reach 2,097,151 with its 21 bits — this is the part that was cut off.

The code point is the hub

Whatever format the bytes are in, decoding them by the matching rule returns the same code point; and from that code point they can be stored in any format:

UTF-8 bytes  ─┐                             ┌─→ UTF-8 bytes
UTF-16 bytes ─┤   decode           encode   ├─→ UTF-16 bytes
UTF-32 bytes ─┼──────────→  code point ─────┼─→ UTF-32 bytes
GBK bytes    ─┘  (lookup)      20320        └─→ GBK bytes (lookup)

So converting between encodings is always two steps: decode to code points, then encode the other way. The middle step cannot be skipped:

text = open("old.txt", encoding="gbk").read()          # GBK bytes → code points
open("new.txt", "w", encoding="utf-8").write(text)     # code points → UTF-8 bytes

Three things have to hold for that route to work: you have to know the original encoding (the bytes will not tell you), the bytes have to be complete and valid ([228, 189] converts to no code point), and the GB standards need a lookup table — and GBK only collected twenty-odd thousand Hanzi, so characters it lacks, emoji among them, simply cannot be stored (GB18030 does not have this problem).

❗ Bytes alone do not reveal which encoding produced them

"你".encode("utf-16-be").decode("utf-8")   # 'O`'

Those two bytes 4F 60 are 你 read as UTF-16, and two ASCII characters O and ` (79 and 96) read as UTF-8. Both readings are valid, neither raises an error, and only one is right.

Which is why the encoding has to be stated explicitly everywhere: <meta charset="utf-8"> on a web page, open(path, encoding="utf-8") when Python reads a file, the encoding shown bottom right in VS Code. decode not raising does not mean you read it correctly.

Back to the tokenizer: the floor is 256 byte values

BPE’s first line now reads plainly:

tokens = list(text.encode("utf-8"))    # 你 → [228, 189, 160]

Any text becomes a string of integers from 0 to 255 before it reaches the tokenizer. BPE then counts which adjacent pair occurs most often across those bytes and merges it into a new entry.

Part 23’s “down to characters” should read “down to bytes”

Writing out BPE’s four steps, part 23 said “start with a vocabulary of every single character” and that rare words break “all the way down to characters” — that is the textbook version. Real byte-level BPE (from GPT-2 onward) has 256 byte values as its floor, and it has been corrected.

The difference is not only wording. Under “down to characters”, “OOV never happens” is a promise that needs a fallback table — what happens on a rare character the vocabulary lacks? Under “down to bytes” it becomes a mechanical fact:

🔑 A byte has only 256 possible values, and the vocabulary takes all 256 in, so any byte sequence is necessarily representable. No <UNK> is needed and no fallback rule — not because it was designed well, but because there is no other possibility.

A Hanzi starts at 3 bytes

This one fact settles the token arithmetic for Chinese. A common Hanzi is 3 bytes, so:

• merged into one block → 1 token
• not merged at all → 3 tokens (three byte fragments)

The sentence 分词是大模型许多怪现象的根源。 is 15 characters and 45 UTF-8 bytes (measured). With no character merged at all it would be 45 tokens; with every character merged into a block, 15 — a real tokenizer lands in between, and where depends on how much Chinese it learned from. How far apart two tokenizers can be is in the next part, with figures OpenAI published.

⚠️ This is the mechanism only, not a conversion into “Chinese costs N times English”. How many tokens a Hanzi actually costs depends on how much Chinese the tokenizer saw in training and which characters it merged into blocks — that belongs to BPE’s training, not to UTF-8. UTF-8 only fixes the floor and the ceiling.

Training data has to be normalised to UTF-8 first

If a corpus mixes GBK and UTF-16 files, the same 你 turns up as C4 E3, 4F 60 and E4 BD A0, three entirely different byte strings. The model sees three unrelated symbols and learns them as three different things.

Which is why normalising the encoding is the first step in cleaning web text — and by the point above, that step has to be “decode by the original encoding, then encode as UTF-8”, never a substitution performed on the bytes.

This part introduces no new mechanism. It only takes “text → bytes” apart: a charset numbers the characters, an encoding stores that number as bytes, and the code point is the one hub between them.

The one-line version: there is a single number and five ways to store it; bytes carry no account of themselves, so the encoding has to be stated.

🔮 The next section merges on exactly those 256 bytes — count which adjacent pair of bytes occurs most often, merge it into a new entry, repeat until the vocabulary is large enough. Everything after encode starts there.