Set the boundaries first: the regex before BPE

Part 28 trained 20 BPE merges on the whole Shakespeare text. The first merged e with a space; later rules joined t to a space, a comma to a space, and a colon to a newline. All are valid choices for BPE: the most frequent adjacent pair wins.

But a word followed by a period, an exclamation mark, or a newline can acquire different representations. Vocabulary entries are spent memorizing these combinations. GPT-2 adds a constraint before BPE: split text into character-category chunks, then prohibit merges across chunk boundaries. A single leading space is an intentional exception.

Text → regex chunks → each chunk's UTF-8 bytes → independent BPE → concatenate IDs

A chunk is not a token. The regex determines where merging is forbidden; learned rules determine how many tokens remain inside each chunk.

Read GPT-2’s regex branch by branch

Historical encoder.py imports the third-party regex package as re:

import regex as re

pattern = r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"""

re.findall(pattern, "Hello've world123 how's are you!!!?")
# ['Hello', "'ve", ' world', '123', ' how', "'s", ' are', ' you', '!!!?']

The standard re module does not support \p{L} or \p{N}. L means Unicode letters, including Hanzi, not just A–Z; N means numbers. \s matches whitespace and \S non-whitespace.

The alternatives separated by | are tried in order at the current position as matching advances through the input. They are not independent results to concatenate.

The space in ? is literal; ? makes that one space optional. It does not mean arbitrary whitespace, so this prefix does not attach a tab to a word.

Why four indentation spaces become three plus one

Writing spaces as ·:

Input:  ····if
Chunks: ['···', '·if']

At the first space, the earlier alternatives cannot skip four spaces to reach a letter. \s+ greedily consumes all four, leaving i. The negative lookahead (?!\S) fails because i is non-whitespace.

The quantifier backtracks by one space. Now the next character is whitespace, so the assertion succeeds. Matching resumes at the fourth space, where ?\p{L}+ consumes ' if'.

🔑 (?!\S) checks the following position without consuming it. It leaves one space for the next word. At a line ending, a whitespace run can be consumed in full.

This also explains why GPT-2’s one-token-per-indentation-space behavior does not come directly from the regex. The first three spaces are already in the same chunk; GPT-2’s learned vocabulary simply does not merge them.

Chinese is not chunked one character at a time

re.findall(pattern, "分词是大模型许多怪现象的根源。")
# ['分词是大模型许多怪现象的根源', '。']

The Hanzi all belong to \p{L} and have no intervening spaces. The regex does not segment Chinese words or individual characters. After conversion to UTF-8, BPE may split one character into several fragments or merge byte fragments across character boundaries. Character-category boundaries, character boundaries, and token boundaries are distinct.

cl100k_base changes the boundary rules

The video compares GPT-2 with the cl100k_base regex used for GPT-4:

pattern = r"""(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+"""

This is the video’s spelling. The local tiktoken 0.14.0 _pat_str uses optimizations such as possessive quantifiers. The comparisons below concern chunking behavior on these inputs, rather than textual equality between regex strings.

Four changes:

1. (?i:...) makes contractions case-insensitive. GPT-2 splits HOW'S into HOW, ', S; here it becomes HOW, 'S.
2. \p{N}{1,3} limits number groups to three digits and omits the optional leading space. 1234567 first becomes 123, 456, 7.
3. Letters may be preceded by one character excluding newlines, letters, and numbers. This allows .append and (y as chunks.
4. Punctuation may retain trailing newlines, with another branch for whitespace followed by newlines. )\n\n\n can stay together.

The last two affect code:

text = "x = self.append(y)\n\n\nreturn x"

Being allowed in one chunk is only a prerequisite. The vocabulary must also contain the merges needed to make it one token.

Cleaner digit groups need not reduce the count

GPT-2 treats 1234567 as one regex chunk, then learned rules split it into 123 | 45 | 67. cl100k_base first imposes 123 | 456 | 7 boundaries; each happens to be one token. Both use three tokens with different digit groupings.

127 + 677 = 804 also takes seven tokens in both:

gpt2:       127 | ·+ | ·6 | 77  | ·= | ·8 | 04
cl100k:     127 | ·+ | ·  | 677 | ·= | ·  | 804

The count alone does not say which chunks the model sees.

Multi-space tokens come from training

Both regex patterns split " hello world!!!" identically: three spaces, hello with a leading space, world with a leading space, and !!!. Count vocabulary entries consisting of at least two spaces:

The count of 85 applies to the local cl100k_base vocabulary. GPT-2 emits three separate 220 IDs; cl100k_base merges them into 262. Same chunks, different learned merges.

Training: count inside chunks, choose globally

Part 28’s get_stats created a dictionary each time. Let it accumulate into a supplied dictionary:

def get_stats(ids, counts=None):
    counts = {} if counts is None else counts
    for pair in zip(ids, ids[1:]):
        counts[pair] = counts.get(pair, 0) + 1
    return counts

Use None as the default, not mutable {}, which would retain shared state across calls. The main change replaces one ID list with one list per chunk:

chunks = re.findall(pattern, text)
ids = [list(chunk.encode("utf-8")) for chunk in chunks]
merges = {}
vocab = {i: bytes([i]) for i in range(256)}

for i in range(vocab_size - 256):
    stats = {}
    for chunk_ids in ids:
        get_stats(chunk_ids, stats)
    if not stats:
        break
    pair = max(stats, key=stats.get)
    idx = 256 + i
    ids = [merge(chunk_ids, pair, idx) for chunk_ids in ids]
    merges[pair] = idx
    vocab[idx] = vocab[pair[0]] + vocab[pair[1]]

merge still replaces every non-overlapping occurrence from left to right. If counts tie, max chooses the first inserted dictionary key; corpus traversal order can therefore affect ties.

🔑 There is not a separate trained table for each chunk. Count pairs inside each chunk into a shared stats dictionary, choose one pair globally, then replace it within every chunk. The vocabulary and rules are shared; a chunk’s last symbol never forms a candidate with the next chunk’s first symbol.

Fifteen bytes, three merges

the cat the hat becomes four chunks: the, cat, the, hat.

Fifteen bytes become nine tokens: compression 15 / 9 ≈ 1.67. Without chunking, round three would select the plus a space, crossing the letter/whitespace boundary.

Same Shakespeare text, one boundary change

The local input.txt contains 1,115,394 bytes. With 20 merges in both runs:

In the chunked run, the first pair (32,116) occurs 23,837 times and the second (104,101) 18,203 times. Merge twelve joins their new symbols into ' the', with 8,684 occurrences at that round. Trailing-space combinations disappear; leading-space combinations remain.

The slight compression loss is the cost of blocking cross-boundary merges. The benefit is the boundary constraint, not a promise of fewer tokens on every input. Frequency counts are rebuilt each training round; new input is encoded with fixed ranks, without those counts.

Compare boundaries and learned merges together

The diagram’s GPT-4 and GPT-4o labels refer to cl100k_base and o200k_base in the video and these comparisons, rather than specifying tokenizers for every current model. cl100k_base has 100,256 ordinary entries. Its 100,277 figure is the ID-space size, including special IDs and gaps, not the number of ordinary merge rules.

o200k_base adds letter-case boundaries: HelloWorld getElementById takes four tokens under cl100k_base and six under o200k_base. The Chinese sample instead drops from 19 to 14; GPT-2 needs 32. A newer tokenizer need not shorten every input. Examine both boundaries and vocabulary.

When a result is unexpected:

1. Print findall chunks to locate the boundary that first prohibits a merge.
2. Inspect rules and ranks within each chunk: is the pair absent, or did an earlier rule consume its symbols?
3. Inspect complete token bytes rather than assuming each byte fragment is a character.

The next article reads GPT-2’s encoder.py. Here, base IDs still equal raw byte values. That implementation first turns bytes into reversible character aliases, then performs string BPE — which is where vocabulary characters such as Ġ, ä, and ½ come from.