What is Ġ in the vocabulary? One round trip through GPT-2's tokenizer

GPT-2’s vocabulary contains strings such as Ġthe and ä½. They are neither corrupted original text nor special tokens. This implementation replaces each byte with a reversible character alias, then runs BPE on alias strings.

Use "Hi 你!" throughout:

Regex chunk        UTF-8 bytes              Byte aliases
'Hi'               [72,105]                 'Hi'
' 你'              [32,228,189,160]          'Ġä½ł'
'!'                [33]                     '!'

Final model IDs: [17250, 220, 19526, 254, 0]

The character 你 uses two IDs, with another for its preceding space. Characters do not have a fixed one-to-one relationship with tokens.

Initialization: two resources, several distinct tables

The resources loaded by get_encoder() are trained tokenizer artifacts, separate from Transformer weights:

The rule loader uses split('\n')[1:-1] to skip the version header and trailing empty line, then enumerates the remaining rules. It depends on the official file ending with a newline: applying that slice to a custom file without the newline would also discard its last rule.

After loading:

The first two rows are different mappings: byte to alias and full symbol to ID. Byte 97 has alias a, but encoder['a'] is 64. Byte 33 has alias !, whose model ID is 0.

Rank and ID differ by 256 here, but serve different purposes

The first few official rules:

Ranks 3–5 are merely omitted from this small table. Checking all 50,000 local official rules confirms:

encoder[left + right] == 256 + rank

A rank answers which rule comes first; an ID identifies which model token this is. Base entries have no merge rank, and special IDs do not follow from this formula. The numerical relationship does not make the concepts interchangeable.

bytes_to_unicode: a misleading name

This function does not decode raw bytes as UTF-8. It builds a 256-entry bijection: each byte value becomes exactly one Unicode character. The inverse restores bytes character by character.

Three ranges retain directly displayable characters:

bs = list(range(ord("!"), ord("~") + 1))        # 33–126: 94
bs += list(range(ord("¡"), ord("¬") + 1))       # 161–172: 12
bs += list(range(ord("®"), ord("ÿ") + 1))       # 174–255: 82
cs = bs[:]

Thus 94 + 12 + 82 = 188 bytes retain numerically identical code points. The remaining 68 — 0–32, 127–160, and 173 — map in ascending byte order to code points 256 + n. Finally, the function applies chr(c) and zips byte values with characters.

Whitespace and control bytes therefore need not appear literally in vocabulary strings, and input spaces cannot collide with the BPE return format’s separators. The no-argument bytes_to_unicode() function is cached with lru_cache. That mapping cache is distinct from an Encoder’s per-chunk self.cache.

The alias Ġ differs from a literal Ġ in the input

Input space:    UTF-8 [32]        → alias Ġ
Input Ġ:        UTF-8 [196,160]   → two separately mapped bytes
Input 你:       UTF-8 [228,189,160] → ä½ł

In Ġthe, read Ġ as an alias for an original space. Do not treat the whole vocabulary as wrongly decoded original text. Unicode characters are being used as containers for bytes; their linguistic meaning is irrelevant to BPE.

Two BPE loops, two different jobs

bpe(token) receives one regex chunk’s alias string, not the whole document. A cache hit returns immediately. Otherwise, word = tuple(token) and get_pairs lists adjacent pairs. A single symbol has no pairs and returns directly; that early-return path does not write to the cache.

Inside the loops:

• The outer loop selects the one pair type with lowest rank among current adjacent pairs.
• The inner loop fixes that pair and scans the whole word left to right, merging all non-overlapping occurrences.
• Only after the scan does new_word replace word; pairs are recomputed for the next rule selection.

This condensed control flow follows openai/gpt-2, src/encoder.py (MIT license):

while True:                                    # outer: choose a rule
    pair = min(pairs, key=lambda p: bpe_ranks.get(p, float("inf")))
    if pair not in bpe_ranks:
        break
    first, second = pair
    new_word = []
    i = 0
    while i < len(word):                       # inner: scan the chunk
        try:
            j = word.index(first, i)
        except ValueError:
            new_word.extend(word[i:])
            break
        new_word.extend(word[i:j])
        i = j
        if i < len(word) - 1 and word[i + 1] == second:
            new_word.append(first + second)
            i += 2
        else:
            new_word.append(word[i])
            i += 1
    word = tuple(new_word)
    if len(word) == 1:
        break
    pairs = get_pairs(word)

The historical source uses a bare except here; the excerpt uses except ValueError to express only the “first not found” branch.

Why a new rule must wait until the scan finishes

With an artificial earliest rule a+a and input aaa:

word:          a | a | a
round output: aa | a

The first two old positions are consumed. The third a cannot reuse the second position. The new aa is written to new_word while scanning still reads the old word, so it participates only in the next round.

j = word.index(first, i) finds only first, not the complete pair. If the neighbor differs from second, the current symbol must be retained. word[i:j] excludes position j, so copying the intervening symbols does not consume first before matching.

Three kinds of finishing

Every effective outer round reduces the symbol count by at least one. Starting with N symbols allows at most N−1 effective merge rounds. One round can merge several occurrences, so the actual count may be smaller. This bounds rounds, not the function’s overall time complexity: every round still scans, constructs candidates, and looks up ranks.

Watch each step execute

The default artificial aaababcabxyz example has nine rules and uses seven rounds. These rules were not trained again on this input:

It ends as aaababc | ab | xyz. Rules b+c and c+a exist, yet never get selected: earlier rules consume their components. This is part 28’s “present in the table, unusable on this input” behavior inside the actual loops.

Switch to the official internationalization example. Of 50,000 rules, the input uses ranks 3, 5, 7, 9, 26, 39, 85, 97, 272, 503, 608, 1378, 22868, 45353: fourteen rounds. The result is international | ization, IDs [45609,1634]. It does not scan the first fourteen rows starting at rank zero.

Three other cases remain available:

• aaaaaaa shows overlapping candidates consuming only non-overlapping positions.
• Single-symbol a skips both the loop and cache write.
• Repeating a chunk reads the cache populated by an earlier real execution.

The animation contains traces captured from the source implementation. Teaching and official examples use different rule tables, explicitly identified in the interface.

BPE’s separator spaces are not input spaces

After merging:

word = " ".join(word)
self.cache[token] = word
return word

The return value is still a string. "Ġä½ł" finishes as symbols Ġ, ä½, ł, so bpe returns "Ġ ä½ ł". encode splits on ordinary spaces and looks up encoder, producing [220,19526,254].

The input space has already become Ġ. Ordinary separator spaces disappear when splitting the return string; they do not add model IDs.

for token in re.findall(self.pat, text):
    token = "".join(self.byte_encoder[b] for b in token.encode("utf-8"))
    bpe_tokens.extend(self.encoder[s] for s in self.bpe(token).split(" "))

Concatenate results for Hi, 你, and ! in chunk order to obtain the five IDs above.

Decode: join bytes first, decode once

text = "".join(self.decoder[token] for token in tokens)
raw = bytearray(self.byte_decoder[c] for c in text)
return raw.decode("utf-8", errors=self.errors)

Look up and concatenate full alias strings; restore each alias to a byte; finally decode the entire byte sequence as UTF-8. Decoding does not run BPE or replay merges backwards.

ID 19526 stores ä½, raw bytes [228,189]. ID 254 stores ł, raw byte [160]. Neither alone is a complete UTF-8 character; each becomes � under the default policy. Joined as [228,189,160], they decode to 你. Decoding each token into text and then concatenating those strings loses this information.

Distinguish errors:

• An ID absent from decoder fails lookup with KeyError.
• Valid IDs whose combined bytes are invalid UTF-8 follow errors, replacing invalid sequences or raising UnicodeDecodeError.

Historical encode does not recognize special markers automatically

encoder['<|endoftext|>'] is indeed 50256, but this encode has no allowed_special branch. Literal input "<|endoftext|>" becomes:

[27, 91, 437, 1659, 5239, 91, 29]    # seven ordinary tokens

The caller must explicitly insert 50256 for the special marker. Meanwhile, decode([50256]) retrieves its registered text. Registering a vocabulary entry, recognizing its spelling during encoding, and stopping generation on its ID are separate operations.

The next article uses minbpe’s integer symbols and special-token policies. The core remains: fixed-rank merges inside chunks, direct ID-to-content lookup, and UTF-8 decoding only after all bytes have been joined.