GPT-2 TOKENIZER

BPE merge process

Merge adjacent symbols within one chunk using fixed rule ranks.

V8 · 1.3
Input byte aliasesaaababcabxyzTeaching example

  1. 1Select the earliest rule
  2. 2Scan and merge
  3. 3Update and repeat
Current adjacent pairsLower ranks merge first
Current operationRound outputi: scan start · j: found position
01 / 73
One pair type per round; new symbols participate next round.← → Step · Space Play / Pause

Traces captured from encoder.py. Teaching cases use artificial rules; the official case uses the published GPT-2 table.