If a larger vocabulary shortens a sequence, why can output computation still increase? Follow parameter counts and ID mappings into continuous and discrete visual representations, walk through eight steps of Token Healing, then examine representation differences through Chinese, numbers, whitespace and JSON/YAML examples. Includes bilingual diagrams and an interactive walkthrough.
Make exactly 6 with unlimited coins of value 1, 3 and 4. Derive the recurrence from the meaning of dp[x]: why it reads x−c, why worse paths can be discarded, and why six rounds of computation lead to a backtrace of just 6→3→0. Then apply the same reasoning to Unigram's Viterbi encoding.
Unigram updates piece probabilities using expected counts over all valid segmentations, then prunes a candidate vocabulary by removal cost. Derive p, w, Z, q, and C on one corpus, use the resulting model for abcabc's final Viterbi step, and verify SentencePiece byte fallback.
cl100k_base supplies bytes-to-rank/ID entries; minbpe reconstructs their left and right components. Follow hello to see why recovery stops strictly below the target rank, then separate merges, expanded vocab, shuffled base IDs, and raw bytes. Special-token policies select dedicated IDs or ordinary BPE, with complete round-trip checks.
GPT-2's encoder.py maps 256 bytes to reversible character aliases before performing string BPE. Follow Hi 你! through resources, ranks and IDs, nested loops, caching, and stop conditions, then join bytes across tokens before UTF-8 decoding. Includes interactive traces of fourteen official merges and seven teaching-example merges.
Unconstrained BPE learns combinations such as e plus a space or punctuation plus whitespace. GPT-2 first splits text with a regex: chunks share one learned merge table, but merges cannot cross their boundaries. Following four indentation spaces into a three-plus-one split separates regex decisions from learned vocabulary decisions.
A model sees nothing but IDs, and the tokenizer translates between text and IDs. Code points make too big a vocabulary and bytes too long a sequence, so BPE starts from 256 bytes and keeps merging the pair that most often sits together into a new token — and once trained, all that remains is a merge table in order. Encoding takes, at each step, the pair learned earliest, which is why ' th' sits in the table yet goes unused when cutting 'the thin': the order decides the result.
Part 23 said a tokenizer turns text into integers, but not that the first step turns text into bytes. A charset only says which number a character is; an encoding only says which bytes that number becomes — and once they are separated, UTF-8's four templates can be worked out by hand, half a Hanzi has a definite reason for showing as �, and the 256 entries at the bottom of a tokenizer's vocabulary stop being a convention and become a mechanical fact.
Starting over on a new dataset: Shakespeare, a 65-character vocabulary, and a bigram baseline at val 2.49 — the control group every later addition gets measured against. These sections hold no new mathematics, but they introduce two things that run through the whole episode: a token is something you define yourself (which is also the real reason for picking Qwen later), and the three shapes (B, T, C). Plus one equivalence worth seeing once: a lookup is a fully connected layer with no bias.