A Vocabulary Does More Than Split Text: Model Costs, Multimodality and Token Boundaries

A different tokenizer may represent the same text with fewer tokens. Three tempting conclusions can follow: a larger vocabulary must save computation; a more compact tokenizer can simply replace the old one; and a model that can read images can also generate them.

Each conclusion skips an interface. The vocabulary maps bytes to IDs; a trained model already associates those IDs with parameter rows; the Transformer processes a sequence of vector positions. Changing the vocabulary, the number of positions or the representation affects different parts of the system.

Keep three quantities separate: V is vocabulary size, T is the input sequence length and D is the vector width at each position. The text-model pipeline is:

Text → token IDs → look up a V×D table → T×D input
     → Transformer → context vectors → score V candidates → choose a next ID

From Token IDs to the Next Token works through that pipeline numerically. Its full interface diagram is retained below before examining vocabulary changes and boundaries.

Double the vocabulary: count parameters, then computation

How many parameters are in a V×D table?

Each token-embedding row corresponds to an ID and contains D learned numbers. The parameter count is therefore VD.

38,400,000 / 19,200,000 = 2: B is 200% of A, a 100% increase. The first fraction uses B as its numerator; the increase uses B−A.

GPT-2 124M uses 50,257×768, or 38,597,376 parameters in its token-embedding table alone. This excludes the position table and Transformer blocks.

With tied input and output weights, that V×D table counts once as independent parameters. Without tying, the two ends cost roughly 2VD, plus V if the output layer has a bias. nanoGPT’s model.py binds wte.weight and lm_head.weight, giving the tied case.

Tying saves a parameter table, but output computation remains: the input looks up selected rows, while the output computes a dot product with every candidate’s weights. Adding ΔV tokens adds roughly ΔVD or 2ΔVD parameters to these parts. It does not automatically enlarge the position table.

How can a shorter T make one part more expensive?

Suppose the same text shrinks from 1,000 to 700 tokens while V doubles and D stays fixed. This is an explicit teaching assumption, not a benchmark of two real models.

For ordinary dense attention, consider its two pairwise matrix products:

Q: T×D, Kᵀ: D×T → QKᵀ: T×T
Each pair of positions requires a D-dimensional dot product: order T²D.

Attention weights: T×T, Values: T×D → output: T×D
Again, order T²D.

The output head, applied at every position, instead computes:

H: T×D, W_headᵀ: D×V → logits: T×V
There are T×V dot products of length D: order TVD.

The three ratios are different:

These ratios describe the corresponding operations during training or prefill. They do not combine attention projections, MLPs, layer counts, batch size, hardware or implementation, so they do not predict total latency. Some generation implementations compute only the last position’s output logits during prefill; the all-position TVD count does not apply unchanged there either.

KV-cached single-token decoding has another cost structure. A new query attends to cached historical keys without rebuilding all historical rows of a T×T matrix; the output head processes one new position, with multiplication cost of order VD.

A larger vocabulary often covers longer fragments, but the training corpus, algorithm and rules also determine the split. It cannot guarantee a shorter sequence for every input. More compact tokenization usually fits more text into a fixed token limit, which by itself does not establish better understanding.

Adding a token changes more than the tokenizer

A trained model’s IDs already have meanings. Suppose tokenizer A maps apple to 100, while B maps banana to 100. Swapping only A for B makes banana retrieve the old E[100], trained with apple; an output of 100 also decodes as banana. Equal vocabulary sizes do not imply compatible IDs.

When vocabulary expansion is actually needed, connect the whole process:

1. Register the new entries while preserving existing ID meanings.
2. Resize the input embedding and required output layers, including tying and biases.
3. Initialize the new parameter rows.
4. Continue training on data that uses the entries, checking that the relevant parameters can update.
5. Evaluate input use, output use and existing tasks, then save the tokenizer, weights and configuration together.

Transformers’ resizing interface takes the new total vocabulary size. This sketch covers the interface, not the subsequent training: tokenizer documentation.

tokenizer.add_tokens(["<new_domain_term>"])
model.resize_token_embeddings(len(tokenizer))
# Training, validation and saving the paired artifacts still follow.

Initialization might use random values or existing token-piece vectors, depending on the implementation. It provides a starting row, not a learned meaning. Fine-tuning schemes that freeze most parameters must also check whether the new input rows and necessary output parameters can train.

Many fine-tuning tasks simply retain the original tokenizer. Conversely, preserving old IDs does not guarantee identical tokenization of old text: an added entry may replace a sequence of previously separate pieces.

Consider the dimensions of a 20×4 table E. Initialize one new row with the mean of the old rows and append it to obtain E′:

Old table E:    20×4
New row:         1×4
Expanded E′:    21×4, with E as its first 20 rows

H: 1×7×4 → H @ E′ᵀ: 1×7×21

One added input row also creates one added tied output candidate. This describes parameter shapes; the new entry still needs to learn its role through training.

An image need not receive a text-vocabulary ID first

Text IDs become vectors through an embedding lookup. Images can join at the vector-sequence interface. Two routes need to be distinguished.

Continuous visual features add sequence positions

Image → vision encoder → visual features → project to LLM width
                                                   ↓
Text → token IDs → text embeddings ───────────→ joint sequence → LLM

Consider hypothetical dimensions with batch size B=1, text vocabulary V=20 and LLM width D=4:

T increases from 3 to 7 while the text vocabulary remains at 20 entries. Here a “visual token” denotes a visual vector position; it need not have an integer ID in the text vocabulary. Separate image-boundary control IDs are another question.

These shapes explain how vectors enter the model. Compatible dimensions only establish an interface; visual and language semantics still need trained alignment.

Discrete visual IDs select rows from a codebook

Another route has an encoder produce local latent vectors, then quantizes them with a finite codebook. Let the four rows be:

ID 0 → (0,0)    ID 1 → (1,0)
ID 2 → (0,1)    ID 3 → (1,1)

For x=(0.9,0.2), choose the smallest squared Euclidean distance:

To ID 0: (0.9−0)² + (0.2−0)² = 0.85
To ID 1: (0.9−1)² + (0.2−0)² = 0.05  ← minimum
To ID 2: (0.9−0)² + (0.2−1)² = 1.45
To ID 3: (0.9−1)² + (0.2−1)² = 0.65

This position selects ID 1; another position, (0.1,0.8), selects ID 2. This numerical example illustrates nearest-neighbor selection; a real encoder, codebook and decoder still require training.

How can finitely many codes represent complex images? An image has many positions. One codebook row is not one complete image: each position chooses a code. K codes at N positions allow at most Kᴺ arrangements. This combinatorial limit does not make every arrangement a valid image or make the representation lossless.

Quantization maps different vectors to the same representative and loses information. Continuous representations may also compress information; “continuous” does not mean inherently lossless.

Reading an image and generating one need different outputs

One discrete generation route predicts visual IDs autoregressively, looks up codebook vectors and decodes them into pixels. One continuous latent route starts from conditioning and noise, produces visual latents through diffusion or flow, then decodes them into pixels.

An image-reading language model therefore still needs a suitable output representation, generation modules and training to produce images. Its existing text lm_head does not acquire that role automatically.

Both representations can coexist in one system. Janus §3.1 separates understanding and generation encodings: continuous visual features for understanding, VQ IDs for generation, and a separate image prediction head. The Qwen3-VL report describes a vision encoder, vision–language merger and LLM. These are architecture examples, not estimates of each approach’s market share.

Audio can likewise use continuous features or discrete codec codes; video also organizes frames and features over time. This section locates the representation interface. Vision encoders, quantizer training and generation models need their own derivations.

One position need not mean one character, digit or space

These tokenization examples use the GPT-2 and cl100k_base configurations in tiktoken 0.14.0. ␠ displays an ordinary space; it is not part of the actual input.

Letters sit inside token pieces

cl100k_base:
strawberry  → str | aw | berry  → [496,675,15717]
␠strawberry → ␠strawberry       → [73700]

A token can hold several characters, and a leading space can change the split. Counting tokens cannot directly count letters inside a word. This does not mean a model can never learn spelling.

For character operations, explicitly expanding s t r a w b e r r y makes the units clearer. Counting three r’s here is a checkable character calculation, not a measurement of LLM success. When the answer must be exact, text.count("r") or a string-reversal program provides a direct route.

Groups of three digits need not align with the units place

cl100k_base’s pre-tokenization limits numeric chunks to 1–3 digits; BPE still chooses merges inside those chunks. Prepending 9 can shift the later boundaries. This is not grouping from the units place of a decimal number.

Single-digit tokens make numeric units more regular, at the cost of some longer sequences. They do not automatically teach carrying, multiplication or multi-step reasoning. A trained model also cannot have its numeric splitting rules replaced independently without adaptation.

Put four spaces back into their code context

Four isolated ordinary spaces take 4 GPT-2 tokens and 1 cl100k_base token. Now encode this complete string with no final newline:

def f():
    return 1

GPT-2 uses 9 tokens, cl100k_base 7. In the latter, the first three indentation spaces form one entry, while the fourth joins return. The isolated-space count cannot be applied mechanically to every code context.

Ordinary pieces also differ from special tokens. An unusual name does not make an entry a control token; registration and tokenizer/protocol policy determine that role. See rule recovery and special tokens for the full mechanism.

A text boundary need not be an ID-concatenation boundary

Encoding two parts separately and joining their IDs may differ from encoding the whole:

encode('Question: ')      = [14924,25,220]
encode('hello')           = [15339]
Concatenate those IDs    = [14924,25,220,15339]
encode('Question: hello') = [14924,25,24748]

Both sequences decode to Question: hello. The tail differs: ␠ | hello versus one ␠hello entry. Whole-text encoding can consider merges across the join; ordinary autoregressive generation appends IDs without automatically withdrawing and merging the final space.

Similarly, encode('hel')=[50222] and encode('lo')=[385] decode together as hello, but encode('hello')=[15339]. A “partial token” means that the text ends with a byte prefix of a longer vocabulary entry. ID 50222 itself is valid and complete.

If training first encodes full text and then cuts windows in the ID sequence, those windows do not split a single hello ID into hel and lo. Training may still contain truncated text or deliberately constructed completion data, and models may generalize. This is a possible boundary-distribution mismatch, not a claim that incomplete inputs cannot be processed.

Whether a user’s message is complete, whether a token boundary is stable and whether an application should offer completion are separate questions.

Token Healing: withdraw an ID while preserving every input byte

The demonstration uses one explicit variant: roll back the last token and impose a strict byte-prefix constraint on its replacement. The input Say hel encodes under cl100k_base as [46864,11591], or Say | ␠hel.

IDs and prefix matches come from the actual vocabulary. All logits are hand set; no language model is called.

Find who is allowed, then use scores to choose

Rolling back ID 11591 leaves model context [46864], corresponding to Say. Save the removed token’s raw bytes 20 68 65 6c, or ␠hel, as held.

The allowed set contains ordinary vocabulary entries whose bytes begin with held:

C = { id | token_bytes(id).startswith(held) }

There are 24 in this vocabulary, including the original ␠hel. Because the original remains eligible, this rule does not force a longer replacement.

Every other ordinary entry gets a hand-set score of −20. Vocabulary lookup decides eligibility, not the preferred continuation. The model forward pass would score the full vocabulary from the shortened context. These operations have different jobs.

Mask an ineligible winner

Let z be the original scores and z′ the masked scores:

id ∈ C: z′[id] = z[id]
id ∉ C: z′[id] = −∞

world=10 is highest but violates the prefix. Masking makes its softmax probability zero. The best allowed score is hello’s 6, so greedy selection returns ID 24748.

For probability sampling, normalize after masking. Subtracting hello’s logit for stability gives this denominator:

1 + exp(−2) + exp(−3) + exp(−5) + 20×exp(−26)

The resulting probabilities are about 83.90% for hello, 11.35% for help, 4.18% for helper and 0.57% for hel. The other 20 allowed entries retain tiny nonzero probabilities. Sampling can differ from greedy selection, but a strict mask confines both to C.

The internal ID changes; the display only gains lo

Rolled-back context: [46864]
Chosen ID:           24748 → bytes ' hello'
New internal history:[46864,24748]
Full decoding:       Say hello
Existing display:    Say hel
New display suffix:  lo

The replacement first covers the saved ␠hel prefix. Only the remaining lo goes to the display, without repeating hello or changing the user’s existing text. In general, an incremental UTF-8 decoder must buffer suffix bytes that do not yet form a complete character. Here lo can be decoded on its own.

Pseudocode makes the objects passed between steps explicit:

context = original ID sequence without its last entry
held = raw bytes of the final entry
C = IDs of entries whose bytes start with held

z = full-vocabulary scores from context
z′[id] = z[id] if id is in C; otherwise −∞
selected = argmax(z′)

replacement = bytes of the selected entry
new display bytes = replacement with the held prefix removed
new internal history = context + [selected]

The scores above are hypothetical values illustrating the selection rule. A real system obtains logits from a model forward pass over the rolled-back context. If it maintains a KV cache, the withdrawn position must also leave the effective cached history.

Why does another interface return 182 instead of 24?

In tiktoken 0.14.0, encode_with_unstable('Say hel') returns stable IDs [46864] and 182 possible completion sequences. The 24 count single ordinary entries covering held; the 182 sequences may contain multiple tokens. They count different objects, and neither is a model probability distribution.

Implementation and version matter. Transformers 4.57.6 generation/utils.py::heal_tokens first calls strip() on the text, then searches token prefixes and adds finite scores with sequence_bias. Finite bonuses do not force every other candidate’s probability to zero, so they are not the −∞ mask above.

One-token rollback also does not solve every boundary. More general constraints need state spanning multiple tokens. Token Healing is optional completion-boundary handling; it neither guarantees correct content nor guarantees that all subsequent IDs equal re-encoding the complete text.

An entry can exist without being reliably repeated by a model

The historical SolidGoldMagikarp report recorded unusual repetition behavior and other failures. The leading space matters when examining this entry’s tokenization:

The same space-prefixed string takes 5 cl100k_base entries: [22925,26509,34015,1609,8035]. These are ordinary entries, not special control tokens.

decode(specified_ID) restores bytes from a table. Asking a model to repeat a string requires it to choose the right ID sequence. Deterministic decoding does not guarantee correct model selection. Differences in vocabulary-training and model-training coverage are a relevant explanation, not an established sole cause. The historical report also records behavior changes, so those observations should not be generalized to other versions or current models.

No input occurrence does not imply a zero gradient

For shared row E[j], output z[j]=h·E[j] and target y, softmax cross-entropy gives:

L = −z[y] + log Σₖ exp(z[k])
∂L/∂z[j] = p[j] − 1[j=y]
∂L/∂E[j] (output path) = (p[j] − 1[j=y]) · h

Even when j≠y, the output path contributes p[j]·h. With tied weights, that gradient reaches the same embedding table.

Suppose the shared table has three two-dimensional rows, with input ID 0 and target ID 1:

E[0] = [ 0.2,  0.3]
E[1] = [ 0.4, −0.1]
E[2] = [−0.2,  0.5]

h = E[0], y = 1
z[j] = h · E[j]

Here h=[0.2,0.3], the logits are [0.13,0.05,0.11], and p[2]≈0.33761. Row 2 receives a gradient of about [0.06752,0.10128] despite appearing as neither input nor target.

This refutes “the gradient must be zero”; it does not establish that the word’s meaning is well learned. Investigate coverage, training distributions and controlled behavior. Deleting entries and renumbering IDs would break the parameter mapping.

Chinese efficiency: the same text can use different token counts

The following examples correspond to GPT-2, cl100k_base and the public Qwen/Qwen3-0.6B tokenizer configuration. Counts refer to the raw strings, excluding chat templates, extra special tokens and added newlines.

Qwen splits the first input into 你好 / ， / 世界 / ！, IDs [108386,3837,99489,6313].

GPT-2 encodes 你 as [19526,254], with raw bytes e4bd and a0. Together, e4bda0 forms the complete UTF-8 character. Replacement characters shown for isolated fragments do not mean the original bytes were lost. cl100k_base instead uses one ID here, [57668].

These four examples illustrate lengths under the specified configurations, not overall language ability. Vocabulary size, corpus and rules may all contribute. A shorter sequence also does not make the Qwen tokenizer a drop-in replacement for a trained GPT-2 model.

This version of the Qwen3-0.6B configuration uses BPE, NFC normalization and ByteLevel processing. Entire Qwen or Llama families should not be generically labeled “SentencePiece tokenizers.” NFC also means arbitrary raw Unicode input need not round-trip byte-for-byte. That normalization behavior is a separate issue from reassembling byte fragments.

Comparisons should identify specific versions and configurations rather than infer tokenization behavior from a broad model-family name.

JSON and YAML: fewer characters can mean more tokens

The same data object can have several serialized representations. Consider this compact JSON:

{"name":"Alice","age":30,"skills":["Python","SQL"],"active":true}

The block-style YAML is:

name: Alice
age: 30
skills:
  - Python
  - SQL
active: true

Counts exclude Markdown fences, chat templates and final newlines. YAML has the fewest characters; compact JSON has the fewest tokens. Token count depends on learned splitting and merging, not a strictly monotonic relationship with character count. Here each punctuation fragment ":" and "," is one token.

Measure complete inputs with the target tokenizer, while also considering parsing, field types and task results. Use a serializer’s compact option for JSON; do not remove spaces inside string values indiscriminately.

Choose according to what has already been trained

BPE and Unigram are algorithms; tiktoken, SentencePiece and Tokenizers are tools; GPT-2 encoding and cl100k_base are specific configurations. Changing tools while preserving encoding and decoding behavior differs from replacing the ID system.

The tokenizer now does more than split text before a model runs. It defines ID meanings, affects the number of positions and sets the boundary from which generation continues. Changes need to be checked along those correspondences.