GPT / FORWARD / GENERATE

GPT / TRAINING / INFERENCE From IDs to logits: the whole neural network D = 16 · two Blocks · two heads · batch omitted; only some features and candidates are drawn. Input: look up rows, then add matching features Token ID sequence [17250, 220, …] Position indices [0, 1, …, T−1] E[id] : T × 16 E: 50,257 × 16 P[position] : T × 16 P: 16 × 16 + Dropout Train 0.1 / eval off X₀ : T × 16 Into Block 1 BLOCK 1 · two attention heads + a per-position MLP First-layer parameters; inspect its calculations below Retain X Retain R LN1 Q / K / V : T × 16 Head 1 · 8D A_used × V Head 2 · 8D A_used × V Concat → W_O → Dropout + R LN2 MLP · applied at each position 16 64 16 Linear → GELU → Linear → Dropout + X₁ T × 16 → T × 16. Attention mixes positions; the MLP mixes features. BLOCK 2 · two attention heads + a per-position MLP Same structure, separate weights; only saved boundary values are available Retain X Retain R LN1 Q / K / V : T × 16 Head 1 · 8D A_used × V Head 2 · 8D A_used × V Concat → W_O → Dropout + R LN2 MLP · applied at each position 16 64 16 Linear → GELU → Linear → Dropout + X₂ T × 16 → T × 16. Attention mixes positions; the MLP mixes features. X₂ : T × 16 Final normalization and vocabulary projection LN_f H_final : T × 16 Train: all rows; generate: last row first 16 50,257 lm_head = Eᵀ Z = H_used Eᵀ Train T×50,257 / generate 1×50,257 Lookup and output scoring share E; every vocabulary entry is scored. Training · every supervised position Generation · the last position Targets y + vocabulary scores → mean CE L = mean(−log p[target]) Last row → temperature / top-k → softmax p_next : 50,257 Backpropagate; accumulate parameter gradients g = ∂L / ∂θ Sample one ID → append it to the context Display characters when UTF-8 bytes are complete AdamW: θ, m, v → θ′, m′, v′ Clear gradients; process another batch EOS / token limit: stop Otherwise: new IDs → lookup; weights stay fixed θ includes trainable E, P, both Blocks and LN_f. Grids: positions. MLP dots: features. Output dots: candidate IDs. Thin lines: linear connections. The two bypasses retain X and R. Line widths do not encode recorded weight magnitudes.