What a Neural Network Actually Is: A Function with 13,002 Parameters
Starting from handwritten digit recognition: what neurons, layers, weights, biases and activation functions each are, and why the whole network is only a function.
INDEX36
Notes
3Blue1Brown · the theory03
01
02
Gradients and Backpropagation: Three Rounds with One Cat
A 2→2→1 network, and where every single number in three rounds of training comes from. Plus two things you only see once you actually run it.
03
Multi-Path Summation: The Σ a Single-Output Network Hides
Going from 1 output neuron to 2 adds one step to backpropagation. That step is not a new rule — the multivariable chain rule had it all along.
Karpathy V1 · micrograd03
04
The Autograd Engine: Rules Belong to the Nodes, Order to the Topological Sort
Automatic differentiation in a hundred lines of Python: a Value class builds the graph, _backward closures hold the local rules, a topological sort does the global scheduling. And why += is the Σ from multi-path summation.
05
From Engine to Network: Three Classes, 41 Parameters
Wrapping Value into Neuron / Layer / MLP. Where the parameter count comes from, how sz's sliding window lines the dimensions up, and why a three-line loop is the entire forward pass.
06
The Training Loop: Loss Wires All 41 Gradients Together
Connecting the parts from the last two: a loss function gathers the outputs into one number, and a three-part loop actually drives it down. Plus what happens when you deliberately skip zeroing the gradients — across ten random seeds.
Karpathy V2 · makemore02
07
From 2 Classes to 27: Only Three Parts to Add
Cat-versus-dog and predicting the next character are the same problem, differing only in that the class count goes from 2 to 27. The three parts that difference requires: one-hot, softmax, cross-entropy — and why cross-entropy is not a fancier squared error.
08
Two Roads to the Same bigram: Counting and Gradient Descent
First build a complete language model with no neural network at all, purely by counting (loss 2.4544), then rebuild it with one-hot, matrix multiplication and softmax (2.46). The two roads meet — which answers what a neural network is actually learning.
Karpathy V3 · MLP05
09
From 27 Dimensions to 2: Compression Is What Creates Structure
One-hot makes a as related to e as it is to q; structure only appears once 27 dimensions are squeezed into 2. The table grows 729 → 531,441 while the first layer shrinks 8,100 parameters → 600 — and a dot product is not a cosine.
10
3,481 Parameters: Same Numbers, Not Interchangeable
The hidden layer is chapter 1's weighted sum, computed for 228,146 samples at once. A tensor is one strip of memory plus a manual: view costs nothing, a transposed tensor cannot be viewed — and hand-written loss matches F.cross_entropy until it returns nan.
11
820× Faster: A Noisy Gradient and a Curve That Only Rules Out
A gradient from 32 samples is wrong, yet 820× faster than the full batch and lands a lower loss. The learning rate comes from a sweep: even spacing in the exponent is even ratios in the rate — and why that U-curve can only rule out, never select.
12
2.1752: The Ruler Comes Before the Tuning
The model fits parameters on train; you fit hyperparameters on val — which is why test needs a third slice. Split by name, not by sample, or the validation set is contaminated on the spot. Tuned, val lands at 2.1752 — and on the embedding plot the vowels have gathered on their own.
13
Maths Catch-Up · The Derivative of a Copy Is 1: How Gradient Reaches the Embedding Table
A gradient always has the shape of whatever it differentiates, and two rules are enough to derive the backward formulas of a linear layer. The Jacobian of C[X] holds only 0s and 1s — and both properties of the embedding gradient, sparsity and accumulation, are written on that one table.
Karpathy V4 · activations & gradients04
14
Saturation from 62.3% to 1.4%: Four Lines of Initialisation, and an Order to Debug In
An initial loss of 17 and six in ten neurons dead on arrival — because the standard deviation is amplified once at each of two matrix products. The fix is four lines, but what actually travels is the checklist of what to check in what order, and why Kaiming loses to a guessed number on this particular network.
15
Maths Catch-Up · Variances Add, Standard Deviations Don't: Where the √n Comes From
The previous part borrowed σ_z = √n · σ_W · σ_x and moved on. This one derives it from the definition of expectation: three rules of arithmetic, why the cross term vanishes, why it is variance and not standard deviation that adds — and why diagnosing a network means watching the variance rather than the mean or the maximum.
16
A Thousandfold Range In, 1.000 Out: What BatchNorm Pays for It
Initialisation only governs step 0; BatchNorm governs every step — vary σ_W by a factor of a thousand and the output after BN is identical. The price is that it becomes the one operation in the network where samples affect each other, which is why inference needs its own statistics, and why model.eval() exists.
17
Divide the Learning Rate by 3: Four Plots That Turn Guessing into Arithmetic
The six-layer network runs, and all four diagnostic plots are in hand. The first three are snapshots of a single step, and all three are green; the one that yields an action item is the fourth — it turns “what should the learning rate be” from a range a sweep can only narrow into a factor you can compute.
Karpathy V5 · backprop by hand05
18
Maths Catch-Up · Summation Is What Reuse Costs: sum(0) or sum(1), Just Compare the Indices
Part 3 gave the Σ of the multivariable chain rule and part 13 used it to derive the linear layer — but that was one special case. This part turns it into a general method: the addition comes from the total differential, the four shapes differ only in which entries of one Jacobian are zero, and whether to sum or broadcast — and along which axis — reads straight off the two index sets.
19
One Gradient That Must Be Exactly Zero: Eight Steps of softmax Backprop, by Hand
The previous part finished deriving the test for tensor calculus; this one applies it eight times over, working back from the loss to the logits. What is worth taking away is not the eight expressions but a way of marking your own work, three operating rules, and one property — a gradient that must be exactly zero — that doubles as a debugging tool.
20
The Same Deviation Is Both Numerator and Denominator: Thirteen Steps Back Through BatchNorm
The previous part stopped at logits — the edge of the network, with not one parameter touched in eight steps. This one goes thirteen steps further in, and parameters appear for the first time. BatchNorm's nine steps are the worst stretch in the episode for the same reason softmax forks: a normalisation's numerator has to go build the denominator first. Along the way: the first sum-backward whose coefficient is not 1, and a mistake only 3% wide whose maxdiff looks like rounding error.
21
The Line That Looks Right: 96 Lookups Landing on 27 Rows, and the End of Ex1
Five steps left, no forks, and the first three are a copy of the previous part. What is genuinely new is the last two: why view's backward is simply its own inverse (one test decides it — did the count of numbers change), and why dC[Xb] += demb, a line that looks entirely correct, is wrong. Finish these five and all 4137 parameter gradients have been worked out by hand.
22
Off by 0.0001: Twenty-Six Steps Compressed into Six, and Hand-Derived Gradients Training the Same Model
The previous three parts derived all twenty-six steps from the loss to every parameter, but that was opening the black box and walking through it. This one puts the box back together: cross-entropy's eight steps compressed into one line, BatchNorm's ten into one, both by the same method — count how many times a variable appears in the forward pass and the backward has that many terms. Then backward() is commented out and the model trains for 200,000 steps: val loss 2.1105, against PyTorch's 2.1106.
Karpathy V7 · building GPT04
23
A Starting Loss Twice as Bad as Guessing: From Shakespeare to the 2.49 Baseline
Starting over on a new dataset: Shakespeare, a 65-character vocabulary, and a bigram baseline at val 2.49 — the control group every later addition gets measured against. These sections hold no new mathematics, but they introduce two things that run through the whole episode: a token is something you define yourself (which is also the real reason for picking Qwen later), and the three shapes (B, T, C). Plus one equivalence worth seeing once: a lookup is a fully connected layer with no bias.
24
One Line Changed: Where Every Symbol in the Attention Formula Comes From
softmax(QKᵀ/√dₖ)V — this part earns each of its symbols one at a time. Starting from the crudest option, plain averaging, it goes through “a lower-triangular matmul is a weighted sum” and “softmax with −∞ masks the future” to “scores computed as q·k”: from averaging to attention, the code changes by one line. Encoder and decoder differ by one line too. And the final √dₖ is part 15's variance formula used in a new place.
25
How the r After “thou” Disappeared: Growing One Attention Head into a GPT
Part 24's attention head goes into the model, then the model gets deeper and bigger, and val falls from 2.49 all the way to 1.48. Multi-head attention adds not a single parameter — the gain comes from division of labour; the FFN supplies the only non-linearity; residuals make a deep network start out as a shallow one, LayerNorm normalises along the other direction, and Dropout drops from the branches, never the trunk. One question asked four times — what follows “thou”? — tracks it all: an r that never once follows “thou” in the text falls from first place to 0.001.
26
The Differences Are All Engineering: A From-Scratch GPT Beside the Paper and nanoGPT
Set beside the 2017 paper, the model written in part 25 is the decoder half with cross-attention removed; set beside nanoGPT, its structure is practically identical and every difference is engineering. The part most worth chewing on is the three lines that merge the heads: c_attn lays the small matrices side by side, and view + transpose moves “head” forward — worked through by hand on a two-head example. Along the way: a correction (a bias on q is not wasted), the 6N compute estimate, and how pre-training, SFT and RLHF turn a model that can only continue text into ChatGPT.
Karpathy V8 · the tokenizer09
27
Basics Catch-Up · One Character, Five Byte Strings: Charsets Number, Encodings Store
Part 23 said a tokenizer turns text into integers, but not that the first step turns text into bytes. A charset only says which number a character is; an encoding only says which bytes that number becomes — and once they are separated, UTF-8's four templates can be worked out by hand, half a Hanzi has a definite reason for showing as �, and the 256 entries at the bottom of a tokenizer's vocabulary stop being a convention and become a mechanical fact.
28
The Merge Table Is the Tokenizer: Why Tokenize, and BPE from Scratch
A model sees nothing but IDs, and the tokenizer translates between text and IDs. Code points make too big a vocabulary and bytes too long a sequence, so BPE starts from 256 bytes and keeps merging the pair that most often sits together into a new token — and once trained, all that remains is a merge table in order. Encoding takes, at each step, the pair learned earliest, which is why ' th' sits in the table yet goes unused when cutting 'the thin': the order decides the result.
29
Set the boundaries first: the regex before BPE
Unconstrained BPE learns combinations such as e plus a space or punctuation plus whitespace. GPT-2 first splits text with a regex: chunks share one learned merge table, but merges cannot cross their boundaries. Following four indentation spaces into a three-plus-one split separates regex decisions from learned vocabulary decisions.
30
What is Ġ in the vocabulary? One round trip through GPT-2's tokenizer
GPT-2's encoder.py maps 256 bytes to reversible character aliases before performing string BPE. Follow Hi 你! through resources, ranks and IDs, nested loops, caching, and stop conditions, then join bytes across tokens before UTF-8 decoding. Includes interactive traces of fourteen official merges and seven teaching-example merges.
31
The IDs already exist: recovering minbpe rules and handling special tokens
cl100k_base supplies bytes-to-rank/ID entries; minbpe reconstructs their left and right components. Follow hello to see why recovery stops strictly below the target rank, then separate merges, expanded vocab, shuffled base IDs, and raw bytes. Special-token policies select dedicated IDs or ordinary BPE, with complete round-trip checks.
32
The same abc, a different split: training and encoding with Unigram
Unigram updates piece probabilities using expected counts over all valid segmentations, then prunes a candidate vocabulary by removal cost. Derive p, w, Z, q, and C on one corpus, use the resulting model for abcabc's final Viterbi step, and verify SentencePiece byte fallback.
33
Algorithms Catch-Up · The Index Is an Amount, the Value Is a Count: Dynamic Programming and Backtracking
Make exactly 6 with unlimited coins of value 1, 3 and 4. Derive the recurrence from the meaning of dp[x]: why it reads x−c, why worse paths can be discarded, and why six rounds of computation lead to a backtrace of just 6→3→0. Then apply the same reasoning to Unigram's Viterbi encoding.
34
A few input rows, a full vocabulary of outputs: from token IDs to sampling
What happens after a tokenizer produces IDs? Follow one five-token model through two embedding lookups, a Transformer Block, every lm_head score, and the training loss. Then work through random draws, temperature, top-k and top-p, with three diagrams, two interactive demos and complete reproducible code.
35
A Vocabulary Does More Than Split Text: Model Costs, Multimodality and Token Boundaries
If a larger vocabulary shortens a sequence, why can output computation still increase? Follow parameter counts and ID mappings into continuous and discrete visual representations, walk through eight steps of Token Healing, then examine representation differences through Chinese, numbers, whitespace and JSON/YAML examples. Includes bilingual diagrams and an interactive walkthrough.
GPT · training and inference01
36
One GPT, Two Complete Paths: Training Updates and Token-by-Token Generation
Follow Hi 你! through position embeddings, LN and QKV, two residual additions, the MLP, vocabulary logits, cross-entropy and AdamW, then trace fixed-parameter generation and incremental decoding. Three diagrams and two interactive walkthroughs distinguish parameters, activations, gradients and context.