From 2 Classes to 27: Only Three Parts to Add

V1 did cat-versus-dog classification; V2 abruptly switches to predicting the next character. They look like two different things.

They are the same thing. Given the current character, what is the next one — and there are 27 possible answers (26 letters plus an end marker). So: given an input, decide which of 27 classes it belongs to.

The structure is identical; only the class count goes from 2 to 27. Forward propagation, backpropagation and gradient descent need not change a word.

That 2→2→2 two-output network from part 3 (cat / dog) was already the prototype; bigram just widens the output layer from 2 neurons to 27.

Only three things genuinely have to be learned — and all three exist purely because there are now more classes.

one-hot: turning a character into numbers

A network only eats numbers, and a is not a number.

Why not just number them (a=1, b=2, …, z=26): numbers carry an ordering, so the network would take z to be “25 times bigger” than a, and letters have no such relationship.

one-hot instead uses a 27-dimensional vector where only the corresponding position is 1 and everything else is 0:

a = [0, 1, 0, 0, …, 0]
b = [0, 0, 1, 0, …, 0]

(Position 0 is reserved for the start/end marker.) Every pair of characters is now equidistant — no false ordering.

The cost is dimensionality: a 50,000-word vocabulary means 50,000 dimensions, which is unacceptable in a real model. Embeddings solve that later; at the bigram stage 27 dimensions is entirely fine.

In the cat/dog task the inputs were already numbers (0.9, 0.8), so this step never came up. one-hot is how a discrete symbol gets through the door.

softmax: turning 27 outputs into probabilities

The last layer emits 27 numbers called logits, positive or negative and of any size. These are not probabilities — probabilities must be non-negative and sum to 1.

softmax(zᵢ) = e^zᵢ / Σⱼ e^zⱼ

The numerator exponentiates this one term; the denominator sums the exponentials of all of them.

Unlike tanh or sigmoid, which take one number in and give one out, softmax takes a set in and gives a set out — it cannot compute the denominator without seeing every term.

Working through actual numbers, with logits [2.0, 1.0, 0.1, −1.0]:

They sum to exactly 1.000. Note that −1.0 became 0.368 — a negative turned positive.

Two properties:

• Order is preserved. Whatever was largest still is
• Gaps are widened. In z, 2.0 is twice 1.0; in probability it is 2.7 times — the exponential pulls them apart

The question left hanging in part 4 gets its answer here: why eˣ cannot be an activation function on its own, yet is everywhere. Because it never appears alone — it is always held in check by a division, and softmax’s denominator is that check. The exp operator added in part 4 was groundwork for exactly this.

For two classes, sigmoid suffices (fix one probability p and the other is 1−p). With 27 classes they have to constrain each other, and that is softmax’s job.

Cross-entropy: why squared error gets replaced

Not “more advanced” — the same principle under different assumptions

Squared error has a theoretical basis too. The two are one principle producing different forms under different assumptions:

Where the formula comes from

Ask: what makes a set of parameters good? Answer: they make the data actually observed as likely as possible.

The probability of all the bigrams occurring together is a product, and the goal is to maximise it. Three purely technical transformations follow:

1. Take the logarithm. Multiplying a couple of hundred thousand numbers below 1 underflows to 0, and is awkward to differentiate. log(ab) = log a + log b turns multiplication into addition, and log is monotonic so the optimum does not move
2. Negate. Optimisers go downhill, so flip it over
3. Divide by the sample count, so the loss does not scale with the dataset

max ∏ Pᵢ  →  max Σ log Pᵢ  →  min ( −(1/N) Σ log Pᵢ )

At no point is there a step of the form “we decided to design it this way.”

The i in the two formulas is not the same i

A notational collision, and an easy one to misread:

• The i in softmax ranges over classes: 27 numbers belonging to one sample, summing to 1
• The i in the cross-entropy sum ranges over samples: N numbers from different samples, which do not sum to 1

Written out properly it is loss = −(1/N) Σᵢ log P(i, tᵢ), where i indexes the sample and tᵢ is that sample’s correct class.

The 27 numbers within each row sum to 1 (softmax acts along the row). The three numbers in the last column do not — they come from different samples and are under no constraint at all.

softmax operates along rows; cross-entropy averages across them.

It takes the correct entry, not the largest one

This is the easiest thing to get wrong. Current character e, correct answer m:

loss = −log(0.02) = 3.91.

Taking the largest would give −log(0.40) = 0.92 — a model that is confidently wrong goes unpunished, and never learns anything.

Precisely because it takes the correct entry, training has a direction: low probability on the correct class → high loss → large gradient → push the correct class up.

Where the other 26 terms went

The full formula sums over all classes, but the true label is one-hot:

loss = −Σⱼ yⱼ log Pⱼ = −(0·log P₀ + … + 1·log P₁₃ + …) = −log P₁₃

The other 26 are multiplied away by 0 — not a decision to ignore them. And since softmax guarantees the total is 1, pushing the correct class up automatically squeezes the rest down; no extra penalty is needed.

Real code indexes directly (probs[target]) rather than actually building a one-hot vector and multiplying — it skips 26 multiplications by zero. PyTorch’s F.cross_entropy also takes indices rather than one-hot vectors.

What “sums to 1 / pushed up / squeezed down / penalty” actually mean

Four words, made concrete. A simplified 4-class demonstration: starting logits [0, 1, −0.5, −0.7], correct answer is class 3, the gradient is just P − y, learning rate 1.0, five steps:

Summing to 1 is a structural constraint created by softmax’s denominator, and holds unconditionally. Look at the “Total” column: the weights change throughout and it stays 1.0000. This is not something training achieved — in Σ eᶻⁱ / Σ eᶻʲ, numerator and denominator are the same number.

Pushed up is what gradient descent does. The loss depends only on the correct class, so the sole way to lower it is to raise that probability — the second column climbs, 0.126 → 0.828.

Squeezed down happens on its own. Columns two and three always sum to 1, so when the correct class reaches 0.828 the rest must fall from 0.874 to 0.172. No code pushes them down; they are displaced by the requirement that the total be 1 — like squeezing a balloon, one end swells and the other must collapse.

before: [0.207, 0.564, 0.126, 0.103]
after:  [0.057, 0.077, 0.828, 0.038]
            ↓      ↓      ↑      ↓

Penalty means a term added to the loss specifically to discourage something. Without softmax’s constraint a model could output [0.9, 0.9, 0.9, 0.9] — the correct class is at 0.9 and the loss is low, but saying “very likely” about every class means nothing was learned. That would require adding a term by hand:

loss = -log(P_correct) + sum(P_wrong)      # that second term is the "extra penalty"

With softmax that term is unnecessary — the probabilities are zero-sum, so anything given to the correct class is automatically taken from the others.

Why squared error fails at classification

Look at the last two rows. Squared error has a ceiling — probabilities live in 0~1, so a single term’s error tops out at 1, and 0.98 is barely distinguishable from 1.00. The gradient approaches zero, so the more catastrophically wrong the model is, the less it can learn — exactly backwards.

Cross-entropy has no ceiling: the more confidently it errs, the more sharply the penalty climbs and the larger the gradient, forcing a correction.

The gradient is cleaner too. Simplified, softmax plus cross-entropy gives:

∂loss/∂zᵢ = Pᵢ − yᵢ

Predicted probability minus true label, in one line. The e and the log cancel each other — not a coincidence, but because they were designed as a pair.

In passing: e and log

e ≈ 2.718281828…        e = lim (1 + 1/n)ⁿ

It comes from compound interest: one unit of money at 100% annual interest earns more the more often it compounds, but does not grow without bound — it converges to e.

Why it is everywhere in mathematics: eˣ is the only function whose derivative is itself. Use 2ˣ and the derivative picks up an awkward constant.

Three things are readable straight off the plot:

Where log(a×b) = log a + log b comes from: let a = eᵐ and b = eⁿ, then ab = e^(m+n), whose log is m+n. Multiplying powers means adding exponents.

Perplexity: the loss, in a number people can feel

perplexity = e^loss

It means: how many options the model is effectively guessing among.

This is the quantified form of “sees further → is more certain.” That number will keep coming down from here on.

Putting it together

a ──one-hot──→ [0,1,0,…] (27-dim) ──×W──→ 27 logits
  ──softmax──→ 27 probabilities (sum 1) ──cross-entropy──→ loss

It is a single-layer 27→27 network, structurally the same as MLP(27, [27]), except:

Everything else is identical — forward, backward, gradient descent, unchanged.

The whole part in one line: cat-versus-dog → (2 classes become 27) → bigram → (context of 1 becomes thousands) → GPT. Along that line, not one step is a leap.

The next part gets to the main feature: download 32,033 real names and build a complete language model without a neural network at all, purely by counting.