From 27 Dimensions to 2: Compression Is What Creates Structure

The previous part’s bigram bottomed out at a loss of 2.4544 for one reason only: it looks at a single preceding character.

Widening the context to three characters lets the loss fall further. But the moment that step is taken, counting collapses on its own — and the fix answers a bigger question along the way: on what grounds does a model have any opinion about combinations it has never seen?

Widening the context breaks counting first

Counting owns exactly one asset: a table. However many context combinations exist, that is how many rows the table has, with 27 columns each.

Note that the denominator has not moved: widening the context produces no new samples. It is still those 32,033 names and 228,146 positions; only what each position looks at got wider. Half a million cells sharing 228,146 samples means fewer than half a sample per cell on average.

And the problem is not that it is too slow to compute — it is that the table cannot be filled. A combination absent from the table has probability 0, so the model calls it impossible. The previous part’s model smoothing (add k to every count) props those zeros up, but what it props them up with is a flat, invented probability — it carries no information about whether this combination resembles any other.

Bengio’s 2003 answer: let similar characters share

A Neural Probabilistic Language Model (Bengio et al., 2003). The paper’s example: the training data contains the cat is walking but never a dog was running. Yet as long as the model has learned cat ≈ dog, the ≈ a and walking ≈ running, it can still assign a sensible probability to the sentence it has never seen.

Counting cannot do this, because every cell in the table is independent: an climbing to 5,438 occurrences has no effect whatsoever on any other cell.

For them to influence each other, “similar” first has to exist numerically. So the network gains a layer:

3 characters (indices) ──embedding──→ 3 × 2 = 6 dims (concatenated)
                       ──hidden layer + tanh──→ 100 ──output layer──→ 27

Compared with the previous part’s single-layer network: an embedding layer and a hidden layer are new, and everything else (softmax, cross-entropy, gradient descent) is untouched. This part covers only the front of that chain — from indices to six numbers.

Building the dataset first: one sliding window

block_size = 3
X, Y = [], []
for w in words:
    context = [0] * block_size          # [0,0,0] reads as '...'
    for ch in w + '.':
        ix = stoi[ch]
        X.append(context)
        Y.append(ix)
        context = context[1:] + [ix]    # drop the leftmost, append on the right

emma yields five samples:

Three details:

[0] * 3 is list repetition, giving [0,0,0]. 0 is the index of . — the start of a name has no preceding context, so a placeholder fills it. Merging the start and end markers into a single . in the previous part pays off a second time here: the placeholder needs no index of its own.

The loop terminates on w + '.': 'emma' becomes 'emma.', the five characters are consumed and it stops — no break needed. That final . is the label of the fifth sample, not an input; it teaches the model when to stop.

Shape change: X goes from (228146,) to (228146, 3), and Y is untouched. The sample count is identical to the previous part; only the width of each input changed.

With block_size = 1 this code degenerates exactly into the previous part’s bigram. Context length is a hyperparameter — by the time we reach the Transformer it will be hundreds or thousands, with the code in the same shape.

Of the half million possible contexts, only a small fraction ever occur; and among those that do, many occur only once or twice — one or two counts estimate nothing reliable. The MLP intends to cover all 531,441 of them with a few thousand parameters. On what grounds is the second half of this part.

C[X]: 228,146 lookups in one line

C = torch.randn((27, 2))     # 27 characters, 2 dimensions each
emb = C[X]                   # (228146, 3) → (228146, 3, 2)

Two lines. The first is the “character → vector” lookup table; the second resolves 228,146 × 3 character indices in one go.

The previous part’s closing line — “one-hot times a matrix is a lookup, and that is the motivation for embeddings” — is cashed in here:

torch.allclose(F.one_hot(torch.tensor(5), 27).float() @ C, C[5])   # True

The derivation was done there; only the conclusion is used here.

What the previous part’s version would look like

oh = F.one_hot(X, num_classes=27).float()   # (228146, 3, 27)
emb = oh @ C

(228146, 3, 27) is 18 million numbers, of which 26/27 ≈ 96% are zero. Matrix multiplication dutifully multiplies through every one of them.

At 27 classes this is merely wasteful. GPT-2’s vocabulary is 50,257 — one-hot at that scale cannot even be stored. So the lookup is not an optimisation trick; it is the only workable approach.

The shape rule for indexing

C[X] produces the first shape in this part worth stopping over. There is exactly one rule:

result shape = shape of the index + whatever dimensions the indexed object has left

Every integer is replaced by a length-2 vector, so one dimension is appended after the index’s shape.

C[5] and C[5:6] hold identical values with a different number of dimensions, and that pair is the one that bites: integer indexing drops a dimension (the notion of “row” disappears), slicing does not (it is still “some rows”, just one of them). The previous part’s IndexError from sum(1) on a 1-D tensor is a consequence of exactly this.

What the three subscripts of emb mean:

Read as one sentence: 228,146 samples, each of which is a 3 × 2 matrix.

ndim is not linear algebra’s “dimension”

There is a genuine collision here, made worse by both usages being current:

Linear algebra’s “dimension” counts components; PyTorch’s ndim counts levels of nesting. For one and the same (a, b): linear algebra calls it a 2-dimensional vector, PyTorch calls it a 1-dimensional tensor.

m × n corresponds to the shape, not to ndim — ndim stays 2 and it is the two numbers inside the shape that vary.

The quickest way to read ndim: print it and count the outermost brackets.

tensor(3.14)       0 levels → ndim 0
tensor([1, 2])     1 level  → ndim 1
tensor([[1, 2]])   2 levels → ndim 2

From this episode onward, print(x.shape) is worth making reflexive. By the time the Transformer is handling four- and five-dimensional tensors, it is the only way to keep the logic straight.

Why 27 dimensions fit into 2

Everything so far was mechanical reshaping. The real question is here: 27 characters, and two numbers suffice — why?

Information is genuinely lost. But what gets lost is the assumption that characters bear no relation to one another — and that assumption was wrong to begin with.

Under one-hot, a relates to e exactly as it relates to q

Any two one-hot vectors have a dot product of 0 — all of them mutually orthogonal — and every distance is √2.

In that representation, a and e (both vowels, followed by highly similar characters) are related exactly as a and q are (q is followed by almost nothing but u).

The consequence is concrete: in the previous part’s heat map an occurs 5,438 times and is among the best-learned combinations, while much rarer combinations have entirely independent parameters under one-hot — not a word of what those 5,438 occurrences taught can be carried across.

A plane cannot hold 27 mutually equidistant points

This is where compression starts working, and it is a geometric necessity rather than something tuned into existence:

In d dimensions, at most d + 1 points can be mutually equidistant. On a plane that is 3 of them, an equilateral triangle; a fourth cannot manage it — placed at the triangle’s centre it is indeed equidistant from all three vertices, but that distance is not the side length.

For 27 points to be mutually equidistant you need at least 26 dimensions. One-hot is precisely those 26 dimensions.

Turned around: give the model only 2 dimensions and something must end up next to something. It has no choice but to decide what crowds with what.

Compression is not optional

The mechanism is a single sentence: similar input vectors, passed through the same network, produce similar outputs.

a ≈ e  ⟹  a·W₁ ≈ e·W₁  ⟹  their predictions are similar too

So what the model learns from an transfers partly and automatically to other combinations. That is generalisation.

And “what sits next to what” is not designed — during training the gradient flows all the way back into C, pushing characters that behave alike towards each other.

If the embedding were also 27-dimensional, the model could degenerate straight back into one-hot (one axis per character) and never learn any structure at all. It is precisely because there are only 2 dimensions — not enough room to go around — that the model is forced into a trade-off. And the trade-off is the structure.

The idea is called representation learning: rather than designing features, force the dimensionality down and let the model discover the data’s own structure.

The parameters saved are only a by-product:

Fewer parameters = less prone to overfitting = better generalisation. But that is a bonus, not the reason embeddings exist.

What those two numbers are in real models

The dimension keeps climbing, yet stays far below the vocabulary — compression was never cancelled, only the ratio changed.

Embeddings trained at scale have a famous property:

king − man + woman ≈ queen

Semantic relations became vector arithmetic. All of it traces back to that single mechanism: not enough room, so things have to crowd.

Measuring resemblance: a dot product is not a cosine

Now that characters have near and far, a ruler is needed.

cos θ = (a · b) / (|a| · |b|)

def sim(a, b):
    va, vb = C[stoi[a]], C[stoi[b]]
    return (va @ vb / (va.norm() * vb.norm())).item()

What separates a dot product from a cosine is exactly that denominator:

a · b = |a| · |b| × cos θ
        └─ two lengths ─┘

Dot product = product of norms × cosine. The two coincide only when both vectors have unit length.

Check it with (1,1) against three vectors pointing the same way (every angle is 0°):

The direction never moved, and the dot product climbed to 20. By analogy: dot product ↔ how far you went, cosine ↔ which way you faced. “One kilometre northeast” and “ten kilometres northeast” point identically and differ tenfold in distance.

Why the length has to be divided out

During training, frequent characters are updated more often and their vectors tend to grow.

• With the dot product: a frequent character has a long vector and scores high against everything — it appears “similar to every other character”
• With the cosine: length is divided out and every character compares directions from the same starting line

Once normalised, the dot product is the cosine

(a · b) / (|a| · |b|)  =  (a/|a|) · (b/|b|)  =  â · b̂

So large-scale retrieval normalises every vector up front, after which a single matrix multiplication yields every cosine:

Cn = C / C.norm(dim=1, keepdim=True)
sim_matrix = Cn @ Cn.T

This formula turns up daily from here on. RAG retrieval is exactly this: encode both the documents and the question as vectors, then use the cosine to find the closest few. The same expression, scaled from 27 characters to tens of thousands of papers — all that changes is how many rows C has and how wide each one is.

One sentence for this part: squeeze 27 mutually perpendicular characters onto a single plane and similar ones finally get a chance to sit near each other — structure is not designed, it is forced out by having too few dimensions.

Each sample now holds three 2-dimensional vectors, six numbers in all. Concatenating them, pushing them through a hidden layer and out to 27 logits is the next part’s business.

The next part runs into two cases of “rewrite it and it breaks”: there are two ways to concatenate three vectors into six numbers, and two ways to compute the loss — and each pair gives identical numbers at wildly different cost.