From Engine to Network: Three Classes, 41 Parameters

The previous part finished Value, an engine that differentiates any expression. But so far the code contains only expressions, not a network.

This part wraps it into Neuron → Layer → MLP. There is no new mathematics here; it is pure packaging.

The three class signatures

This is the easiest thing to get muddled, so here are the arguments and return values:

Thirty lines of code; everything in this part is about them:

import random

class Neuron:
    def __init__(self, nin):
        self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]
        self.b = Value(random.uniform(-1, 1))

    def __call__(self, x):
        act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
        return act.tanh()

    def parameters(self):
        return self.w + [self.b]


class Layer:
    def __init__(self, nin, nout):
        self.neurons = [Neuron(nin) for _ in range(nout)]

    def __call__(self, x):
        outs = [n(x) for n in self.neurons]
        return outs[0] if len(outs) == 1 else outs

    def parameters(self):
        return [p for neuron in self.neurons for p in neuron.parameters()]


class MLP:
    def __init__(self, nin, nouts):
        sz = [nin] + nouts
        self.layers = [Layer(sz[i], sz[i+1]) for i in range(len(nouts))]

    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]

Value is the class from the previous part, unchanged. There is not one _backward in these three classes — they merely call Value’s operators, and the gradient machinery was buried in there all along.

The second argument to Layer(3, 4) is an int; the one to MLP(3, [4]) is a list. MLP(3, 4) raises an error.

Where the parameter count comes from

The number of weights equals the number of inputs the neuron receives, because every input needs a weight of its own to be weighted by:

z = w₁x₁ + w₂x₂ + w₃x₃ + b

There is always exactly one bias; it pairs with no input. So each neuron has n+1 parameters.

Taking MLP(3, [4, 4, 1]):

41 in total.

Looking at the diagram, count the lines arriving at any node in layer 1 — three of them, so three weights, plus its own bias: four parameters per neuron. Four lines arrive at each node in layer 2. The number of lines is the number of weights.

Only one thing here is actually chosen by a person:

sz’s sliding window

sz = [nin] + nouts        # [3] + [4,4,1] = [3, 4, 4, 1]
self.layers = [Layer(sz[i], sz[i+1]) for i in range(len(nouts))]

sz is the width of every layer in order, and adjacent pairs make a layer:

sz = [3, 4, 4, 1]
      └┬┘ └┬┘ └┬┘
   Layer(3,4)(4,4)(4,1)

This sliding-window construction encodes the constraint “the previous layer’s output is the next layer’s input” straight into the indices, so the dimensions cannot fail to line up.

The two 4s in Layer(4, 4) mean different things: the left is layer 1’s neuron count (how many numbers it emits), the right is layer 2’s own neuron count. They happen to be equal only because nouts has two 4s in it; write [5, 2, 1] and they differ.

Three lines are the whole forward pass

def __call__(self, x):
    for layer in self.layers:
        x = layer(x)
    return x

Everything hinges on x = layer(x) overwriting itself — Python evaluates the right side first, then assigns to the left:

The raw input appears only in the first pass; after that x holds intermediate activations.

The question that kept coming up in earlier parts — how does one layer’s a become the next layer’s x — is answered by this one assignment. The very same Value objects: called an output by one layer, an input by the next. No conversion, no copy, just a change of role.

This is not recursion: the number of iterations is fixed before it starts (len(self.layers)). The real recursion is build inside backward(), where the graph’s depth is unknown and its shape is a tree.

parameters() rolls up across three levels

Neuron ──concat──→ Layer ──flatten──→ MLP ──flatten again──→ 41

How to read a nested comprehension

[  p  for layer in self.layers  for p in layer.parameters()  ]
   ↑        └──── outer ────┘        └───── inner ─────┘
 output

The two for clauses run in the same order as ordinary loops (outer first); only the produced p has been hoisted to the front. It is equivalent to:

result = []
for layer in self.layers:
    for p in layer.parameters():
        result.append(p)

It has to be flattened. A single-level [layer.parameters() for layer in ...] gives a list of lists, and the later p.grad = 0 fails outright.

What comes back are references, not copies

This is what makes a training loop possible at all:

for p in n.parameters():
    p.data += -0.05 * p.grad     # genuinely changes the network's weights

Check it: n.parameters()[0] is n.layers[0].neurons[0].w[0] → True

The flattened order puts weights first, bias last, because of return self.w + [self.b]:

A few points of Python

__call__: the parentheses are the trigger. Writing n merely refers to the object and does nothing; writing n([2.0, 3.0]) triggers __call__. It is the same mechanism by which a + b triggers __add__. PyTorch overloads it too, which is why you write model(x) rather than model.forward(x).

zip: pairing by position.

sum((wi*xi for wi, xi in zip(self.w, x)), self.b)

Weights and inputs are two parallel lists that must correspond by position. zip meshes them like a zipper, and when the lengths differ it truncates to the shorter one.

That truncation is a hazard: a wrong input length raises nothing, it just silently discards the excess. PyTorch checks shapes.

The second argument to sum(..., self.b) is the start value. The accumulation runs b → b + w₀x₀ → b + w₀x₀ + w₁x₁. Starting from the bias folds it in for free, and builds one node fewer. Without a start value it would begin at 0, which is what needs __radd__ (added in the previous part).

The + in self.w + [self.b] is list concatenation, not vector addition. [w₀, w₁] + [b] → [w₀, w₁, b], so self.b needs the square brackets; self.w + self.b raises a TypeError. One and the same +: concatenation for a list, element-wise addition for an ndarray, a new graph node for a Value — it depends entirely on the type of the left operand.

Why random.uniform(-1, 1) has to be random. Initialised to all zeros, or all the same, the neurons in a layer would produce identical outputs → receive identical gradients → remain identical after the update. Several neurons collapse into one. Random initialisation exists to break the symmetry.

The range −1~1 is chosen because tanh saturates beyond |z| > 3 (slope under 0.01). Weights that are too large put z in the saturated region from the start, and training stalls.

Only the part of the graph that took part gets gradients

Call backward() on a single neuron’s output and only the parameters on that one chain have gradients; the rest of the layer is all zeros:

outs = layer(x)
outs[0].backward()
# layer.neurons[0]'s parameters have gradients
# layer.neurons[1]'s parameters are all 0

Because gradients only flow through the part of the graph that participated. The neurons in a layer are parallel, mutually indifferent chains; they are all activated only once they are gathered into a single loss.

Which is exactly the next step: write a loss function that collects every output into one number, and only then do all 41 parameters have gradients.

In short

The three classes contain no new mathematics; they only package:

• a Neuron is a set of weights plus one activation
• a Layer is a row of parallel Neurons
• an MLP is Layers in series

The hard parts — closures, the topological sort, += — were finished in the previous part; this one just wraps them into a shape that is pleasant to use. That “hard underneath, thin on top” structure is common in framework design; PyTorch’s nn.Module is the same.

Three things come out of it: any structure in one line of declaration — going from a hand-written expression to MLP(3, [4,4,1]), changing the architecture means changing one argument; layers demystified — reading nn.Linear and nn.Sequential later, the structure is identical, only with matrices in place of scalars; the groundwork for a training loop — with parameters() handing back references to every parameter, the loop that actually drives the loss down becomes writable.

The network can compute, but it cannot yet learn — those 41 parameters are still random.

Building a small dataset, writing a loss function, and running a training loop until the loss really comes down is the next part.