What a Neural Network Actually Is: A Function with 13,002 Parameters

Take handwritten digit recognition: given a 28×28 greyscale image, the network has to decide which of the digits 0 to 9 it shows.

A neuron is a container holding one number

The number is between 0.0 and 1.0, and it is called the activation. Closer to 1 means more “active”; closer to 0 means inactive.

More precisely, a neuron is a function: it takes the outputs of every neuron in the previous layer and produces one number in 0–1.

The layers: 784 → 16 → 16 → 10

784 in the input layer. The image is 28×28 = 784 pixels; each pixel’s greyscale value (0.0 black to 1.0 white) is the activation of the corresponding neuron.

Two hidden layers of 16 each. They sit between input and output. They are called “hidden” because their inputs and outputs are not directly observed. Two layers of 16 is an arbitrary choice, picked to keep the demonstration small.

10 in the output layer, one per digit 0–9. Whichever neuron has the highest activation is the network’s answer.

Why have those middle layers at all

The hope is that the network abstracts layer by layer:

• hidden layer 1 recognises small edges (short segments, small arcs)
• hidden layer 2 assembles edges into stroke parts (loops, vertical bars, hooks)
• the output layer assembles parts into digits (loop + bar = 9)

“Break a hard problem into small ones, then recombine” is the central motivation for having layers.

Worth stating plainly: this is only the hope. What the network actually learns can be inspected after training by visualising the weights — and the result looks rather different from this. That is a topic for later.

Weights and biases

Every neuron in one layer connects to every neuron in the next, and each connection carries a weight.

weighted sum = w₁·a₁ + w₂·a₂ + … + wₙ·aₙ

A positive weight means that when the input neuron is active it “encourages” the next neuron to be active; a negative one “inhibits” it. The magnitude is the strength of the connection.

Then a bias b is added to the weighted sum:

weighted sum + b

The bias controls the threshold — how easily the neuron fires. A negative bias means the weighted sum has to be large before anything happens; a positive one makes firing easier.

The activation function

The weighted sum plus bias can be any number at all (3.7, −2.1), but it needs to be squashed into 0–1.

Sigmoid flattens anything into 0–1: a large positive number → close to 1; a large negative number → close to 0; 0 → 0.5.

a = σ(weighted sum + bias) = σ(w₁a₁ + w₂a₂ + … + wₙaₙ + b)
σ(z) = 1 / (1 + e⁻ᶻ)

Modern networks more often use ReLU (Rectified Linear Unit): negative input gives 0, positive input passes through unchanged. Simpler, and it trains better.

13,002 parameters

Counting every adjustable number in this network:

13,002 in all. “Learning” means finding the best values for those 13,002 numbers.

The whole network is one function

All of the above is far more compact as matrix multiplication:

a⁽¹⁾ = σ( W · a⁽⁰⁾ + b )

W is the weight matrix (k×n), a⁽⁰⁾ the previous layer’s activation vector (n×1), b the bias vector (k×1).

So the network is fundamentally one enormous function: 784 numbers (pixel values) go in, several rounds of matrix multiplication and squashing happen, and 10 numbers come out (a confidence per digit). It involves 13,002 parameters, but it is still just a function.

How this differs from ordinary programming

Ordinary programming: a person writes the rules out explicitly (if-else).

A neural network: a person designs only the structure, then uses data to let the network “learn” the rules. You are not writing an algorithm that recognises digits; you are writing an algorithm that can learn to recognise digits.

That is the structure — but those 13,002 parameters are still random, so the network can do nothing at all.

Getting them to the right values is what the next part is about.