Karpathy V1 · node granularity

Expand tanh, and the gradients come out identical

The same tanh, this time assembled from six primitives: *, exp, +, **-1. Not one derivative has to be worked out by hand — watch the engine chain it together itself.

Forward

—

CODE

e’s gradient = two paths accumulated

via the den path—

via the num path—

e.grad, total—

This is what += is for. e is used by both the numerator and the denominator, so two _backward calls each write to e.grad in turn, and += accumulates them automatically. With =, one of them is lost.

Against the packed version

expanded, n.grad—

packed, n.grad+0.5000

Use ← → to step through. Green = forward, orange = backward, red = e’s two paths and the moment they accumulate.