Interactive Explainer
A Visual Proof That One Layer Can Compute Anything
The Universal Approximation Theorem says a neural net with a single hidden layer can mimic almost any function. This page rebuilds Michael Nielsen's beautiful visual argument, one grabbable piece at a time—you'll flatten a sigmoid into a step, glue two steps into a bump, design a whole curve by hand, then stack towers into a 3-D surface. No proof. Just pictures you can drag.
The claim that sounds too good
Here is one of the most quoted results in machine learning: a feed-forward network with a single hidden layer can approximate any continuous function to any accuracy, given enough neurons. One layer. Any function. People repeat it like a slogan—but almost nobody can see why it's true.
We're going to make it visible, following Michael Nielsen's construction from Chapter 4 of Neural Networks and Deep Learning. Two honest caveats first, because the theorem is a little narrower than the slogan:
The whole plan, in seven moves
- A sigmoid neuron is an S-curve.
- Crank its weight and the S becomes a step.
- Subtract two steps to get a bump.
- Line up bumps to design any 1-D curve.
- In 2-D, cross two bumps and threshold to get a tower.
- Tile towers to build any surface.
- Notice what the proof doesn't promise.
The trick the whole way through: stop thinking of the hidden layer as “a network” and start thinking of it as a kit of interchangeable bricks. Let's forge the first brick.
One sigmoid neuron is an S-curve
A single hidden neuron with a sigmoid activation computes
where $\sigma(z) = 1/(1 + e^{-z})$ is the smooth S-shaped sigmoid. The weight $w$ controls how steep the S is; the bias $b$ slides it left and right. That's the neuron's entire repertoire—a single soft “on/off” ramp.
Drag both sliders and watch the curve tilt and slide.
Crank the weight → a step function
When the weight is enormous, the sigmoid stops looking like a gentle S and snaps into a step function: $0$ on the left, $1$ on the right, with a near-vertical cliff in between. That's a huge simplification—a step is described by a single number: the position of its cliff.
The cliff sits where the sigmoid's input is zero, i.e. $wx + b = 0$, so $x = -b/w$. Nielsen's key move is to rename the neuron by this position. Define the step position
Now we can forget $w$ and $b$ and just say “a step at $s$.” The first slider below sets $s$ directly (it solves for the bias behind the scenes). The second shows why a big weight is what lets us pretend a sigmoid is a clean step.
Slide the steepness up to make a sharp cliff, then down to watch it melt back into a soft S.
Two steps make a bump
Let's build this one slowly, by stacking sigmoids one at a time. Below is a little workbench: a list of sigmoid neurons and a plot. Each neuron is drawn as a dashed curve in its own colour; the orange line is the running total—everything added up. Use the ● button to switch a neuron on or off, and watch the total change.
Start with the two neurons already there but turn the second one off: you're left with a single rising step at $s_1$, with output weight $+1$. It goes up and stays up—not very local. Now turn the second one back on. It's a step at $s_2$, but its output weight is $-1$, so it's a step that goes down. Add the two together:
To the left of $s_1$ both are $0$, so the total is $0$. Between $s_1$ and $s_2$ only the first is on, so the total sits at $+1$. After $s_2$ the down-step kicks in and cancels it back to $0$. What's left is a bump—a little plateau between the two steps. That's our brick.
Toggle the second neuron off and on to see the bump appear and vanish. Drag the positions and weights. Then press Add sigmoid and stack a few more—every curve you build is just a sum of these soft steps.
Design a curve by hand
Chop $[0, 1]$ into $N$ adjacent slabs and drop one bump on each. The hidden layer's output becomes a row of independent plateaus—bump $i$ contributes its height on slab $i$:
Each slider below is one bump's height. The faint shaded blocks are the bumps themselves; the blue line is what the network actually outputs as it rides along their tops. Set the bars to trace the orange target—then add more slabs for a finer trace.
Try it by hand, or hit Auto-fit to snap every bar to the target. Then drag $N$ up and watch the staircase tighten.
One small wrinkle — safe to skip
You may have noticed the bars don't sit at exactly the same heights as the target. Here's why, in one breath: a real output neuron takes the bumps' total and squishes it through one more sigmoid. So to make the final answer land on, say, $0.8$, the bumps don't aim for $0.8$—they aim for the value that becomes $0.8$ after the squish. That value is written $\sigma^{-1}(0.8)$, read “sigma-inverse,” and it just means undo the sigmoid.
You never have to think about this to use the tool above. But if you press “Peek: the un-squished goal,” the plot switches to that stretched-out world—and there the bumps literally add up to the goal, no squishing in the way.
So far: bricks on a line
Pause and take stock. We turned a neuron into a step, two steps into a bump, and a row of bumps into any 1-D curve we like. The recipe was: chop the input into slabs, raise a bump to the right height on each.
Now we want functions of two inputs—a height for every point on a floor, i.e. a surface. The plan is exactly the same, but our brick has to become a tower: something that's tall in one little square of the floor and flat everywhere else. Building that tower is the one genuinely clever step in the whole argument, so we'll take it slowly.
Two inputs: build a tower, step by step
With two inputs $x$ and $y$, a neuron's “step” becomes a wall: it only listens to $x$, so it's a cliff running all the way along $y$. Watch the four panels below—they're the same construction as before, just lifted into 3-D. Drag any slider; all four update together. Drag any panel to orbit it (or use the Rotate slider) and look around the build.
- 1. Wall. One step in $x$ — a cliff.
- 2. Ridge. Two steps subtracted — a bump in $x$, a long wall-top.
- 3. Cross. Add a ridge in $x$ and a ridge in $y$. They pile to height $2h$ where they overlap, $h$ on the arms.
- 4. Tower. Pass the cross through one more sigmoid with a threshold between $h$ and $2h$. Only the $2h$ overlap survives.
Tile towers into any surface
A tower is a movable, resizable block of adjustable height. Cover the floor with a grid of them and set each tower's height to the value of the target surface at that spot. The sum is a 3-D “staircase” that hugs the surface—and just like in 1-D, a finer grid means a closer fit. That's universal approximation for two inputs, and the same move tiles into any number of inputs.
The Single spike target makes the towers easy to see—they stack into a little stepped pyramid. Drag resolution up to add towers and watch the blocks shrink toward the smooth target. Drag a surface to orbit it (or use the Rotate slider) and admire the skyline.
What the proof does not say
You've now built the whole argument with your own hands: step → bump → designed curve → tower → tiled surface. It's genuinely a proof of universality. But four things quietly get lost when it's flattened into a slogan:
“One hidden layer is enough in practice.”
Enough width suffices in principle—but the width can
be astronomical. Our tiling spent $4$ neurons per tower and still
gave a blocky fit. Depth lets a network build bumps from
bumps and reuse parts, which is why real networks are deep, not
absurdly wide.
“Training will find this network.”
We placed every step and height by hand. The proof shows a
good network exists; it's silent on whether gradient
descent can discover it. Existence and learnability are different
questions.
“Sigmoids are special.”
Nothing here needs the sigmoid in particular—any non-linear
activation that can act like a step (or a hinge, for ReLU) works.
The S-curve just made the step picture vivid.
“It means networks generalize.”
The construction fits the target on the region you tile—it's
about representing a function, not predicting well on new
inputs. Generalization is a separate story told by data and
inductive bias.