Interactive Explainer

Why 1×1 Convolutions Save Compute

A $3\times3$ convolution that maps many channels to many channels is expensive: cost grows with $C_{\text{in}}\times C_{\text{out}}$. Inception's trick is to squeeze the channels down with a cheap $1\times1$ convolution first, do the pricey $3\times3$ on the narrow stack, and it can come out several times cheaper for the same output shape. Slide the channel counts and watch the two bars diverge.

~7 min Deep Learning · Modern CNNs · Efficiency

The Lab

Direct vs. bottleneck, in FLOPs

Both paths take a $C_{\text{in}}$-channel feature map of size $H\times W$ and produce a $C_{\text{out}}$-channel map of the same spatial size. The direct path does one $3\times3$ convolution. The bottleneck path first projects to $C_{\text{mid}}$ channels with a $1\times1$ convolution, then does the $3\times3$ on that narrower stack. The bars below are total multiply–adds, drawn to scale.

Presets:

$C_{\text{in}}$ = 256

$C_{\text{mid}}$ (bottleneck) = 64

$C_{\text{out}}$ = 256

Spatial $H = W$ = 28

Two bars to scale: the direct $3\times3$ path versus the $1\times1$-then-$3\times3$ bottleneck (split into its two convolutions).

Direct 3×3 —

Bottleneck total —

Savings ratio —

Push $C_{\text{mid}}$ up past $C_{\text{in}}$. The savings ratio drops below $1$: the bottleneck now costs more than the direct conv, because the $1\times1$ is no longer a squeeze. The trick only pays off when $C_{\text{mid}}$ is genuinely small — a narrow waist between two wide layers. That waist is exactly what "bottleneck" names.

The Payoff

Where the factor comes from

For an output of $H\times W$ pixels and $C_{\text{out}}$ channels, each output value is a dot product over a $K\times K \times C_{\text{in}}$ patch, so a convolution costs

\text{MACs} = H \cdot W \cdot C_{\text{out}} \cdot K^2 \cdot C_{\text{in}} .

Putting the two paths side by side (with $K=3$ for the spatial conv, $K=1$ for the projection):

\underbrace{HW\cdot 9\, C_{\text{in}} C_{\text{out}}}_{\text{direct}} \quad\text{vs.}\quad \underbrace{HW\, C_{\text{mid}}\big(C_{\text{in}} + 9\,C_{\text{out}}\big)}_{\text{bottleneck}} .

The expensive $3\times3$ now runs on $C_{\text{mid}}$ input channels instead of $C_{\text{in}}$ — that is where the big saving lives.
The extra $1\times1$ costs only $HW\,C_{\text{in}}C_{\text{mid}}$: cheap, because a $1\times1$ has no spatial extent, just a per-pixel channel mix.
Net win whenever $C_{\text{mid}}$ is small relative to $C_{\text{in}}$ and $C_{\text{out}}$ — the whole point of a bottleneck.

This one idea recurs everywhere. GoogLeNet/Inception uses $1\times1$ convolutions to shrink channels before the $3\times3$ and $5\times5$ branches. ResNet's "bottleneck block" wraps a $3\times3$ between a $1\times1$ down-projection and a $1\times1$ up-projection. MobileNet's depthwise-separable convolution is the same instinct pushed further. Cheap channel mixing before expensive spatial mixing is a load-bearing pattern of modern CNNs.