Interactive Explainer

A Gated Residual Block

This is the repeating unit inside WaveNet and modern Temporal Convolutional Networks. One dilated causal convolution feeds two branches — a $\tanh$ that proposes content and a $\sigma$ that decides how much of it passes. Their product is reshaped, then a skip copy peels off while a residual copy adds back to the input. Step through it one stage at a time and turn the gate knob to watch the valve open and close.

~9 min Deep Learning · Sequence Models · TCN / WaveNet

Prelude

Two branches, one product

A plain convolution outputs a single pre-activation and squashes it. A gated block instead computes two projections of the same input and lets one modulate the other. The $\tanh$ branch proposes a signed value in $(-1, 1)$ — the content. The $\sigma$ branch produces a number in $(0, 1)$ — a per-channel gate, or valve. Multiply them:

z \;=\; \tanh\!\big(W_f * x\big) \;\odot\; \sigma\!\big(W_g * x\big) .

Where the gate is near $0$ the channel is shut off; near $1$ the content flows through untouched. On top of this gate sit the two paths that make very deep stacks trainable: a skip connection that routes each block's output straight to the final head, and a residual connection that adds the block's output back onto its input so gradients have a short, unobstructed route home.

The Lab

Step through one block

Press Step to reveal the dataflow one stage at a time, from the input $x$ all the way to the residual add that feeds the next block. Each box shows a $4$-channel activation as signed bars. Then drag the gate bias — it slides the $\sigma$ branch toward shut ($0$) or open ($1$) and you can watch the product $z$ shrink or grow, channel by channel.

Gate bias (shifts $\sigma$ branch) $b$ = 0.0

One gated residual block. Content ($\tanh$) is modulated by the gate ($\sigma$); the product is projected, then skips out and adds back.

content tanh gate σ product z 1×1 output

Stage 0 / 7. The input activation $x$ enters the block. Press Step to send it through the dilated causal convolution.

Stage 0 / 7

Mean gate $\bar\sigma$ —

Signal let through —

Drive the gate to its extremes. Slide the bias to $-4$: every $\sigma$ collapses toward $0$, the product $z$ flatlines, and almost nothing reaches the output — yet the residual path still carries $x$ forward untouched, so the block degrades gracefully to a no-op. Slide it to $+4$: the valves open, and the full content passes. The gate is a learned, per-channel volume knob.

The Payoff

Why gate, skip, and residual all earn their place

The gate gives the network a multiplicative, data-dependent switch. Additive activations like ReLU can only push; a $\sigma$ valve can also hold back, which is what lets these models be selective about long-range context.
The skip path collects a contribution from every block and sums them at the output head, so even shallow-depth features reach the prediction without being overwritten.
The residual path makes each block compute a correction $x \leftarrow x + f(x)$ rather than a full replacement. Gradients flow back through the identity term unattenuated — the same trick that lets ResNets and Transformers stack dozens of layers.

\text{skip} = W_o * z, \qquad x_{\text{next}} = x + W_o * z .

Stack them. WaveNet chains dozens of these blocks with dilations cycling $1, 2, 4, \dots, 512$. Each block is cheap, the residual keeps optimisation stable, the skips aggregate multi-scale evidence, and the gates decide, at every layer, which channels of the past deserve to speak.