Interactive Explainer

Causal Convolution

A convolution slides a small kernel over a sequence. Make it causal and it is forbidden from peeking into the future: the output at time $t$ may only touch inputs at $t$ and earlier. Add left-padding to keep the length intact, and dilation to see far into the past without spending a single extra weight. Drag the kernel below and watch the rules enforce themselves.

~8 min Deep Learning · Sequence Models · TCN / WaveNet

Prelude

One kernel, one rule: no looking ahead

A 1-D convolution with kernel size $K$ computes each output as a weighted sum of $K$ neighbouring inputs. In a plain conv those neighbours straddle the current position — some are in the future. For streaming data (audio, text, forecasts) that is cheating: at time $t$ you simply do not have $x[t+1]$ yet. A causal convolution fixes the window so every tap lands on the present or the past:

y[t] \;=\; \sum_{k=0}^{K-1} w_k \; x[\,t - k\cdot d\,], \qquad t - k\,d \le t .

Here $d$ is the dilation: the stride between taps. With $d=1$ the taps are adjacent; with $d>1$ they spread out, opening holes so the kernel reaches further back in time using the exact same $K$ weights. The stretch of input a single output can see is its receptive field, $(K-1)\,d + 1$.

The Lab

Slide the kernel across the sequence

The top row is the input sequence $x$; the bottom row is the output $y$. The dashed orange line is now — the current time $t$. Everything to its right is the shaded future, off-limits to a causal kernel. The blue-outlined cells are the $K$ taps feeding the current output. Move now, resize the kernel, stretch the dilation, and toggle left-padding.

Kernel size $K$ = 3

Dilation $d$ = 1

Now (time $t$) $t$ = 8

Causal 1-D convolution. Output $y[t]$ reads only taps at $t, t-d, \dots, t-(K{-}1)d$ — never across the now line.

kernel tap current output padding zero future (masked)

Receptive field $(K{-}1)d+1$ —

Input length 16

Output length —

Left-pad zeros —

Try this. Turn left-padding off and drag now to the far left. The kernel's oldest taps fall off the edge of the sequence (marked in orange) — there is no $x[-1]$ to read, so no output can be produced there. That is exactly why the output is shorter than the input, and why the missing left cells are greyed out below.

The Payoff

Why dilation is the whole trick

Push the dilation slider from $1$ to $4$ and keep the kernel size fixed at $K=3$. The kernel still has only three weights — but its receptive field leaps from $3$ to $9$. Stack a handful of such layers with dilations $1, 2, 4, 8, \dots$ and the receptive field grows exponentially with depth while the parameter count grows only linearly. This is the engine behind WaveNet and Temporal Convolutional Networks (TCNs).

Causality is a masking rule, not a new operation: forbid any tap with index $> t$. It lets a conv model streaming data honestly.
Left-padding of $(K{-}1)d$ zeros restores the length so the output lines up one-to-one with the input — essential for stacking layers.
Dilation buys reach for free: the same $K$ weights, spaced $d$ apart, cover $(K{-}1)d+1$ timesteps.

Connect it up. A causal conv is the convolutional cousin of an RNN: both process a sequence respecting time's arrow. But where an RNN must unroll step by step, a stack of dilated causal convs sees the whole past in parallel — which is why TCNs often train faster than recurrent models on the same forecasting task.