Interactive Explainer

Shifted Windows (Swin)

Full self-attention compares every patch to every other patch — cost grows with the square of the number of patches. Swin instead attends only within small windows, which is cheap and linear. The trick that keeps it from being blind across window seams: on alternate layers, shift the window grid by half a window so patches that were on opposite sides of a boundary now sit together. Click a patch to see exactly who it can talk to.

~8 min Deep Learning · Vision Transformers · Attention

Prelude

Why not just attend to everything?

A vision transformer chops an image into a grid of $N \times N$ patches, giving $N^2$ tokens. Global self-attention forms every pair, so its cost scales as $(N^2)^2 = N^4$ — quadratic in the number of tokens. Double the resolution and the attention bill goes up sixteen-fold. That does not survive high-resolution images.

Swin's fix is to restrict attention to non-overlapping windows of $M \times M$ patches. Each window does full attention inside itself and ignores everything outside. With a fixed window size $M$, the total cost is

\underbrace{\left(\tfrac{N}{M}\right)^{2}}_{\text{\# windows}} \times \underbrace{(M^{2})^{2}}_{\text{pairs per window}} \;=\; N^{2} M^{2},

which is linear in the number of tokens $N^2$. The catch is obvious: a patch can never attend to anything outside its window. Information cannot cross a seam. The next section shows the cure.

The Lab

Windows, and the shift that stitches them

Below is a grid of image patches partitioned into attention windows (drawn with bold seams). Click any patch to select it: the patches it can attend to — exactly its own window — light up, and everything else dims. Now flip the Shift switch. The whole window grid slides by half a window, so your selected patch lands in a new window with new neighbours. Stack an unshifted layer and a shifted layer and, over two layers, a patch reaches across boundaries in every direction.

Grid size $N$ = 8

Window size $M$ = 4

Each cell is an image patch. Bold lines are window seams; attention stays inside a window. The selected patch is ringed; its window (the patches it attends to) is highlighted. Toggle Shift and watch that set jump across the old seam.

Patches this one attends to —

Windows this layer —

Full-attention pairs $N^4$ —

Windowed pairs $\sum |w|^2$ —

Push the sliders. As $N$ grows, the orange “full-attention pairs” number explodes like $N^4$ while the green “windowed pairs” number crawls up like $N^2 M^2$. That gap is the entire reason window attention exists: it buys you high-resolution transformers that would otherwise be unaffordable.

The Payoff

Shift, then shift back: growing the receptive field

A single window layer is myopic — a patch sees only its $M \times M$ neighbourhood. But Swin alternates: layer $L$ uses the plain partition, layer $L{+}1$ uses the shifted one (displaced by $\lfloor M/2 \rfloor$). A patch near a seam attends to one group in layer $L$, then a different, overlapping group in layer $L{+}1$. Composed, the two layers let information flow across the boundary in every direction — without ever forming a single global attention matrix.

Cheap. Every layer stays $O(N^2 M^2)$: linear in tokens, never $N^4$.
Connected. Alternating shifted / unshifted windows knit the seams, so the effective receptive field grows layer by layer until it spans the whole image.
Hierarchical. Swin also merges patches between stages, shrinking the grid and enlarging what each token summarises — the pyramid that makes it a drop-in backbone for detection and segmentation.

The real implementation. Rather than build ragged windows at the edges, Swin performs a cyclic shift of the whole feature map, runs ordinary equal-sized window attention, and uses an attention mask so that patches wrapped around from the opposite edge do not actually attend to each other. The picture above draws the honest partition; the trick is just an efficient way to realise the same neighbour sets on a GPU.