Will a detector trained on the left find the kilns on the right?

The kilns have the same shape everywhere. Their colour, and everything around them, changes.

Sentinel-2 tiles, 128 × 128 pixels (1.28 km across), with kiln boxes from SentinelKilnDB (Mondal et al., NeurIPS 2025). Box colours mark the three kiln types.

Each source of images, such as a region, a camera, a season or a drawing style, is a domain.

MixStyle is a domain generalization method. It never sees the target domain.

Report accuracy on each held-out domain, and the average of the four.

Each version is our guess about how a new region might look.

A real region changes soil, crops, water, haze and roofs together, in ways we did not choose.

The blend shows two scenes at once. It is a double exposure, and no region looks like that.

On PACS, none of these general-purpose tricks beats the plain network on the unseen domain.

MixStyle does all three inside the network. First: where does the look of an image live inside a CNN?

A layer with C filters returns C maps. Each map says where its pattern appears, and how strongly.

ResNet-18 after its first stage, for 16 images of 224 × 224: 16 × 64 × 56 × 56.

Fix one image and one channel: you are left with one H × W map. MixStyle works on these maps.

import torch
import torchvision

net = torchvision.models.resnet18()
x = torch.randn(16, 3, 224, 224)                  # 16 RGB images
h = net.maxpool(net.relu(net.bn1(net.conv1(x))))  # the stem
h = net.layer1(h)                                 # first residual stage
print(h.shape)                                    # torch.Size([16, 64, 56, 56])

h[b, c] is one 56 × 56 map: image b, channel c. Dimensions 2 and 3 are height and width.

$\fx{x_2} = 2\,\fx{x_1} + 3$: stretched by 2 and lifted by 3. Same arrangement, different level and spread.

Both normalized maps are identical. Removing μ and σ removed everything that differed; the arrangement is all that is left.

x = torch.tensor([[[[0., 0.], [2., 2.]]],
                  [[[3., 3.], [7., 7.]]]])               # (B, C, H, W) = (2, 1, 2, 2)
mu = x.mean(dim=(2, 3), keepdim=True)                     # [1, 5], shape (2, 1, 1, 1)
sig = x.std(dim=(2, 3), keepdim=True, unbiased=False)     # [1, 2]
x_hat = (x - mu) / sig                                    # both [[-1, -1], [1, 1]]

dim=(2, 3) averages over height and width only: one μ and one σ per image and per channel.

For 16 images and 64 channels: 16 × 64 means and 16 × 64 standard deviations.

Same fields, roads and kilns as west UP. Bihar's colours, brightness and contrast.

A channel's μ says how much of its pattern an image has; its σ says how unevenly. Both differ from region to region. Bold marks the largest value in each column, grey the smallest.

Huang and Belongie's reading: normalizing each image with its own μ and σ removes its style.

From the partner we take only its two numbers, μ(x̃) = 5 and σ(x̃) = 2.

λ is the share of our own statistics that we keep. The rest comes from the partner.

• ✓mean 3 = μ_mix, spread 1.5 = σ_mix
• ✓normalized: [−1, −1; 1, 1], our pattern, and our label

Each channel keeps its pattern. Its μ and σ move towards the partner's.

On PACS, blending averages 82.8%; always swapping, 82.1% (the paper's Table 4).

x = torch.tensor([[[[0., 0.], [2., 2.]]],               # our map, x[0]
                  [[[7., 3.], [7., 3.]]]])              # its partner, x[1]
mu = x.mean(dim=(2, 3), keepdim=True)                    # [1, 5]
sig = x.std(dim=(2, 3), keepdim=True, unbiased=False)    # [1, 2]
x_hat = (x - mu) / sig                                   # the two patterns

lam = 0.5
mu_mix = lam * mu[0] + (1 - lam) * mu[1]       # 0.5 * 1 + 0.5 * 5 = 3
sig_mix = lam * sig[0] + (1 - lam) * sig[1]    # 0.5 * 1 + 0.5 * 2 = 1.5
out = x_hat[0] * sig_mix + mu_mix              # [[[1.5, 1.5], [4.5, 4.5]]]

mu[1] and sig[1] are the partner's statistics; x_hat[0] is our pattern.

perm = torch.randperm(x.size(0))    # a random order of 0, 1, ..., B-1
mu2, sig2 = mu[perm], sig[perm]      # each image's partner statistics

perm = torch.arange(B - 1, -1, -1)          # [5, 4, 3, 2, 1, 0]
perm_j, perm_i = perm.chunk(2)              # [5, 4, 3] and [2, 1, 0]
perm_j = perm_j[torch.randperm(B // 2)]     # for example [4, 3, 5]
perm_i = perm_i[torch.randperm(B // 2)]     # for example [0, 2, 1]
perm = torch.cat([perm_j, perm_i], 0)       # [4, 3, 5, 0, 2, 1]

Images 0, 1, 2 (first region) get partners 4, 3, 5 (second region). Images 3, 4, 5 get partners 0, 2, 1.

Mixing inside one domain helps a little. Mixing across domains helps most.

One λ per image, shared by all its channels: shape B × 1 × 1 × 1. With the paper's α = 0.1, most images get a small nudge or a near-swap; the λ = 0.5 of Section 6 is rare.

Made-up 2 × 2 maps, small enough to check by hand.

Image 1 keeps only 10% of its own statistics. Image 3 keeps all of them.

image 1, channel 0: $\fmix{\mu_{\text{mix}}} = \flam{0.1}\cdot\fmu{1} + \flam{0.9}\cdot\fmu{6} = \fmix{5.5}$, $\;\fmix{\sigma_{\text{mix}}} = \flam{0.1}\cdot\fsig{1} + \flam{0.9}\cdot\fsig{2} = \fmix{1.9}$

mu = x.mean(dim=(2, 3), keepdim=True)                         # (B, C, 1, 1)
sig = (x.var(dim=(2, 3), keepdim=True) + 1e-6).sqrt()         # (B, C, 1, 1)
perm = torch.randperm(x.size(0))                              # partners
lam = torch.distributions.Beta(0.1, 0.1).sample((x.size(0), 1, 1, 1))
mu_mix = lam * mu + (1 - lam) * mu[perm]                      # (B, C, 1, 1)
out = (x - mu) / sig * (lam * sig + (1 - lam) * sig[perm]) + mu_mix

The variance line follows the released code: x.var divides by H·W − 1 and adds a small ε before the square root. Section 8 explains both.

def forward(self, x):
    if random.random() > self.p:    # with probability 1 - p
        return x                    # pass the batch through unchanged

One coin per layer per batch. The paper gives no reason for p = 0.5; one plausible one is that unmixed batches keep the network used to real statistics.

At test time MixStyle does nothing. The deployed model is an ordinary CNN, and it never needs an image from the new region.

mu, sig = mu.detach(), sig.detach()    # statistics become constants for autograd

Toy loss: the top-left output. Detached, its gradient reaches only the cell below it, times σ_mix / σ(x) = 1.5, and none reaches the partner.

import random, torch, torch.nn as nn

class MixStyle(nn.Module):
    def __init__(self, p=0.5, alpha=0.1, eps=1e-6):
        super().__init__()
        self.p, self.eps = p, eps
        self.beta = torch.distributions.Beta(alpha, alpha)

    def forward(self, x):
        if not self.training or random.random() > self.p:
            return x                                        # test time, or not this batch
        B = x.size(0)
        mu = x.mean(dim=[2, 3], keepdim=True)
        sig = (x.var(dim=[2, 3], keepdim=True) + self.eps).sqrt()
        mu, sig = mu.detach(), sig.detach()
        lmda = self.beta.sample((B, 1, 1, 1)).to(x.device)  # one lambda per image
        perm = torch.randperm(B)                            # partners
        mu_mix = mu * lmda + mu[perm] * (1 - lmda)
        sig_mix = sig * lmda + sig[perm] * (1 - lmda)
        return (x - mu) / sig * sig_mix + mu_mix

layer = MixStyle(p=0.5, alpha=0.1)
print(sum(q.numel() for q in layer.parameters()))    # 0

• x.var divides by H·W − 1; the paper's equation divides by H·W. Every σ changes by the same factor ( for 56 × 56), and it cancels in the output.
• eps only matters for a flat channel, such as one that ReLU set entirely to 0.
• Cross-domain mode assumes each batch is two equal halves from two domains, in that order. The data loader must build batches that way.
• With several MixStyle layers, each draws its own shuffle. The paper found this slightly better than one shared order: 82.8% versus 82.4% on PACS.

The candidate places are after res1, res2, res3 and res4.

Early stages: gains. Anything that includes the last stage: worse than no MixStyle at all.

Best here: res12. The last stage hurts again, badly.

After the last stage, μ is the class evidence. Mixing it mixes the evidence of two images and keeps one image's label.

net = torchvision.models.resnet18()
h = torch.randn(4, 512, 7, 7)                                      # a res4 output
torch.allclose(net.avgpool(h).flatten(1), h.mean(dim=(2, 3)))      # True

Sketches of the pattern in the paper's Figure 4 (illustrative points). Shade marks the domain; shape marks the class.

class ResNetMixStyle(nn.Module):
    def __init__(self, num_classes):
        super().__init__()
        self.net = torchvision.models.resnet18(num_classes=num_classes)
        self.ms = MixStyle(p=0.5, alpha=0.1)

    def forward(self, x):
        n = self.net
        x = n.maxpool(n.relu(n.bn1(n.conv1(x))))
        x = self.ms(n.layer1(x))      # after res1
        x = self.ms(n.layer2(x))      # after res2
        x = self.ms(n.layer3(x))      # after res3
        x = n.layer4(x)               # no MixStyle here
        return n.fc(n.avgpool(x).flatten(1))

ResNet-18+ MixStyle, random shuffle+ MixStyle, domain labels

Average: 79.5% → 82.8% without domain labels, 83.7% with them. The largest gains are on art paintings and sketches.

On PACS there is no trade-off: accuracy on the seen domains stays the same while accuracy on the unseen one rises.

MixStyle helps both networks in both directions. RandomErase, a standard re-identification trick, hurts.

Reported as curves in the paper's Figure 3.

Uneven gains: large on some domains (Clipart, MNIST-M), none on others (Artistic). L2A-OT, with its image generator, is slightly ahead on both.

Same boxes, same labels, same loss. MixStyle only changes what the later layers see during training.

Pixel-level illustration on the west UP tile. Inside the detector the same operation acts on early feature maps.

import random


def two_region_batches(index_by_region, batch_size):
    while True:
        a, b = random.sample(list(index_by_region), 2)          # two different regions
        half = batch_size // 2
        yield (random.sample(index_by_region[a], half)          # first half: region a
               + random.sample(index_by_region[b], half))       # second half: region b

index_by_region maps a region name to the dataset indices of its tiles, for example {'pk': [...], 'br': [...]}. The generator never ends; the training loop stops after its planned number of steps.

One region held out at a time, several seeds, plus the in-region score to check nothing was lost.

Look at the errors on the new region first. MixStyle addresses a shift in how things look. A shift in what is there needs new data.

1. Take an early feature map x of shape B × C × H × W.
2. For every image and channel, compute μ and σ over H × W, and detach them.
3. Give every image a partner by shuffling the batch: at random, or across two domains.
4. Draw one λ per image from Beta(0.1, 0.1) and blend: μ_mix, σ_mix.
5. Output σ_mix · (x − μ) / σ + μ_mix: the image's own pattern in a blended look. The label is unchanged.
6. Only in training, on half the batches, after the early stages. Never after the last one.

torch.manual_seed(0)
random.seed(0)
x = torch.randn(8, 16, 7, 7, requires_grad=True)
ms = MixStyle(p=1.0)                                   # always mixes in training
y = ms(x)
assert y.shape == x.shape                              # shape unchanged
assert not torch.allclose(y, x)                        # training: it changes x
y.sum().backward()
assert x.grad is not None                              # gradients reach x
ms.eval()
assert torch.equal(ms(x), x)                           # identity at test time
assert sum(q.numel() for q in ms.parameters()) == 0    # nothing to learn

Run after the Section 8 class, which imports random and torch.

• Zhou, Yang, Qiao and Xiang, Domain Generalization with MixStyle, ICLR 2021 · the paper
• KaiyangZhou/mixstyle-release · the released code
• MixStyle Neural Networks for Domain Generalization and Adaptation · the authors' extended version
• Huang and Belongie, Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization, ICCV 2017 · AdaIN
• Ulyanov, Vedaldi and Lempitsky, Instance Normalization: The Missing Ingredient for Fast Stylization, 2016
• Zhou, Liu, Qiao, Xiang and Loy, Domain Generalization: A Survey · the wider field
• SentinelKilnDB · the kiln tiles used here (Mondal et al., NeurIPS 2025)
• Companion notebook, MixStyle from scratch on 5 × 5 images: view · open in Colab