WIDTH ON PAPER ≠ WIDTH IN USE

Four neurons can behave like one.

Train the same 2 → 4 ReLU → 1 network on balanced XOR. Change only how its four hidden neurons begin, then watch their decision hinges and measured feature rank.

Choose initialization

All four parameter slots contain the same numbers. They are separate arrays, not shared weights.

Apply gradient descent

Step: 0
Learning rate: 0.20
Update: full batch

Identical initialization loaded. Predict what can change after one step.

Prediction to test

Exact copies remain exact copies. The model has nominal width 4, but only effective width 1.

IDENTICAL INITIALIZATION

Can four overlaid hinges solve XOR?

100 fixed points · 25 per quadrant

Binary cross-entropy—mean over 100 points

Accuracy—balanced XOR

Max neuron separation—max ‖(w,b)_i − (w,b)_j‖

Hidden feature rank—rank of the 100 × 4 activation matrix

MODEL OUTPUT

Decision surface

—

class 0 class 1 training point

Background colour is the model probability. The outlined points are the balanced training set.

HIDDEN LAYER

Four ReLU hinges and active directions

—

Each line is w_j^Tx + b_j = 0. Its arrow points toward the half-plane where that ReLU is active.

WHAT THE METRICS MEAN

The problem is lost representational capacity.

Identical

All four activations are the same column. Rank stays 1 and separation stays 0, so the output is only a scaled copy of one ReLU feature. On this XOR set it typically settles near 75%: one quadrant remains wrong.

Slightly perturbed

The equality constraint is gone. The coloured hinges can separate and the feature rank rises above 1. Breaking symmetry is necessary; it does not guarantee that this particular start will find the best XOR solution.

Independent

Distinct starting directions let gradient descent combine multiple half-planes. With the fixed seed here, all four alternating XOR regions are learned and the measured accuracy reaches 100%.