ONE LINEAR MODEL, TWO VIEWS

One line cannot learn XOR. One more column can.

Train a logistic regression on XOR and watch the best possible line fail. Then append the feature x₁x₂: the same linear model separates the lifted points with a flat plane in three dimensions, and that plane, seen from above, is the curve you get in the input plane.

Choose the data

XOR labels: blue where x₁ and x₂ share a sign, orange where the signs differ.

Choose the features

Logistic regression on the raw inputs.

Set or learn the weights

w₁ · x₁ 0.70 w₂ · x₂ −0.45 w₃ · φ 0.00 b · bias 0.15

Step: 0
Learning rate: 0.50
Update: full batch

Try the sliders first.

Prediction to test

With x₁ and x₂ alone, the loss cannot drop below ln 2 ≈ 0.693, the loss of guessing 0.5. With x₁x₂ added, it can go to zero.

TWO FEATURES: x₁ AND x₂

Can one straight line separate XOR?

100 points · 25 per corner

Binary cross-entropy—chance = ln 2 = 0.693

Accuracy—100 points

Boundary in input space—the set z = 0 drawn in (x₁, x₂)

Boundary in feature space—always flat

INPUT SPACE

What the classifier draws in (x₁, x₂)

—

p(y = 1) near 0 p(y = 1) near 1 class 0 class 1 boundary z = 0

Background colour is the model probability. The dark curve is where z = 0, traced from the live weights.

—

FEATURE SPACE

The same boundary in (x₁, x₂, φ)

surface z = φ, coloured by p plane z = 0 plane ∩ surface its shadow on the floor

Drag to orbit, scroll to zoom, arrow keys to rotate, 0 to reset. Edge-on turns the plane into a line. Top reproduces the left panel.

—

TRAINING

Loss after every gradient step

—

The dashed line is ln 2, the loss of predicting 0.5 everywhere. The logistic loss is convex, so gradient descent reaches the best weights for the chosen features.

WHAT THE PICTURES PROVE

The nonlinearity was never in the classifier.

No line can do it

Suppose a line puts both blue corners, (1, 1) and (−1, −1), on its positive side. A half-plane is convex, so it also contains their midpoint, the origin. The same argument for the orange corners (1, −1) and (−1, 1) puts the origin on the negative side. Contradiction.

Training is not the problem. The logistic loss is convex in (w, b), so gradient descent reaches the global optimum, and on symmetric XOR that optimum is w = 0: predict 0.5 everywhere, loss ln 2.

One column fixes it

On the corners the new column x₁x₂ is +1 for blue and −1 for orange, so it separates the classes on its own. The weights (0, 0, −1) give z = −x₁x₂: p = σ(−1) ≈ 0.27 on blue, σ(+1) ≈ 0.73 on orange, and scaling w₃ sharpens both.

In feature space that boundary is the flat sheet u₃ = 0. Train with the product feature and watch the plane settle near it.

Same boundary, two views

In feature coordinates the boundary is b + w₁u₁ + w₂u₂ + w₃u₃ = 0, a plane, flat by construction. Substitute u₃ = x₁x₂ and solve: x₂ = −(b + w₁x₁) / (w₂ + w₃x₁), a hyperbola with asymptotes x₁ = −w₂/w₃ and x₂ = −w₁/w₃.

Press Edge-on to see the plane as a line, then Top to see its trace as the curve on the left. Same object, two projections.

Networks learn φ

The decoy x₁² + x₂² equals 2 at every corner and helps nobody: a useful feature must differ across classes. The ReLU feature max(0, x₁ + x₂ − 1) is one hidden neuron with weights (1, 1) and bias −1, chosen by hand. It flags the (1, 1) corner, and with the line it solves XOR.

A hidden layer is a bank of such units whose weights are learned by the same gradient descent. The ReLU playground shows that happening.