Time-Series Transformers, Patched

What's hard about time-series forecasting?

Three things make time series harder than a regular ML problem:

• Multiple seasonalities. Daily, weekly, yearly — sometimes all in one signal.
• Distribution drift. A sensor's mean and variance shift over time. Yesterday's training distribution isn't today's test distribution.
• Long horizons. Predicting the next point is easy. Predicting 96 points ahead from a 96-point context is where most methods fall apart.

The four baselines you owe yourself

• Seasonal-naive. Predict $\hat y_{t+h} = y_{t + h - S}$ where $S$ is the seasonal period (24 hours, 7 days). It's a one-line baseline and beats the deep models 30% of the time.
• ARIMA / ETS. Classical statistical models with explicit trend / seasonality / autoregressive terms. Strong on smooth, well-behaved signals.
• LSTM / GRU. Recurrent networks. Decade of dominance; finally outperformed on long horizons by patched transformers.
• PatchTST. Treat the sequence the way a Vision Transformer treats an image: split into non-overlapping patches, embed each patch, run a vanilla transformer over the patch tokens, project back to the forecast horizon.

The patch trick

The earlier wave of time-series transformers (Informer, Autoformer, FEDformer) treated each timestep as a token. At long context lengths this drowns the model in tokens and the $O(N^2)$ cost balloons. PatchTST groups every $P$ consecutive points into one token (just like ViT does with $P \times P$ patches of an image), reducing token count by $P\times$ and giving the model a slightly smoother view of local structure.

That's it. Patch length $P$ is the only real new hyperparameter. Channel-independence (predict each channel with the same shared transformer, so multivariate isn't forced into a giant vocabulary) is the second trick. The third — instance normalisation per series — handles distribution drift.

Channel independence — the unintuitive choice

PatchTST predicts each variate $v$ using the same shared transformer, with no cross-variate attention until a final projection. Reasoning:

• Overfitting protection. Cross-variate attention has $V^2$ parameters per layer; in benchmarks with $V \sim 10$ variates and a few thousand training windows, it overfits aggressively.
• Transferability. A channel-independent model can train on series with different variate counts (one model, many tasks). This is what enables foundation-style pretraining.
• The trade-off. You lose the ability to learn that "channel A leads channel B by 3 steps". When that cross-channel signal matters, use iTransformer (each variate = one token, attention is over variates not over time).

Empirically the two cover different regimes: PatchTST wins on univariate-ish panels with many time steps; iTransformer wins when cross-variate structure dominates.

RevIN — the instance normalisation that fixes distribution drift

A time series usually has a non-stationary mean and variance. The classical fix is differencing or log-transforming; PatchTST's modern fix is Reversible Instance Normalization (Kim et al., 2022):

Standardise the input window using its own mean/variance. Run the model on the normalised series. Un-normalise the forecast with the same statistics. Three lines of code; consistently +1–2% horizon-RMSE improvement on benchmark splits. Critical for chartof-the-decade-quality results.

Race the four models live

Below: a synthetic AQ-flavour signal (daily + weekly + noise + slow drift). Slide the horizon and patch length; all four models forecast and the RMSE updates.

What 2024-2026 has taught us

• State-space models (Mamba, S5). Linear attention with a structured state-space recursion. Much faster on very long contexts; for forecasting they trade blows with PatchTST and increasingly win on long horizons.
• iTransformer. Treats each variate as a token rather than each timestep, so the transformer attention learns cross-series structure (which other sensors predict which). Strong on multivariate panels.
• Foundation models for time series. TimesFM (Google), Chronos (Amazon), Lag-Llama, Moirai — big pretrained models that zero-shot or one-shot a new time-series task. Useful when you have many short series with no per-series fine-tune budget.
• Forecast distributions. Quantile regression heads, conformal calibration, deep AR with normalising flows — the field has moved from "predict a number" to "predict an interval". The BO and Bayesian playground articles are useful here.

How to set up a forecasting evaluation that doesn't lie

• Backtesting, not random splits. Use rolling-origin evaluation: train up to time $t$, forecast $[t+1, t+H]$, advance $t$. Random splits over time leak information.
• Multiple metrics. Report MSE and MAE (different sensitivity to outliers), plus a scale-free metric (sMAPE or MASE) so you can aggregate across datasets.
• Compare against the seasonal-naive. Always. A model that doesn't beat $\hat y_{t+h} = y_{t+h-S}$ isn't a forecasting model.
• Interval coverage, not just point forecast. Report 90%-CI coverage and width — a tight interval that misses 30% of the time is worse than a wider one that hits 90%. Wrap with conformal prediction for cheap distribution-free intervals.
• Long horizon = horizon ≥ context. The standard benchmarks (ETT, Weather, Traffic, ECL) all evaluate H ∈ {96, 192, 336, 720}. Anything shorter is a different problem.

Reading list

• Nie, Nguyen, Sinthong, Kalagnanam (2023) — A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. The PatchTST paper.
• Kim et al. (2022) — Reversible Instance Normalization.
• Liu et al. (2024) — iTransformer.
• Gu & Dao (2023) — Mamba.
• Das et al. (2024) — TimesFM.
• Ansari et al. (2024) — Chronos.
• Hyndman & Athanasopoulos — Forecasting: Principles and Practice. The classical baseline-and-method reference.