Interactive Explainer
Earth-Observation Foundation Models, in Depth
A deep tour: why ImageNet weights underperform on satellites, which pretraining recipes have actually scaled, the 2026 model zoo (Prithvi, SatMAE, Clay, AnySat, MMEarth, …), a live multi-band MAE demo, and an end-to-end worked example from raw Sentinel-2 to a fine-tuned downstream model.
Why is EO different from ImageNet?
A foundation model trained on ImageNet has seen 1.2M three-channel RGB photos shot by humans for humans. A satellite tile is none of those things:
- Multi-spectral, not RGB. Sentinel-2 ships 13 bands (visible + near-IR + short-wave IR + a few atmospheric correction bands). Landsat-8 has 11. Hyperspectral sensors have 200+. ImageNet backbones have learned to read three channels in a fixed order; the moment you feed them 13, the patch-embed layer can't even initialise.
- Calibrated radiance, not aesthetic exposure. Pixel values are physical reflectances, not sRGB. A pixel of value 0.27 in the SWIR-1 band means something quantitative. Auto-exposure + colour-balance assumptions in ImageNet pretraining work against you.
- Multi-temporal. Most EO tasks need a sequence of revisits to see crop growth, urban change, seasonality. ImageNet sees one frame, full stop.
- Multi-resolution / multi-modal. Optical (10m), radar (SAR, 5–20m), elevation (30m), nightlights, climate. A foundation model that knows only one of these misses 80% of the signal.
- Heavy class imbalance + huge unlabeled stash. 10–20 land-cover classes, but most of the planet is one of three (water, forest, cropland). Labels are expensive (often human/expert); unlabelled tiles are essentially free.
- Hierarchical spatial scales. A tile covers a few square km. The same task may need pixel-level (segmentation), patch-level (kiln in this 64×64 chip), or tile-level (deforestation rate over 1km²) outputs. ImageNet classification heads don't generalise.
The 2026 EO foundation-model zoo
By mid-2026 there are roughly a dozen actively-maintained EO foundation models. They differ in modality, pretraining objective, and downstream coverage:
| Model | Year | Backbone | Modalities | Pretrain objective | Best at |
|---|---|---|---|---|---|
| SatMAE | 2022 | ViT-L | Optical (multi-spectral, multi-temporal) | MAE with band+time masking | Multi-temporal classification |
| Prithvi-100M (NASA / IBM) | 2023 | ViT-L | HLS (Sentinel-2 + Landsat-8) | MAE on 6-band tiles | Flood, burn-scar, crop type |
| Prithvi-EO 600M / 2.0 | 2024 | ViT-H | HLS + temporal | MAE + temporal modelling | Hierarchical change detection |
| SatMAE++ | 2023 | ViT-L | Optical + SAR | MAE + cross-modal contrastive | Cross-sensor generalisation |
| SatlasPretrain | 2023 | Swin-B | Sentinel-1, Sentinel-2, NAIP | Supervised on 137 tasks | Multi-task fine-tuning |
| Clay v1 | 2024 | ViT-B/L | Multi-modal embedding | MAE + CLIP-style language alignment | Search / retrieval / zero-shot |
| MMEarth | 2024 | ViT-B | 12 modalities (S2, S1, DEM, climate, biomass…) | Multi-modal MAE | Modality-flexible downstream |
| AnySat | 2024 | JEPA-style | Any optical config (any bands, any res, any time) | Predictive (V-JEPA flavour) | Heterogeneous deployment |
| Presto | 2023 | 1D Transformer over pixels | Sentinel-2 + S1 + ERA5 | MAE on per-pixel time series | Crop-type, very small fine-tune sets |
| GFM (Mendieta et al.) | 2023 | ViT-B | Optical | Continual MAE pretraining from a generic ViT | Cheap to retrain in-domain |
| IBM/NASA Granite-EO | 2024 | ViT-H/22 | HLS | MAE + climate-conditioned objectives | Operational deployment |
Two clusters: the MAE-on-multispectral camp (Prithvi, SatMAE, Granite) and the multi-modal camp (Clay, MMEarth, AnySat). The first is simpler and stronger on a fixed sensor; the second is more flexible at deployment time when you don't know what bands or sensors you'll get.
The pretraining objective: masked autoencoder, but with twists
Almost every successful EO FM uses a masked autoencoder (Step 1 of the vision-pretraining article). The twists matter:
- Spectral masking. Mask whole bands, not just patches. The encoder must learn band-band relations (e.g. NDVI = (NIR - Red) / (NIR + Red) is implicit in any optical FM). SatMAE and MMEarth do this.
- Temporal masking. Mask whole timesteps of a multi-temporal stack. Forces the model to interpolate phenology / weather / change. Prithvi 2.0 and Presto.
- Modality drop. With probability $p$, drop an entire modality (SAR / DEM / climate). Forces graceful degradation. MMEarth standard recipe.
- Resolution + GSD encoding. Add the ground-sample distance (e.g., 10m vs 30m) as a token embedding so the model can normalise scale. Clay and AnySat depend on this.
- Climate / location conditioning. Many recent models condition on lat/lon and season. Useful for generalisation to unseen regions.
Live: multi-band MAE on a synthetic Sentinel-2 tile
We synthesise a 6-band 64×64 tile (mimicking Sentinel-2's B2/B3/B4/B8/B11/B12) with realistic correlations across bands (vegetation correlates green and NIR; water suppresses NIR; soil pops in SWIR). Drag the mask ratio and which bands to mask; the reconstruction is a smart fill from the visible patches, mimicking what an MAE decoder converges to.
From raw Sentinel-2 to a fine-tuned downstream model
The end-to-end pipeline from a Copernicus product to a deployed kiln-detector / crop-type / biomass model.
| Stage | Tooling | Why it matters |
|---|---|---|
| 1. Download | Sentinel Hub, Microsoft Planetary Computer, AWS Open Data, NASA HLS | L1C (top-of-atmosphere) vs L2A (surface reflectance) — pick L2A unless you have your own atmospheric correction |
| 2. Tile + normalise | rasterio, rioxarray | Reproject to a common CRS, tile to 224×224 chips, normalise per-band by training-set mean/std |
| 3. Filter | cloud / snow masks (s2cloudless, SCL band) | ~30% of optical tiles are clouded; if you don't filter, the FM learns to predict clouds and downstream tasks underperform |
| 4. Pretrain (or load) | Hugging Face hub: ibm-nasa-geospatial/Prithvi-100M, made-with-clay/Clay-v1 | Almost always load; you only pretrain if your domain is genuinely off-distribution (hyperspectral, micro-satellite) |
| 5. Adapt encoder | linear probe → LoRA → full FT | For ≤10k labels, linear probe wins; for ≥10k, LoRA on attention; for ≥100k, full FT |
| 6. Task head | UperNet / DPT / linear classifier | For segmentation, UperNet over a ViT; for chip classification, a global-pool + MLP; for regression, an MLP |
| 7. Eval | geospatially held-out tiles (no spatial leakage!) | Random split overestimates by a lot; use spatial blocking (k-fold over geographic blocks) — this is the most common error in EO papers |
| 8. Deploy | onnx / tflite / TorchServe with TIFF preprocessing baked in | The data pipeline is the deployment risk, not the model |
Where general foundation models fit
DINOv2, SAM2, CLIP, and the new generation of vision-language models also work on EO — sometimes very well, sometimes terribly. Rules of thumb:
- DINOv2 features on RGB tiles. Match or beat ImageNet-pretrained ResNets on small-data EO classification. Worse than EO-specific FMs on multi-spectral or multi-temporal data.
- SAM2 for segmentation prompts. Excellent zero-shot for "draw me a polygon around this object" if the object is in RGB and visually distinct (kilns, ships, buildings, large fields). Falls apart on small-object or spectral-distinguishable classes.
- CLIP for retrieval. Find tiles "containing a brick kiln" via text query — works surprisingly well on RGB. Combine with EO-specific embedding for re-ranking.
- Pixtral / GPT-4o for visual QA on tiles. Useful when you want quick exploration ("are there fires visible in this tile?"), useless when you want quantitative radiometric answers (the thermal-trap argument applies).
- VLM + EO FM stack. The 2026 frontier: EO FM embeds the tile; a small adapter projects to LLM space; the LLM answers free-form questions over the tile. EarthGPT / RemoteCLIP / GeoChat are early examples.
Practical tips before you build
- Don't pretrain unless you must. Start with Prithvi or Clay; you almost certainly can't beat their tens-of-thousands of GPU-hours. Pretrain only for a domain shift not covered (UAV, micro-satellite, hyperspectral with novel bands).
- The label budget is the model. 200 labelled chips with a frozen FM linear probe will beat 200 chips with a from-scratch ResNet by ~15-30 mAP points.
- Tile size matters. 224×224 is the standard for ViT-pretrained FMs; many EO objects (small kilns, buildings, vehicles) need finer; reach for SAM-style high-res or sliding window.
- Active learning + spatial blocking. Combine the active learning article with spatial-block CV to spend your label budget on tiles that genuinely improve generalisation.
- Conformal prediction on outputs. Wrap any deployed EO model in conformal prediction for trustworthy intervals; the marginal coverage guarantee is exactly what regulators want.
- Calibration matters more here than in ImageNet. Predicted probability of "kiln" in policy applications must be calibrated; use temperature scaling per region.