AI Tech Inspire

Rectified Flows for Clearing Clouds in Satellite Imagery

Clouds hide a lot more than weather. For Earth observation and geospatial analytics, they obscure crops, roads, and shorelines—exactly the pixels engineers care about. At AI Tech Inspire, we spotted a technical direction that has developers buzzing: using rectified flow models to remove clouds from optical satellite imagery.

Snapshot: what’s being explored

  • Exploration of rectified flow (RF) models for removing clouds in optical satellite images.
  • Common baselines include CNNs, diffusion models (e.g., DiffCR), and multi-temporal transformers.
  • RFs aim for high image quality with fewer sampling steps than diffusion while preserving stability and smooth transport.
  • Plan: train a conditional RF to map cloudy images to cloud-free outputs, conditioned on cloud masks, temporal neighbors, or SAR for thick clouds.
  • Considering pixel-space vs latent-space RF using a pretrained VAE/autoencoder.
  • Looking for prior work applying RFs to image restoration and remote sensing.
  • Seeking techniques for stabilizing conditional RF training and improving sample efficiency.
  • Requests for open datasets in multi-temporal and SAR-optical settings (e.g., Sentinel, Landsat).
  • Interest in architectures, loss functions, and metrics like PSNR, SSIM, SAM, and FID.

Why rectified flows are a compelling angle for de-clouding

Diffusion models set the bar for generative restoration quality, but they can be step-hungry. Rectified flows approximate a straight transport path between a source and target distribution and learn a velocity field that moves samples along that path. In practice, this often means fewer ODE steps than diffusion while keeping the training and sampling dynamics stable.

For cloud removal, that “source” distribution is the cloudy image (plus any conditions), and the “target” is a clean view of the same scene. A conditional RF learns a smooth map between the two. The hypothesis: RFs can maintain the sharp edges and spectral consistency critical for remote sensing while finishing in tens of steps instead of hundreds.

Key takeaway: RFs trade the noise-schedule dance of diffusion for a simpler, often faster, transport story—appealing for operational pipelines.

Conditioning choices: cloud masks, time neighbors, and SAR

De-clouding is only as good as the context fed into the model. Practical conditioning strategies include:

  • Cloud masks: Provide the model an explicit “these pixels are unreliable” channel. Masks can come from rule-based detectors or learned segmenters (e.g., S2-Cloudless-style).
  • Temporal neighbors: Nearby cloud-free snapshots help reconstruct true content instead of hallucinating. Watch for georegistration drift and phenological changes.
  • SAR (e.g., Sentinel-1): When clouds are thick, radar penetrates and anchors structure. Conditioning on SAR improves stability, particularly over water bodies, urban edges, and forest canopies.

Implementation-wise, a standard U-Net or a transformer backbone can fuse these channels with cross-attention or concatenation. If you’re prototyping, the ecosystem around PyTorch or TensorFlow plus Hugging Face model hubs and datasets will speed iteration.

Pixel-space vs latent-space RF

Two viable paths:

  • Pixel-space RF: Direct modeling of multispectral bands (e.g., Sentinel-2) keeps radiometry intact. Pros: faithful spectral handling and no compression artifacts. Cons: heavier compute and memory, especially for large swaths or many bands.
  • Latent-space RF: Use a pretrained VAE/autoencoder (similar in spirit to Stable Diffusion) and learn the flow in latent space. Pros: faster and lighter sampling. Cons: the autoencoder must preserve spectral relationships across bands—a nontrivial requirement for remote sensing.

For engineers, the decision hinges on available CUDA memory, throughput targets, and how much spectral fidelity your downstream tasks (e.g., vegetation indices) demand. On constrained GPUs, latent-space RF may be the pragmatic starting point. Optimize with mixed precision, gradient checkpointing, and CUDA graph capture where possible.

Architecture and loss design: practical recipes

RF training typically minimizes an MSE between predicted and “teacher” velocity along the rectified path. In conditional setups:

  • Backbone: Multi-scale U-Net (with residual blocks) or a Swin-like transformer. For SAR+optical fusion, early fusion with learnable per-band scaling plus mid-level cross-attention works well.
  • Condition injection: Concatenate masks as channels; encode temporal neighbors with a small encoder and inject via cross-attention; encode SAR with separate normalization and a positional alignment module.
  • Losses: Combine velocity loss with content-consistency terms. Examples: L1/L2 on valid (non-cloud) pixels, perceptual/LPIPS on RGB composites, SAM for spectral fidelity, and MS-SSIM for structure. If latents are used, add a reconstruction regularizer on decoded outputs to preserve radiometry.
  • Regularization: Spectral norm on attention layers, gradient clipping, Exponential Moving Average (EMA) of weights, and data augmentations respectful of geospatial semantics (e.g., rotations, flips, mild color jitter on RGB only when scientifically valid).

Avoid over-penalizing areas the model cannot see (e.g., thick clouds with no temporal neighbor). Mask-aware losses and uncertainty weighting help prevent artifacts.

Stabilizing conditional RF training

Here are field-tested stability tips:

  • Curriculum on conditioning: Start with easier cases (thin clouds, good temporal matches), then gradually introduce difficult samples.
  • Micro-conditioning: Provide scalar tokens for sun angle, acquisition time gap, sensor metadata; feed them through MLPs into attention blocks.
  • Alignment-aware batching: Group patches by georegistration quality; unstable alignment can dominate gradients.
  • Time sampling: Train with stratified timesteps along the rectified path; bias more samples to early and late transport phases where errors accumulate.
  • Mixed clean/noisy targets: Occasionally swap in identity mappings (cloud-free to cloud-free). It anchors the model and reduces hallucinations.

Debugging hint: set CUDA_LAUNCH_BLOCKING=1 during tricky backprop issues, then turn it off for speed.

Datasets and benchmarks to consider

Open data is plentiful, but benchmarks for authentic cloud removal need careful curation:

  • Sentinel-2: Global coverage, 10–60 m, multiple bands. Pair cloudy scenes with nearest cloud-free acquisitions; extract cloud masks via public tools.
  • Landsat 8/9: Rich historical archive; good for evaluating across seasons and sensors.
  • SEN12MS: Co-registered SAR+optical, helpful for SAR-conditioned training and testing.
  • DFC 2020/2022 (IEEE GRSS): Multi-sensor datasets that can be adapted for restoration tasks and downstream land-cover evaluation.

Pro tip: create train/val/test splits across different biomes and seasons to test generalization. Share trained checkpoints and inference scripts on Hugging Face for reproducibility.

How to measure success without fooling yourself

Standard vision metrics are a start, but remote sensing has quirks:

  • PSNR/SSIM: Useful for structure, but insensitive to spectral consistency across bands.
  • SAM (Spectral Angle Mapper): Crucial when downstream analysis depends on ratios and indices.
  • FID/LPIPS: Gauge perceptual realism, but beware domain shift; compute on appropriately trained encoders for satellite imagery.
  • Downstream task metrics: Land-cover classification accuracy, change detection F1, or crop-type mapping performance on de-clouded outputs.
  • Physical plausibility checks: NDVI consistency, water body reflectance ranges, and time-series smoothness to catch hallucinations.

Consider a human-in-the-loop review for edge cases (e.g., urban waterfronts), since small errors can look visually plausible yet break analytics.

Why this matters for practitioners

Fewer steps and stable transport align with operational needs: near-real-time delivery, lower GPU bills, and simpler deployment. For teams already comfortable with diffusion-style codebases, rectified flows are a natural extension. It’s reasonable to prototype in PyTorch, then port to TensorRT or ONNX for production.

From a systems view: if RFs reach comparable quality to diffusion with 10–20 solver steps, the throughput gains can make the difference between overnight batch and same-day delivery.

Open questions worth exploring

  • Can latent-space RFs preserve subtle spectral cues needed for agronomy and hydrology? If not, what VAE architectures minimize band-wise bias?
  • What is the best way to quantify “safe” de-clouding that avoids hallucinating new roads or fields? Are uncertainty maps sufficient for users?
  • How robust is SAR conditioning across incidence angles and speckle levels? Do learned despeckling modules help the RF downstream?
  • Which ODE solvers and step schedules provide the best quality vs. latency trade-off for RF sampling?

Getting started: a pragmatic path

If you’re curious to try this:

  • Start with a pixel-space RF on RGB+NIR for Sentinel-2 patches; condition on cloud masks.
  • Add temporal neighbors and evaluate whether SAM/PSNR gains justify the data loader complexity.
  • Introduce SAR conditioning for heavily clouded regions and compare stability under extreme cloud fractions.
  • Benchmark against a strong diffusion baseline (e.g., a U-Net trained in the style of popular diffusion repos) and a transformer-based multi-temporal model.
  • Iterate toward latent-space once the pipeline and metrics are stable.

Engineers don’t need to reinvent the wheel: reuse mature diffusion/RF training utilities, keep instrumentation tight, and publish ablations. This space is moving fast, and rectified flows offer a fresh, efficient lens on a stubborn Earth observation problem. The next time clouds roll in, a well-trained RF might be the shortcut between noisy pixels and actionable insights.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.