GenerativeAI

Official benchmark and evaluation repository for:

Generative AI in Remote Sensing: A Unified Perspective from Tasks to Foundation Models
Sirui Wang, Jiang He, Natalia Blasco Andreo, Zhitong Xiong, and Xiao Xiang Zhu
2026

This repository provides testing scripts, benchmark data organization, model-specific evaluation folders, and metric calculation code for remote sensing generative AI models.

The repository is designed for evaluation and reproduction. For each tested model, users should first follow the original model repository to configure its environment and download the required checkpoints. Then, use the scripts and data structure provided here to run inference and compute metrics under a unified benchmark protocol.

---

News

• 2026-06: Initial release of the benchmark repository.
• 2026-06: Released testing folders for text-to-image, mask/layout-to-image, and super-resolution tasks.
• 2026-06: Released unified benchmark data structure under Data/Inputs/.
• 2026-06: Released metric calculation scripts for T2I, M2I, and SR evaluation.
• 2026-06: Added model-specific folders for representative text-to-image remote sensing generators, including CRS-Diff, DiffusionSat, FLUX.1, GeoRSSD, GeoSynth, Stable Diffusion 3.5, Text2Earth, and Txt2Img-MHN.

---

Repository Structure

The repository is organized by task and evaluation stage.

GenerativeAI/
├── Data/
│   ├── Inputs/
│   │   ├── T2I/
│   │   ├── M2I/
│   │   └── SR/
│   └── Results/
│       ├── T2I/
│       ├── M2I/
│       └── SR/
│
├── T2I/
│   ├── CRS-Diff/
│   ├── DiffusionSat/
│   ├── Flux.1/
│   ├── GeoRSSD/
│   ├── GeoSynth/
│   ├── Stable_diffusion_3.5/
│   ├── Text2Earth/
│   └── Txt2Img-MHN/
│
├── M2I/
│   └── ...
│
├── SR/
│   └── ...
│
├── metrics_utils_t2i.py
├── metrics_utils_m2i.py
└── metrics_utils_sr.py

Folder Meaning

Path	Description
Data/Inputs/	Unified benchmark inputs used in the paper.
Data/Inputs/T2I/	Text prompts and reference images for text-to-image evaluation.
Data/Inputs/M2I/	Mask/layout/OSM inputs and reference images for mask/layout-to-image evaluation.
Data/Inputs/SR/	Low-resolution and reference images for super-resolution evaluation.
Data/Results/	Recommended location for storing generated outputs.
T2I/	Model-specific testing scripts or reproduction notes for text-to-image models.
M2I/	Model-specific testing scripts or reproduction notes for mask/layout-to-image models.
SR/	Model-specific testing scripts or reproduction notes for super-resolution models.
metrics_utils_t2i.py	Metric calculation script for text-to-image generation.
metrics_utils_m2i.py	Metric calculation script for mask/layout-to-image generation.
metrics_utils_sr.py	Metric calculation script for super-resolution.

---

Important Note on Model Reproduction

This repository does not replace the original repositories of each model.

For each reproduced model, please follow this workflow:

1. Go to the original GitHub repository of the model.
2. Install the environment required by that model.
3. Download the official or released checkpoints.
4. Use the benchmark inputs provided in Data/Inputs/.
5. Save generated outputs to Data/Results/.
6. Run the corresponding metric script in this repository.

This design avoids modifying third-party implementations and allows each model to be evaluated with its official environment and checkpoint.

---

Benchmark Tasks

We provide evaluation support for three representative remote sensing generation tasks.

---

1. Text-to-Image Generation

Task folder:

T2I/

Input data:

Data/Inputs/T2I/

Recommended output path:

Data/Results/T2I/<model_name>/

Included model folders:

T2I/
├── CRS-Diff/
├── DiffusionSat/
├── Flux.1/
├── GeoRSSD/
├── GeoSynth/
├── Stable_diffusion_3.5/
├── Text2Earth/
└── Txt2Img-MHN/

Typical workflow:

# 1. Configure the original model environment.
# 2. Download the required checkpoint from the original model repository.
# 3. Run the model using inputs from Data/Inputs/T2I/.
# 4. Save generated images to Data/Results/T2I/<model_name>/.
# 5. Compute metrics.

python metrics_utils_t2i.py --help

Metrics may include:

• FID
• CLIPScore
• BLIP image similarity
• CLIP-based semantic consistency
• NIQE
• BRISQUE

---

2. Mask/Layout-to-Image Generation

Task folder:

M2I/

Input data:

Data/Inputs/M2I/

Recommended output path:

Data/Results/M2I/<model_name>/

Typical workflow:

# 1. Configure the original model environment.
# 2. Download the required checkpoint.
# 3. Run the model using masks, layouts, or OSM inputs from Data/Inputs/M2I/.
# 4. Save generated images to Data/Results/M2I/<model_name>/.
# 5. Compute metrics.

python metrics_utils_m2i.py --help

Metrics may include:

• FID
• CLIPScore
• BLIP image similarity
• CLIP-based class semantic consistency
• LPIPS
• NIQE
• BRISQUE

Note on semantic consistency:
The provided semantic consistency score is based on CLIP similarity between the generated image and text labels derived from the input mask. It measures class-level semantic agreement, not pixel-level layout alignment. If pixel-level layout consistency is required, users should additionally apply a semantic segmentation model to the generated image and compute mIoU or pixel accuracy against the input mask.

---

3. Super-Resolution

Task folder:

SR/

Input data:

Data/Inputs/SR/

Recommended output path:

Data/Results/SR/<model_name>/

Typical workflow:

# 1. Configure the original model environment.
# 2. Download the required checkpoint.
# 3. Run super-resolution inference using inputs from Data/Inputs/SR/.
# 4. Save generated images to Data/Results/SR/<model_name>/.
# 5. Compute metrics.

python metrics_utils_sr.py --help

Metrics may include:

• PSNR
• SSIM
• LPIPS
• NIQE
• BRISQUE
• SAM

Note on SAM:
SAM refers to Spectral Angle Mapper. When only RGB bands are available or comparable, SAM should be interpreted as a spectral-angle proxy rather than a full multispectral or hyperspectral spectral-fidelity evaluation.

---

Data Organization

The benchmark data are stored under:

Data/Inputs/

A typical structure is:

Data/Inputs/
├── T2I/
│   ├── prompts/
│   ├── references/
│   └── metadata/
├── M2I/
│   ├── masks/
│   ├── references/
│   └── metadata/
└── SR/
    ├── low_resolution/
    ├── references/
    └── metadata/

Generated outputs should be saved under:

Data/Results/

For example:

Data/Results/
├── T2I/
│   ├── CRS-Diff/
│   ├── DiffusionSat/
│   ├── Flux.1/
│   └── Text2Earth/
├── M2I/
│   └── <model_name>/
└── SR/
    └── <model_name>/

Please keep file names aligned with the corresponding input or reference images so that metric scripts can correctly match generated images with references.

---

Metric Scripts

This repository provides three task-specific metric scripts.

metrics_utils_t2i.py
metrics_utils_m2i.py
metrics_utils_sr.py

Run the help command to check supported arguments:

python metrics_utils_t2i.py --help
python metrics_utils_m2i.py --help
python metrics_utils_sr.py --help

Metric Summary

Metric	Full Name	Type	Reference Required	Better
FID	Fréchet Inception Distance	Distribution-level realism	Real image set	Lower
CLIPScore	CLIP-based image-text alignment	Text-image semantic alignment	Text prompt	Higher
Semantic Consistency	CLIP-based class semantic consistency	Mask-derived class-text alignment	Mask-derived text	Higher
BLIP Similarity	BLIP image-image feature similarity	Reference-based semantic similarity	Reference image	Higher
PSNR	Peak Signal-to-Noise Ratio	Pixel-level reconstruction fidelity	Reference image	Higher
SSIM	Structural Similarity Index Measure	Structural similarity	Reference image	Higher
LPIPS	Learned Perceptual Image Patch Similarity	Perceptual distance	Reference image	Lower
NIQE	Naturalness Image Quality Evaluator	No-reference visual quality	No	Lower
BRISQUE	Blind/Referenceless Image Spatial Quality Evaluator	No-reference visual quality	No	Lower
SAM	Spectral Angle Mapper	Spectral-angle consistency	Reference image	Lower

---

Recommended Reproduction Protocol

For each model:

Step 1: Install the model environment from the original GitHub repository.
Step 2: Download the official checkpoint.
Step 3: Prepare benchmark inputs from Data/Inputs/<task>/.
Step 4: Run model inference.
Step 5: Save generated results to Data/Results/<task>/<model_name>/.
Step 6: Run the corresponding metric script.
Step 7: Report the mean and standard deviation of each metric.

For example:

# Text-to-image evaluation
python metrics_utils_t2i.py \
    --real_dir Data/Inputs/T2I/references \
    --gen_dir Data/Results/T2I/Text2Earth \
    --prompt_file Data/Inputs/T2I/prompts/prompts.txt

# Mask/layout-to-image evaluation
python metrics_utils_m2i.py \
    --real_dir Data/Inputs/M2I/references \
    --gen_dir Data/Results/M2I/<model_name> \
    --mask_dir Data/Inputs/M2I/masks

# Super-resolution evaluation
python metrics_utils_sr.py \
    --real_dir Data/Inputs/SR/references \
    --gen_dir Data/Results/SR/<model_name> \
    --lr_dir Data/Inputs/SR/low_resolution

The exact argument names may depend on the released script version. Please check:

python metrics_utils_<task>.py --help

---

Notes on Fair Comparison

The benchmark is intended as a diagnostic comparison, not as a definitive leaderboard.

Different models may have different:

• input requirements,
• checkpoint availability,
• training data,
• API access,
• supported modalities,
• supported spatial resolutions,
• prompt sensitivity,
• generation randomness.

Therefore, metric results should be interpreted together with qualitative examples and task-specific limitations.

---

Citation

If you use this repository, please cite:

@article{wang2026generative,
  title   = {Generative AI in Remote Sensing: A Unified Perspective from Tasks to Foundation Models},
  author  = {Wang, Sirui and He, Jiang and Blasco Andreo, Natalia and Xiong, Zhitong and Zhu, Xiao Xiang},
  journal = {arxiv},
  year    = {2026}
}

---

Acknowledgements

We thank the authors of the original models, datasets, and open-source libraries used in this benchmark. Please also cite the original papers and repositories of any model you evaluate with this benchmark.

---

Contact

For questions, please open an issue or contact the authors. Sirui Wang
Email: sirui.wang@tum.de Xiao Xiang Zhu
Email: xiaoxiang.zhu@tum.de