🚗 SAFE: Spatial-temporal Adaptation in Federated Environments for CAV Accident Anticipation under Heterogeneity

Official codebase for the thesis: SAFE: Spatial-temporal Adaptation in Federated Environments for CAV Accident Anticipation under Heterogeneity

To achieve zero-fatality Intelligent Transportation Systems (ITS), automotive safety mechanisms must transition from reactive mitigation (e.g., airbags, AEB) to proactive Traffic Accident Anticipation (TAA).

SAFE is a privacy-preserving, communication-efficient, and highly personalized federated learning framework designed to deploy state-of-the-art spatial-temporal video foundation models (VideoMAEv2) across Connected and Autonomous Vehicles (CAVs).

🏆 Key Contributions & The "Trilemma"

Deploying massive spatial-temporal video models on CAVs presents a critical trilemma. SAFE resolves all three deployment barriers:

1. Data Privacy (GDPR Compliance): Instead of centralizing highly sensitive dashcam video, SAFE utilizes Federated Learning (FL) to retain all raw data on the edge. Only mathematical gradients are shared.
2. V2I Communication Bottleneck: Transmitting full VideoMAEv2 weights requires >300 MB per round, crippling vehicular networks. By freezing the backbone and injecting Low-Rank Adaptation (LoRA) modules, SAFE drastically compresses transmission payloads down to ~19 MB.
3. Environmental Heterogeneity (Non-IID Data): Standard FL suffers from catastrophic weight divergence across extreme edge-cases (e.g., heavy snow vs. clear urban nights). SAFE employs Personalized Federated Learning anchored by a proximal constraint (FedProx), empowering dynamic local adaptation without catastrophic forgetting.

📊 Performance Highlights

SAFE balances universal driving knowledge with extreme local domain mastery. Notably, at the critical 1.5-second Time-to-Accident (TTA) threshold—the essential window for physical evasive maneuvers—the personalized edge models significantly outperform generalized baselines.

Evaluation Subset	Standard Global FL (mAP)	SAFE Personalized (mAP)	Early TTA (1.5s) AP Gain
Client 1 (Standard Urban)	0.6436	0.6474	0.5098 ➔ 0.5169
Client 2 (Diverse Non-Urban)	0.6475	0.6579	0.4715 ➔ 0.4831
Client 3 (Adverse Weather)	0.6237	0.6271	0.4218 ➔ 0.4266
Client 4 (Extreme Lighting)	0.8550	0.8615	0.6667 ➔ 0.7342

SAFE retains general intelligence: Evaluating the extreme-specialized Client 4 model on the unified global dataset actually yielded a higher mAP (0.6386) than the global baseline (0.6338).

📁 Repository Structure

├── configs/                  # YAML experiment configurations (centralized, global, personalized, ablations)
├── dataloaders/              # Nexar dataset loaders and Non-IID hierarchical client partitioners
├── fl_core/                  # Core Federated Learning logic (Server aggregation, Client training)
├── models/                   # Frozen VideoMAEv2, LoRA injection logic, and custom MLP classifier
├── scripts/                  # Ready-to-use SLURM bash scripts for massive multi-GPU training
├── utils/                    # Safety metrics (mAP @ TTA), Focal Loss, and Mixup augmentation
├── viz/                      # Visualization scripts for generating thesis plots (ablation/evaluation)
├── preprocess_videos.py      # Extracts 16-frame dense temporal windows from raw 2s video clips
├── train_centralized.py      # Experiment 1: Centralized Baseline (Upper Bound)
├── train_federated.py        # Experiment 2: Standard Global FL (FedAvg)
├── train_personalized.py     # Experiment 3: SAFE Framework (FedProx Personalization)
└── evaluate.py               # Unified evaluation script for TTA mAP metrics

🚀 Getting Started

1. Environment Setup

Create a Conda environment and install the required dependencies:

conda create -n safe python=3.10 -y
conda activate safe
pip install torch torchvision
pip install transformers peft pandas scikit-learn pyyaml opencv-python matplotlib seaborn

2. Data Preparation

This framework relies on dashcam collision datasets (e.g., Nexar). Ensure your raw videos and master_metadata.csv are placed in data/raw/.

Run the preprocessing script to extract the critical 2.0-second spatial-temporal windows (16 frames) prior to the collisions:

python preprocess_videos.py
# Or via SLURM: sbatch scripts/slurm_preprocess.sh

3. Model Weights

Download the pre-trained VideoMAEv2-base weights from HuggingFace/OpenGVLab and place them in the ./checkpoints/pretrained/videomaev2-base/ directory.

🔬 Running the Experiments

The research is divided into three distinct experimental paradigms. You can run the Python scripts directly or use the provided SLURM scripts for cluster execution.

Experiment 1: Centralized Baseline (Theoretical Upper Bound)

Pools all data into a single node, bypassing privacy restrictions to establish the performance ceiling.

sbatch scripts/slurm_centralized.sh
# To evaluate: sbatch scripts/slurm_centralized_evaluate.sh

Experiment 2: Standard Global FL

Simulates strict decentralized privacy and communication constraints (~19MB LoRA payloads) using standard FedAvg.

sbatch scripts/slurm_federated_global.sh
# To evaluate: sbatch scripts/slurm_federated_global_evaluate.sh

Experiment 3: SAFE Framework (Personalized)

The core contribution. Loads the converged global model and executes constrained Federated Parameter-Efficient Fine-Tuning (FedPEFT) using FedProx to master local domains without catastrophic forgetting.

sbatch scripts/slurm_federated_personalized.sh
# To evaluate across all clients: sbatch scripts/slurm_federated_personalized_evaluate.sh

Ablation Studies

To reproduce the architectural and hyperparameter ablation studies (LoRA rank sensitivity and target module payload vs. performance trade-offs):

sbatch scripts/slurm_ablation_lora_arc.sh
sbatch scripts/slurm_ablation_lora_ranks.sh

📈 Visualizing Results

The repository includes scripts to generate professional, thesis-ready visualizations (Convergence curves, TTA degradation, Domain mastery, and Ablation trade-offs).

python viz/viz_training.py
python viz/viz_evaluation.py
python viz/viz_ablation.py

Generated plots will be saved directly to the viz/figures/ directory.

📝 Citation

If you find this code or research helpful for your own work, please consider citing:

@thesis{hok2026safe,
  title={SAFE: Spatial-temporal Adaptation in Federated Environments for CAV Accident Anticipation under Heterogeneity},
  author={Hok, Layheng},
  year={2026},
  type={Bachelor's Thesis},
  school={Southern University of Science and Technology (SUSTech)},
  url={https://github.com/Layheng-Hok/SAFE}
}

🛡️ License

This project is licensed under the MIT License - see the LICENSE file for details.