CommandLM: Data-Driven Behavior Level Descriptor for Ego Vehicles

CommandLM is a multimodal large language model that generates concise, human-readable behavior descriptions for ego vehicles from fused multi-sensor data. This work enables interpretable, behavior-level decision-making essential for safety, trust, and regulation in autonomous driving systems.

Paper: CommandLM: Data-Driven Behavior-Level Descriptor for Ego Vehicles — IEEE Intelligent Vehicles Symposium (IV) 2026. arXiv link coming soon.

Overview

CommandLM processes temporally fused bird's-eye view (BEV) representations from LiDAR and multi-camera inputs via a Q-Former adapter connected to a quantized, LoRA-fine-tuned large language model. The system produces intent-aware, interpretable captions suitable for planner supervision and safety auditing.

Key Features

• Temporal BEV Fusion: Processes up to 2 seconds of synchronized multi-sensor data for coherent temporal behavior reasoning
• Q-Former Compression: Efficiently bridges high-dimensional visual features to the language domain
• LoRA-Fine-Tuned LLM: Uses Qwen-2.5 14B with 8-bit quantization for efficient inference on consumer GPUs
• Object-Centric Grounding: Replaces pixel-coordinate references with semantic object references for improved interpretability
• Linguistic Diversity: Dataset augmentation generates 40,000+ diverse behavior-focused question-answer pairs

Architecture

CommandLM follows a three-stage multimodal stack:

1. Sensor Fusion (BEVFusion): Fuses six RGB cameras and LiDAR into unified BEV representation
2. Temporal Encoding: Stacks BEV tensors over 2 seconds (24 frames @ 12 Hz) for temporal context
3. Q-Former Compression: Compresses temporal BEV sequence into 32 learnable query tokens
4. LLM Integration: Vision-language interface bridges Q-Former outputs to Qwen-2.5 14B LLM

Results

CommandLM is compared against a BLIP-2 baseline that shares the same Q-Former foundation but has no temporal BEV fusion, which isolates the gain from temporal fusion + behavior-focused data. Higher is better.

Metric	BLIP-2	CommandLM
METEOR	0.10	0.38
CIDEr	0.07	0.67
SPICE	0.07	0.13
BERT-F1	0.86	0.88
Behavior Exploration Rate (valid, rule-compliant)	41.6%	58.0%
Ground-Truth Matching (matches human driving)	34.5%	47.1%

In human evaluation, 58% of generated behaviors were rated valid and rule-compliant. Average inference is 1.78 s, and the full pipeline fits in a 24 GB GPU.

Installation & Setup

Dependencies

install mmcv following: https://github.com/open-mmlab/mmcv?tab=readme-ov-file install bevfusion following: https://github.com/open-mmlab/mmdetection3d/tree/main/projects/BEVFusion

pip install -r requirements.txt

mount nuscenes dataset to data/nuscenes

Weights and Dataset

dataset and model will be published shortly

Usage

The CommandLM-nuScenes QA dataset and trained weights will be released (see above). With those in place you can train or run the model directly — building the dataset yourself is optional, only needed to regenerate it from DriveLM-nuScenes.

Train — scripts/train_single.py (single GPU) or scripts/train.py (multi-GPU via torchrun). Only the adapters, Q-Former, and LoRA weights are updated; BEVFusion stays frozen:

python scripts/train_single.py \
  --dataset_path data/nuscenes \
  --version v1.0-trainval \
  --frames 24 \
  --model <path or name of Qwen2.5-14B> \
  --num_epochs 1 \
  --batch_size 10 \
  --use_bev_fusion

Run python scripts/train.py --help for the full option list (LoRA resume --peft_model, context-aware training, and more).

Evaluate — open scripts/evaluation.ipynb to generate behavior captions and compute the reported metrics.

(Optional) Regenerate the dataset — the scripts in scripts/data_preparation/ rebuild CommandLM-nuScenes from DriveLM-nuScenes (sanitize tags → scene summaries → behavior QA via Qwen-2.5 → preprocess BEV).

Citation

If you use this work, please cite:

@inproceedings{tokic2026commandlm,
  title     = {CommandLM: Data-Driven Behavior-Level Descriptor for Ego Vehicles},
  author    = {Tokic, Boris and Selzer, Constantin and Flohr, Fabian B.},
  booktitle = {Proceedings of the IEEE Intelligent Vehicles Symposium (IV)},
  year      = {2026}
}

License

• Code in this repository is released under the MIT License, © 2025 Boris Tokic.
• Dataset (CommandLM-nuScenes) and released annotations, when published, are licensed under CC BY-NC-SA 4.0 (non-commercial). They are derived from nuScenes and DriveLM (both CC BY-NC-SA 4.0) and inherit those terms; they are not covered by the MIT License.
• The associated paper is © 2026 IEEE.