Spatial Perception

Spatial Perception

A robotics perception engineering project focused on building 3D spatial understanding using synchronized RGB cameras and LiDAR.

Built using CARLA, ROS2, OpenCV, and YOLOv8, the project progresses from camera–LiDAR integration to multi-camera perception, unified spatial understanding, and 360° environmental perception.

This project builds upon the 2D perception stack developed in the companion repository:

ROS2 Autonomous Perception Stack — 2D Perception

---

---

Technology Stack

Category	Technologies
Simulation	CARLA 0.9.15
Robotics Middleware	ROS2 Humble
Computer Vision	OpenCV
Object Detection	YOLOv8m-seg TensorRT INT8
3D Sensor	LiDAR
Programming Language	Python
Communication	CycloneDDS
Environment	Windows 11 + WSL2 Ubuntu 22.04

---

System Architecture

RGB Cameras + LiDAR
          │
          ▼
Sensor Synchronization
          │
          ▼
Multi-Modal Perception
          │
          ▼
Spatial Understanding
          │
          ▼
Unified World Representation

---

Completed Phases

🚧 Phase 7 — Sensor Fusion Foundations

Established the foundation for spatial perception by integrating LiDAR with RGB cameras, calibrating multi-sensor geometry, associating 2D detections with 3D point clouds, and preparing the perception pipeline for object-level distance estimation.

Milestones

• ✅ 7A — LiDAR Integration

  • Integrated a 32-channel LiDAR sensor with the RGB perception pipeline.
  • Published synchronized PointCloud2 data and validated real-time visualization.
  • 📁 View Phase 7A

• ✅ 7B — Camera–LiDAR Calibration

  • Calibrated RGB camera and LiDAR sensors using intrinsic and extrinsic parameters.
  • Implemented point cloud projection and verified exact timestamp synchronization.
  • 📁 View Phase 7B

• ✅ 7C — 2D–3D Association

  • Associated projected LiDAR points with YOLOv8 segmentation masks to generate object-specific point clouds.
  • Developed deterministic recording and offline replay pipelines for repeatable perception experiments.
  • 📁 View Phase 7C

• 🚧 7D — Object Distance Estimation

  • LiDAR-based object distance estimation
  • Monocular distance estimation
  • Camera–LiDAR distance fusion

---

🚧 Phase 8 — Multi-Sensor Perception

Documentation will be published after project milestone release.

📁 View Phase 8

---

🚧 Phase 9 — Unified Spatial Perception

Documentation will be published after project milestone release.

📁 View Phase 9

---

Demonstrations

Phase 7 — Sensor Fusion Foundations

7A — LiDAR Integration

Integrated a 32-channel LiDAR sensor into the ROS2 perception pipeline and validated synchronized PointCloud2 visualization in RViz2.

---

7B — Camera–LiDAR Calibration

Projected LiDAR points onto synchronized RGB images through camera calibration, coordinate transformation, and perspective projection.

---

7C — 2D–3D Association

Associated projected LiDAR points with YOLOv8 segmentation masks to generate object-specific point clouds using a deterministic offline replay pipeline.

---

---

Future Work

Planned areas of development include:

• Multi-camera perception
• 360° environment perception
• Unified world representation
• Bird's-Eye View generation
• Multi-camera overlap resolution
• Spatial perception evaluation

---

Project Journal

Detailed planning, development notes, experiments, and engineering decisions are maintained separately:

• roadmap/README.md