Autonomous Drone Target Tracking in Webots

A Mavic 2 Pro drone follows a moving vehicle using only its onboard camera and depth sensor. The system combines YOLO object detection, Kalman state estimation, prediction-based gating, mission planning, and cascaded PID flight control, with the flight controller written from scratch because Webots accepts propeller speeds rather than velocity commands.

The simulator provides the true vehicle position only for evaluation and for aiming the camera on the first frame. Ground truth is never used by the tracking or control path.

---

Repository Layout

README.md
requirements.txt
src/
    drone_tracker.py        Flight controller: YOLO, Kalman filtering, gating,
                            planning, PID control, safety checks, telemetry
    geometry.py             Conversions between image, camera, body, and world
                            coordinate frames
tests/
    test_drone_tracker.py   Unit tests for the Kalman filter, PID controller,
                            mission states, safety rules, and transformations
evaluation/
    evaluate.py             Turns the flight logs into tracking, detection,
                            runtime, and failure-analysis plots
webots/
    car_mover.py            Supervisor that drives the target vehicle in a
                            circle, so the scenario is exactly repeatable
results/
    eval_summary.csv        Summary statistics for the published run
    main_plots/             The four figures referenced below
    additional_plots/       Trajectory, confidence, centring, and runtime plots
    raw/
        flight_log.csv.gz   One telemetry row per simulation step
        detections.csv      Every YOLO box offered, before gating

---

System Overview

The tracker achieves autonomous behaviour through six connected layers:

Camera and depth sensors
        ↓
YOLO object detection
        ↓
Prediction-based gating
        ↓
World-frame Kalman filtering
        ↓
Target-relative planning
        ↓
Cascaded PID control
        ↓
Propeller commands

A. YOLO Object Detection

A lightweight YOLOv8n model detects vehicles in the onboard camera image. The detector is restricted to the COCO car and truck classes.

To reduce computational cost, YOLO runs once every four simulation steps. During the intermediate steps, the Kalman filter predicts the vehicle's movement.

Each detection provides:

Bounding box:  (x, y, width, height)
Confidence:    detector confidence score
Class:         car or truck

The centre of an accepted bounding box is combined with the depth image to obtain a three-dimensional vehicle measurement:

pixel position + depth + drone pose
                    ↓
vehicle position (x, y, z) in world coordinates

B. Linear Kalman State Estimation

A Linear Kalman Filter estimates the vehicle state:

$$ \mathbf{x} = [x,\ y,\ z,\ v_x,\ v_y,\ v_z]^T $$

The filter uses a constant-velocity motion model:

$$ \mathbf{x}_{k|k-1} = F\mathbf{x}_{k-1|k-1} $$

This produces a new vehicle position and velocity prediction every simulation step, including the steps where YOLO does not run.

When a measurement $\mathbf{z}$ is accepted, the Kalman correction is:

$$ K = PH^T(HPH^T + R)^{-1} $$

$$ \mathbf{x}_{k|k} = \mathbf{x}_{k|k-1} + K(\mathbf{z}-H\mathbf{x}_{k|k-1}) $$

The difference between the measured vehicle position and the predicted position is the innovation. The Kalman gain $K$ decides how strongly the measurement should correct the estimate.

This allows the drone to coast through short detector gaps while smoothing noisy measurements.

C. Prediction-Based Gating

YOLO occasionally identifies scenery or other objects as vehicles. A prediction-based gate prevents these detections from changing the tracked target.

1. The Kalman filter predicts the vehicle's latest world position.
2. The world prediction is projected into the camera image.
3. YOLO produces possible car and truck bounding boxes.
4. Only boxes sufficiently close to the predicted pixel position are considered.
5. The closest box inside the gate is selected.
6. Its pixel centre and depth are converted into a world measurement.
7. A second world-space test rejects measurements more than 10 metres from the Kalman prediction.
8. Accepted measurements correct the Kalman estimate.

The image-space gate changes size using the filter's doubt, which measures its position uncertainty:

Low doubt  → small gate → harder to accept the wrong object
High doubt → wider gate → easier to recover after missed detections

This creates a balance between avoiding false detections and allowing target reacquisition.

D. Target-Relative Planning

The drone does not fly directly toward the estimated vehicle position. Instead, the planner creates a target point:

• 12 metres behind the vehicle
• 8 metres above the vehicle
• Facing toward the vehicle
• With the camera gimbal pointed toward the estimated target

The desired position is recalculated every simulation step using the latest Kalman estimate.

When the target is temporarily lost, the system progresses through a mission state machine:

STARTING → FOLLOWING → GUESSING_NEXT_POS → SEARCHING → GOING_HOME

This separates normal tracking, short-term prediction, active searching, and mission termination.

E. Cascaded PID Flight Control

Webots accepts individual propeller speeds rather than a direct drone velocity command. The controller therefore uses three connected control layers:

Desired world position
        ↓
Desired world velocity
        ↓
Desired roll and pitch
        ↓
Four propeller speeds

The position controller uses PID feedback:

$$ u(t) = K_p e(t) + K_i \int e(t) ; dt + K_d \frac{de(t)}{dt} $$

where the position error is

$$ \mathbf{e} = \mathbf{x}_{goal} - \mathbf{x}_{drone} $$

• $P$ reacts to the current tracking error.
• $I$ corrects persistent steady-state error.
• $D$ dampens rapid changes and oscillation.

The estimated vehicle velocity is also added as a feed-forward term, reducing the steady lag that occurs when following a continuously moving target.

---

Mission Safety

Every motor command passes through a dedicated safety layer before reaching the propellers. The safety system checks for:

• Invalid or non-finite motor commands.
• Flight below the minimum permitted altitude.
• Flight above the maximum permitted altitude.
• Movement outside the allowed geofence.
• Motor speeds exceeding physical limits.

Safety is separated from the PID controller so that no command can reach the motors without an independent validation step.

---

Reproducibility

The system is not deterministic, and the measured spread is reported here rather than hidden.

The controller is closed-loop: detections steer the drone, and the drone's position determines what the detector sees next. A single detection gained or lost early in a run changes the trajectory, which changes the viewing angle, which changes every detection afterwards. Small numerical differences therefore do not stay small.

Two runs at identical settings (DETECT_CONF = 0.01, everything else unchanged):

Metric	Run A	Run B
Median Kalman position error	1.70 m	1.70 m
Longest gap without an accepted measurement	15.6 s	16.0 s
Position error at end of that gap	42.5 m	44.7 m
Median YOLO inference time	49 ms	50 ms
Boxes offered by the detector	1332	1154
Gate precision	100%	100%
Gate recall	99.0%	99.0%

Headline tracking metrics vary by roughly 3-5% between runs. Gating metrics do not vary at all. The qualitative conclusions below reproduce; the exact figures should be read as samples, not constants.

Single-number results in this README come from one run each and are labelled with their sample size. A proper campaign would use at least three runs per configuration and report the range.

---

Quantitative Evaluation

Each run is an 79-84 second Webots simulation in which the target vehicle travels a circular path at approximately 3.5 m/s.

Detector:              YOLOv8n
Detector frequency:    once every 4 simulation steps
Kalman model:          linear constant velocity
Target classes:        car and truck
Camera:                400 x 240

Metric	DETECT_CONF 0.06 (n=1)	DETECT_CONF 0.01 (n=2)
Median Kalman position error	1.73 m	1.70 m
95th-percentile Kalman error	—	28.5 m
Median raw measurement error	1.65 m	1.65 m
Gate precision	100%	100%
Gate recall	99.0%	99.0%
False acceptance rate	0%	0%
Median YOLO inference time	51.7 ms	49-50 ms
Longest gap without a measurement	20.7 s	15.6-16.0 s
Median horizontal station-keeping error	19.8 m	17.9 m
Median height-offset error	27.0 m	25.8 m

The two configurations are not a controlled comparison: one run against two is not enough to attribute the difference to the threshold rather than to run-to-run variation, which the section above shows is of a similar size.

Gating Performance

The detector produced low-confidence false candidates, but the prediction-based gate prevented them from changing the tracked target. Counted directly from detections.csv for one run: of 1154 boxes offered, 1111 were on the true vehicle and 43 were not. The gate accepted 1100 of the 1111 and none of the 43.

• 100% gate precision: every accepted detection was on the true vehicle.
• 99.0% gate recall: almost every true vehicle detection offered by YOLO passed the gate.
• 0% false acceptance: no detection labelled as another object was accepted.

Gating protected the Kalman estimate without being responsible for the final tracking loss.

A note on the confidence-threshold plot in results/additional_plots/: it shows the split that best separates true detections from junk, and that split is unstable across runs (0.07 in one, 0.29 in another). The reason is the sample — the separation is being fitted to roughly 40-60 negative examples. The plot is retained to document how DETECT_CONF was chosen, not as a result.

---

Failure Analysis: Overhead Viewing Angle

The vehicle remained visible in the camera during the main failure period, but YOLO detection performance decreased sharply as the drone's viewing angle became more overhead.

Detector recall by viewing angle, for the two repeated runs, with the number of simulation steps in each bin:

Viewing angle below horizontal	Run A	Run B
10-20°	65% (n=17)	38% (n=24)
20-30°	77% (n=622)	68% (n=615)
30-40°	85% (n=233)	86% (n=240)
40-50°	92% (n=135)	95% (n=138)
50-60°	52% (n=143)	26% (n=303)
60-70°	17% (n=853)	18% (n=632)
70-80°	12% (n=263)	7% (n=263)

Recall peaks between 30 and 50 degrees and collapses above 50 degrees. The well-populated bins agree closely between runs; the 10-20° bin has fewer than 25 samples and should not be read as a measurement.

The drone failed to maintain the requested eight-metre height offset and gradually climbed to more than 50 metres above the vehicle. This produced the following failure chain:

Altitude tracking error
        ↓
Viewing angle becomes increasingly overhead
        ↓
Vehicle appearance differs from YOLO's strongest training views
        ↓
YOLO offers fewer true vehicle detections
        ↓
No accepted measurements for 16-21 seconds
        ↓
Kalman filter continues using constant velocity
        ↓
Vehicle follows a curved path while the estimate travels approximately straight
        ↓
Position error grows from about 1.5 m to 42-70 m

The results show that:

• Gating was not the cause of the loss. It accepted no false detection in any run.
• The Kalman estimate remained accurate while receiving measurements.
• YOLO recall fell strongly at steep viewing angles.
• The constant-velocity assumption broke down only after a prolonged measurement gap.
• The altitude controller was the upstream weakness that pushed perception outside its reliable operating range.

This failure analysis is retained rather than hidden, because it demonstrates how perception, estimation, and control failures propagate through an autonomous system.

---

Main Evaluation Plots

1. Kalman Error During Detection Loss

The estimate stays within roughly 1-2 metres while measurements arrive. During the final gap the estimate diverges, because the vehicle continues around a curved path while a constant-velocity model travels straight. The vertical axis is logarithmic so that both the sub-metre working range and the final divergence are readable on one plot.

2. YOLO Recall Against Viewing Angle

Detection performance peaks between 30 and 50 degrees and falls sharply once the view becomes overhead.

3. Detector and Gate Events During Failure

The gate accepts nearly every true vehicle detection offered during the 55-70 second interval. The final loss begins when YOLO stops offering detections, rather than when the gate rejects them. This is the evidence that separates a perception failure from an association failure.

4. Following Distance and Height

Height above the vehicle climbs steadily past the requested 8 m to more than 50 m. This is the control error that drove the viewing angle out of the detector's reliable range, and it is the upstream cause of everything in the failure chain above.

Trajectory, confidence-versus-angle, confidence-threshold, centring, and runtime-cost plots are in results/additional_plots/. Median YOLO inference cost was about 50 ms per detection step, and the simulation ran at roughly 0.25x real time.

---

Installation

pip install -r requirements.txt

opencv-contrib-python specifically, not opencv-python. Webots must be installed separately and configured to use the Python environment containing these packages.

Running the Simulation

src/drone_tracker.py is the Webots controller for the drone and webots/car_mover.py is the supervisor that moves the target vehicle. Point each Webots controller directory at these files, then run the world. The tracker writes flight_log.csv and detections.csv into the controller directory as it flies.

Running the Tests

PYTHONPATH=src python -m unittest discover -s tests

Reproducing the Evaluation

python evaluation/evaluate.py results/raw/flight_log.csv.gz

The evaluation script compares:

• True vehicle position against the Kalman estimate.
• Raw measurements against simulator ground truth.
• Accepted and rejected detections.
• Detector confidence against viewing angle.
• Gate precision and recall.
• Vehicle position inside the camera frame.
• Following distance and height.
• Detector latency and simulation speed.

Ground truth is used only inside the evaluation pipeline, never as an input to the flight controller.

---

Known Limitations

• Altitude hold is the main defect. The drone climbs steadily instead of holding the requested 8 m offset, which is what eventually breaks perception.
• The motion model is constant-velocity. The target drives a circle, so the model is wrong by construction and only survives because measurements arrive often enough to correct it. A CTRV model with an extended or unscented filter would coast through gaps far better.
• Results vary between runs, as documented above. Figures quoted from a single run should be treated accordingly.
• The detector is stock COCO YOLOv8n, never fine-tuned on overhead views of this vehicle. The supervisor already knows the true vehicle position, so auto-labelled training frames are available at no cost.

---

Technical Skills Demonstrated

• Closed-loop autonomous system design.
• Real-time object detection using YOLO.
• Multi-rate perception and control.
• Linear Kalman filtering in world coordinates.
• Prediction-based data association and gating.
• Camera, body, and world coordinate transformations.
• Depth-based three-dimensional localisation.
• Cascaded PID flight control.
• State-machine mission supervision.
• Safety and command validation.
• Unit testing of control and estimation components.
• Telemetry design and quantitative failure analysis.
• Experiment design, reproducibility measurement, and honest reporting of variance.
• Performance profiling and experiment-driven debugging.