Abstract

Autonomous vehicle perception systems must not only classify objects correctly but also report when they do not know. A system that is confidently wrong is more dangerous than one that escalates ambiguous inputs to a planner that can slow down or request human attention. Uncertainty quantification in AV perception has matured from academic curiosity to engineering requirement, with production teams at Waymo, Motional, and others now treating calibrated uncertainty as a first-class output alongside bounding boxes and segmentation masks.

Sources of Uncertainty in Perception

AV perception uncertainty divides usefully into aleatoric (irreducible, from sensor noise and occlusion) and epistemic (reducible, from model under-training on rare inputs). The split matters for system design: aleatoric uncertainty should feed into object state covariances passed to the tracker and planner, while high epistemic uncertainty signals distribution shift and should trigger fallback behaviors or data collection flags. In practice, most deployed models conflate the two because estimating epistemic uncertainty at inference time is expensive. Ensembles of 5-10 models remain the most reliable approach, with recent work from Aurora and academic groups showing that deep ensembles outperform Monte Carlo dropout and single-model conformal prediction in the tail of the distribution.

Conformal Prediction as a Calibration Layer

Conformal prediction has attracted significant interest in AV perception because it provides distribution-free coverage guarantees without retraining. Applied as a post-hoc wrapper over a base detector, it converts raw confidence scores into prediction sets guaranteed to contain the true label with user-specified probability on held-out calibration data. The limitation is exchangeability: calibration data must be drawn from the same distribution as deployment, which is a strong assumption for geographically expanding AV programs. Teams at Wayve and in the ETH Zurich AV group have published domain-adaptive conformal methods that partially relax this requirement by conditioning coverage on covariate-matched subsets.

Propagating Uncertainty Downstream

Uncertainty estimates are only useful if downstream modules consume them. Most publicly documented AV stacks pass point estimates from perception to prediction and planning, discarding uncertainty. Propagating full distributions through a multi-object tracker (e.g., using an unscented Kalman filter or a particle filter with likelihood weighting) increases computational cost but allows planners to generate risk-aware trajectories. Baidu Apollo’s open-source stack and the nuPlan benchmark both provide hooks for uncertainty-aware object state representations, giving researchers a standard interface to evaluate end-to-end gains from better-calibrated perception.

Validation at Scale

Validating uncertainty calibration requires large held-out datasets with ground truth labels and is complicated by the rarity of the hardest cases. Simulation-in-the-loop testing, using photorealistic renderers like CARLA extensions or Waymo’s internal sim, can stress-test uncertainty estimates against injected adversarial conditions. The field lacks standardized calibration metrics for detection tasks, with Expected Calibration Error adapted from classification not cleanly generalizing to bounding-box regression. Closing this metric gap is an active area of collaboration between AV safety teams and academic reliability research groups.