Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities

Abstract

Foundation models are revolutionizing autonomous driving perception, transitioning the field from narrow, task-specific deep learning models to versatile, general-purpose architectures trained on vast, diverse datasets. This survey examines how these models address critical challenges in autonomous perception, including limitations in generalization, scalability, and robustness to distributional shifts. The survey introduces a novel taxonomy structured around four essential capabilities for robust performance in dynamic driving environments: generalized knowledge, spatial understanding, multi-sensor robustness, and temporal reasoning. For each capability, the survey elucidates its significance and comprehensively reviews cutting-edge approaches. Diverging from traditional method-centric surveys, our unique framework prioritizes conceptual design principles, providing a capability-driven guide for model development and clearer insights into foundational aspects. We conclude by discussing key challenges, particularly those associated with the integration of these capabilities into real-time, scalable systems, and broader deployment challenges related to computational demands and ensuring model reliability against issues like hallucinations and out-of-distribution failures. The survey also outlines crucial future research directions to enable the safe and effective deployment of foundation models in autonomous driving systems.

arXiv comment32 pages, 14 figures, accepted at IEEE Open Journal of Vehicular Technology (OJVT)

References in corpus (10)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Knowledge Distillation: A Survey
• Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• 3D Object Detection for Autonomous Driving: A Survey
• Masked Autoencoder for Self-Supervised Pre-training on Lidar Point Clouds
• A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective
• SAM3D: Zero-Shot 3D Object Detection via Segment Anything Model
• HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction
• AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving