Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy

Abstract

With the rapid advancement of artificial intelligence and robotics, the integration of Large Language Models (LLMs) with 3D vision is emerging as a transformative approach to enhancing robotic sensing technologies. This convergence enables machines to perceive, reason and interact with complex environments through natural language and spatial understanding, bridging the gap between linguistic intelligence and spatial perception. This review provides a comprehensive analysis of state-of-the-art methodologies, applications and challenges at the intersection of LLMs and 3D vision, with a focus on next-generation robotic sensing technologies. We first introduce the foundational principles of LLMs and 3D data representations, followed by an in-depth examination of 3D sensing technologies critical for robotics. The review then explores key advancements in scene understanding, text-to-3D generation, object grounding and embodied agents, highlighting cutting-edge techniques such as zero-shot 3D segmentation, dynamic scene synthesis and language-guided manipulation. Furthermore, we discuss multimodal LLMs that integrate 3D data with touch, auditory and thermal inputs, enhancing environmental comprehension and robotic decision-making. To support future research, we catalog benchmark datasets and evaluation metrics tailored for 3D-language and vision tasks. Finally, we identify key challenges and future research directions, including adaptive model architectures, enhanced cross-modal alignment and real-time processing capabilities, which pave the way for more intelligent, context-aware and autonomous robotic sensing systems.

45 pages, 15 figures, MDPI Sensors Journal

References in corpus (17)

• Training language models to follow instructions with human feedback
• Geometric deep learning: going beyond Euclidean data
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• LAION-5B: An open large-scale dataset for training next generation image-text models
• A Survey on Multimodal Large Language Models
• Visual Instruction Tuning
• PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies
• Survey on Evaluation Methods for Dialogue Systems
• Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding
• On the Explainability of Natural Language Processing Deep Models
• Advancements in Point Cloud Data Augmentation for Deep Learning: A Survey
• ProcTHOR: Large-Scale Embodied AI Using Procedural Generation
• 3D-LLM: Injecting the 3D World into Large Language Models
• Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
• Multimodal End-to-End Learning for Autonomous Steering in Adverse Road and Weather Conditions
• Uncertainty-Aware Fusion: An Ensemble Framework for Mitigating Hallucinations in Large Language Models
• Uncertainty Quantification and Decomposition for LLM-based Recommendation