Argus: Leveraging Multiview Images for Improved 3-D Scene Understanding With Large Language Models

Abstract

Advancements in foundation models have made it possible to conduct applications in various downstream tasks. Especially, the new era has witnessed a remarkable capability to extend Large Language Models (LLMs) for tackling tasks of 3D scene understanding. Current methods rely heavily on 3D point clouds, but the 3D point cloud reconstruction of an indoor scene often results in information loss. Some textureless planes or repetitive patterns are prone to omission and manifest as voids within the reconstructed 3D point clouds. Besides, objects with complex structures tend to introduce distortion of details caused by misalignments between the captured images and the dense reconstructed point clouds. 2D multi-view images present visual consistency with 3D point clouds and provide more detailed representations of scene components, which can naturally compensate for these deficiencies. Based on these insights, we propose Argus, a novel 3D multimodal framework that leverages multi-view images for enhanced 3D scene understanding with LLMs. In general, Argus can be treated as a 3D Large Multimodal Foundation Model (3D-LMM) since it takes various modalities as input(text instructions, 2D multi-view images, and 3D point clouds) and expands the capability of LLMs to tackle 3D tasks. Argus involves fusing and integrating multi-view images and camera poses into view-as-scene features, which interact with the 3D features to create comprehensive and detailed 3D-aware scene embeddings. Our approach compensates for the information loss while reconstructing 3D point clouds and helps LLMs better understand the 3D world. Extensive experiments demonstrate that our method outperforms existing 3D-LMMs in various downstream tasks.

Accepted by TNNLS2025

References in corpus (11)

• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Scaling Instruction-Finetuned Language Models
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Kosmos-2: Grounding Multimodal Large Language Models to the World
• LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
• MultiModal-GPT: A Vision and Language Model for Dialogue with Humans
• 3D-LLM: Injecting the 3D World into Large Language Models
• LAVIS: A Library for Language-Vision Intelligence
• SceneGPT: A Language Model for 3D Scene Understanding