UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
arXiv:2501.02341 · doi:10.1016/j.inffus.2025.103158
Abstract
Low-altitude mobility, exemplified by unmanned aerial vehicles (UAVs), has introduced transformative advancements across various domains, like transportation, logistics, and agriculture. Leveraging flexible perspectives and rapid maneuverability, UAVs extend traditional systems' perception and action capabilities, garnering widespread attention from academia and industry. However, current UAV operations primarily depend on human control, with only limited autonomy in simple scenarios, and lack the intelligence and adaptability needed for more complex environments and tasks. The emergence of large language models (LLMs) demonstrates remarkable problem-solving and generalization capabilities, offering a promising pathway for advancing UAV intelligence. This paper explores the integration of LLMs and UAVs, beginning with an overview of UAV systems' fundamental components and functionalities, followed by an overview of the state-of-the-art in LLM technology. Subsequently, it systematically highlights the multimodal data resources available for UAVs, which provide critical support for training and evaluation. Furthermore, it categorizes and analyzes key tasks and application scenarios where UAVs and LLMs converge. Finally, a reference roadmap towards agentic UAVs is proposed, aiming to enable UAVs to achieve agentic intelligence through autonomous perception, memory, reasoning, and tool utilization. Related resources are available at https://github.com/Hub-Tian/UAVs_Meet_LLMs.
References in corpus (96)
- Training language models to follow instructions with human feedback
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- LLaMA: Open and Efficient Foundation Language Models
- Learning to Prompt for Vision-Language Models
- Remote Sensing Image Scene Classification: Benchmark and State of the Art
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- LoRA: Low-Rank Adaptation of Large Language Models
- Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark
- PaLM: Scaling Language Modeling with Pathways
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
- GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
- A Survey of Large Language Models
- Flamingo: a Visual Language Model for Few-Shot Learning
- Large Language Models are Zero-Shot Reasoners
- DINOv2: Learning Robust Visual Features without Supervision
- Gemini: A Family of Highly Capable Multimodal Models
- DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
- A Survey on Multimodal Large Language Models
- Visual Instruction Tuning
- Exploring Models and Data for Remote Sensing Image Caption Generation
- Safety-enhanced UAV Path Planning with Spherical Vector-based Particle Swarm Optimization
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- Multi-Task Learning with Deep Neural Networks: A Survey
- A Comprehensive Overview of Large Language Models
- PaLM-E: An Embodied Multimodal Language Model
- Constitutional AI: Harmlessness from AI Feedback
- GLM-130B: An Open Bilingual Pre-trained Model
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
- SAM 2: Segment Anything in Images and Videos
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
- A Survey on In-context Learning
- Automatic Chain of Thought Prompting in Large Language Models
- Large Language Models: A Survey
- FILIP: Fine-grained Interactive Language-Image Pre-Training
- ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
- ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth
- Segment Everything Everywhere All at Once
- Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
- Person Re-identification in Aerial Imagery
- DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
- RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Decomposed Prompting: A Modular Approach for Solving Complex Tasks
- Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
- EVA-CLIP: Improved Training Techniques for CLIP at Scale
- InsPLAD: A Dataset and Benchmark for Power Line Asset Inspection in UAV Images
- SegGPT: Segmenting Everything In Context
- A Multi-viewpoint Outdoor Dataset for Human Action Recognition
- 3D-LLM: Injecting the 3D World into Large Language Models
- MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
- Understanding the planning of LLM agents: A survey
- InternLM2 Technical Report
- Segment Any Point Cloud Sequences by Distilling Vision Foundation Models
- Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective
- A Survey on Hallucination in Large Vision-Language Models
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second
- BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
- DisasterResponseGPT: Large Language Models for Accelerated Plan of Action Development in Disaster Response Scenarios
- SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory
- When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
- TypeFly: Flying Drones with Large Language Model
- Agent as Cerebrum, Controller as Cerebellum: Implementing an Embodied LMM-based Agent on Drones
- TanDepth: Leveraging Global DEMs for Metric Monocular Depth Estimation in UAVs
- Harnessing Large Vision and Language Models in Agriculture: A Review
- A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Education
- Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design
- DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
- ScaleDepth: Decomposing Metric Depth Estimation into Scale Prediction and Relative Depth Estimation
- AeroVerse: UAV-Agent Benchmark Suite for Simulating, Pre-training, Finetuning, and Evaluating Aerospace Embodied World Models
- Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology
- Point-SAM: Promptable 3D Segmentation Model for Point Clouds
- CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentation
- LLMI3D: MLLM-based 3D Perception from a Single 2D Image
- EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment
- NEUSIS: A Compositional Neuro-Symbolic Framework for Autonomous Perception, Reasoning, and Planning in Complex UAV Search Missions
- Integrating Large Language Models for UAV Control in Simulated Environments: A Modular Interaction Approach
- Defining and Evaluating Physical Safety for Large Language Models
- VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos
- Leveraging YOLO-World and GPT-4V LMMs for Zero-Shot Person Detection and Action Recognition in Drone Imagery
- Exploring Spatial Representation to Enhance LLM Reasoning in Aerial Vision-Language Navigation
- UAV-VLA: Vision-Language-Action System for Large Scale Aerial Mission Generation
- GOMAA-Geo: GOal Modality Agnostic Active Geo-localization
- Neuro-LIFT: A Neuromorphic, LLM-based Interactive Framework for Autonomous Drone FlighT at the Edge
- NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation
- Applying Unsupervised Semantic Segmentation to High-Resolution UAV Imagery for Enhanced Road Scene Parsing
- Evaluating Voice Command Pipelines for Drone Control: From STT and LLM to Direct Classification and Siamese Networks
- EmbodiedSAM: Online Segment Any 3D Thing in Real Time
- ASMA: An Adaptive Safety Margin Algorithm for Vision-Language Drone Navigation via Scene-Aware Control Barrier Functions
- SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environments
- Cross-View Meets Diffusion: Aerial Image Synthesis with Geometry and Text Guidance
- CityNav: A Large-Scale Dataset for Real-World Aerial Navigation
Cited by in corpus (6)
- All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles
- GeoNav: Empowering MLLMs with dual-scale geospatial reasoning for language-goal aerial navigation
- LLM-Enabled In-Context Learning for Data Collection Scheduling in UAV-assisted Sensor Networks
- PINA: Prompt Injection Attack against Navigation Agents
- Talk Less, Fly Lighter: Autonomous Semantic Compression for UAV Swarm Communication via LLMs
- AirVista-II: An Agentic System for Embodied UAVs Toward Dynamic Scene Semantic Understanding