SituationalLLM: Proactive language models with scene awareness for dynamic, contextual task guidance
arXiv:2406.13302 · doi:10.12688/openreseurope.18551.1
Abstract
Large language models (LLMs) have achieved remarkable success in text-based tasks but often struggle to provide actionable guidance in real-world physical environments. This is because of their inability to recognize their limited understanding of the user's physical context. We present SituationalLLM, a novel approach that integrates structured scene information into an LLM to deliver proactive, context-aware assistance. By encoding objects, attributes, and relationships in a custom Scene Graph Language, SituationalLLM actively identifies gaps in environmental context and seeks clarifications during user interactions. This behavior emerges from training on the Situational Awareness Database for Instruct-Tuning (SAD-Instruct), which combines diverse, scenario-specific scene graphs with iterative, dialogue-based refinements. Experimental results indicate that SituationalLLM outperforms generic LLM baselines in task specificity, reliability, and adaptability, paving the way for environment-aware AI assistants capable of delivering robust, user-centric guidance under real-world constraints.
arXiv commentRevised Submission to Open Research Europe
References in corpus (15)
- Decoupled Weight Decay Regularization
- Learning Transferable Visual Models From Natural Language Supervision
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- LoRA: Low-Rank Adaptation of Large Language Models
- A Survey on Large Language Model based Autonomous Agents
- Visual Instruction Tuning
- Reflexion: Language Agents with Verbal Reinforcement Learning
- A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models
- Scene Graph Generation from Objects, Phrases and Region Captions
- RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models
- Multi-modal Situated Reasoning in 3D Scenes
- Can LLMs plan paths in the real world?
- See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses
- Human Pose Descriptions and Subject-Focused Attention for Improved Zero-Shot Transfer in Human-Centric Classification Tasks
- EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents