A Joint Model of Language and Perception for Grounded Attribute Learning
arXiv:1206.6423
Abstract
As robots become more ubiquitous and capable, it becomes ever more important to enable untrained users to easily interact with them. Recently, this has led to study of the language grounding problem, where the goal is to extract representations of the meanings of natural language tied to perception and actuation in the physical world. In this paper, we present an approach for joint learning of language and perception models for grounded attribute induction. Our perception model includes attribute classifiers, for example to detect object color and shape, and the language model is based on a probabilistic categorial grammar that enables the construction of rich, compositional meaning representations. The approach is evaluated on the task of interpreting sentences that describe sets of objects in a physical workspace. We demonstrate accurate task performance and effective latent-variable concept induction in physical grounded scenes.
Appears in Proceedings of the 29th International Conference on Machine Learning (ICML 2012)
References in corpus (1)
Cited by in corpus (23)
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Visual Question Answering: A Survey of Methods and Datasets
- Towards a Visual Turing Challenge
- Text to 3D Scene Generation with Rich Lexical Grounding
- Generalized Grounding Graphs: A Probabilistic Framework for Understanding Grounded Commands
- Learning Models for Following Natural Language Directions in Unknown Environments
- Generating Multi-Sentence Lingual Descriptions of Indoor Scenes
- Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction
- Cornell SPF: Cornell Semantic Parsing Framework
- A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
- Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts
- A Deep Architecture for Semantic Parsing
- Deep Bayesian Active Learning for Multiple Correct Outputs
- Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video
- A Review of Methodologies for Natural-Language-Facilitated Human-Robot Cooperation
- Balancing Shared Autonomy with Human-Robot Communication
- Weakly-supervised Neural Semantic Parsing with a Generative Ranker
- Learning an Executable Neural Semantic Parser
- Learning Executable Semantic Parsers for Natural Language Understanding
- Unanimous Prediction for 100% Precision with Application to Learning Semantic Mappings
- Visual Reasoning with Natural Language
- Planning, Inference and Pragmatics in Sequential Language Games
- An Overview of Computational Approaches for Interpretation Analysis