Grounded Semantic Composition for Visual Scenes

Abstract

We present a visually-grounded language understanding model based on a study of how people verbally describe objects in scenes. The emphasis of the model is on the combination of individual word meanings to produce meanings for complex referring expressions. The model has been implemented, and it is able to understand a broad range of spatial referring expressions. We describe our implementation of word level visually-grounded semantics and their embedding in a compositional parsing framework. The implemented system selects the correct referents in response to natural language expressions for a large percentage of test cases. In an analysis of the system's successes and failures we reveal how visual context influences the semantics of utterances and propose future extensions to the model that take such context into account.

References in corpus (2)

• Grounding the Lexical Semantics of Verbs in Visual Perception using Force Dynamics and Event Logic
• Ubiquitous Talker: Spoken Language Interaction with Real World Objects

Cited by in corpus (8)

• Acquiring Grounded Representations of Words with Situated Interactive Instruction
• Text to 3D Scene Generation with Rich Lexical Grounding
• Context-based Word Acquisition for Situated Dialogue in a Virtual World
• Referring to the recently seen: reference and perceptual memory in situated dialog
• "How to best say it?" : Translating Directives in Machine Language into Natural Language in the Blocks World
• "Show me the cup": Reference with Continuous Representations
• Towards an Indexical Model of Situated Language Comprehension for Cognitive Agents in Physical Worlds
• Using Soft Constraints To Learn Semantic Models Of Descriptions Of Shapes