Grounding the Lexical Semantics of Verbs in Visual Perception using Force Dynamics and Event Logic

Abstract

This paper presents an implemented system for recognizing the occurrence of events described by simple spatial-motion verbs in short image sequences. The semantics of these verbs is specified with event-logic expressions that describe changes in the state of force-dynamic relations between the participants of the event. An efficient finite representation is introduced for the infinite sets of intervals that occur when describing liquid and semi-liquid events. Additionally, an efficient procedure using this representation is presented for inferring occurrences of compound events, described with event-logic expressions, from occurrences of primitive events. Using force dynamics and event logic to specify the lexical semantics of events allows the system to be more robust than prior systems based on motion profile.

Cited by in corpus (16)

• Grounded Semantic Composition for Visual Scenes
• Grounded Language Learning in a Simulated 3D World
• Advances in Human Action Recognition: A Survey
• A Probabilistic Logic Programming Event Calculus
• Specific-to-General Learning for Temporal Events with Application to Learning Event Definitions from Video
• Interactive Grounded Language Acquisition and Generalization in a 2D World
• Learning to Win by Reading Manuals in a Monte-Carlo Framework
• VoxML: A Visualization Modeling Language
• Context-based Word Acquisition for Situated Dialogue in a Virtual World
• Sensorimotor Input as a Language Generalisation Tool: A Neurorobotics Model for Generation and Generalisation of Noun-Verb Combinations with Sensorimotor Inputs
• Extensible Grounding of Speech for Robot Instruction
• Robots Learning to Say `No': Prohibition and Rejective Mechanisms in Acquisition of Linguistic Negation
• Probabilistic Complex Event Recognition: A Survey
• Revisiting Active Perception
• A Review of Verbal and Non-Verbal Human-Robot Interactive Communication
• Multimodal Semantic Simulations of Linguistically Underspecified Motion Events