Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning

Abstract

This paper addresses a new problem of understanding human gaze communication in social videos from both atomic-level and event-level, which is significant for studying human social interactions. To tackle this novel and challenging problem, we contribute a large-scale video dataset, VACATION, which covers diverse daily social scenes and gaze communication behaviors with complete annotations of objects and human faces, human attention, and communication structures and labels in both atomic-level and event-level. Together with VACATION, we propose a spatio-temporal graph neural network to explicitly represent the diverse gaze interactions in the social scenes and to infer atomic-level gaze communication by message passing. We further propose an event network with encoder-decoder structure to predict the event-level gaze communication. Our experiments demonstrate that the proposed model improves various baselines significantly in predicting the atomic-level and event-level gaze

arXiv commentAccepted by ICCV 2019

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Interaction Networks for Learning about Objects, Relations and Physics
• Discriminative Embeddings of Latent Variable Models for Structured Data

Cited by in corpus (11)

• Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
• Spatio-Temporal Graph for Video Captioning with Knowledge Distillation
• LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention
• Hierarchical Human Parsing with Typed Part-Relation Reasoning
• Cascaded Human-Object Interaction Recognition
• Decentralized Structural-RNN for Robot Crowd Navigation with Deep Reinforcement Learning
• Ano-Graph: Learning Normal Scene Contextual Graphs to Detect Video Anomalies
• Consistency-Aware Graph Network for Human Interaction Understanding
• Detecting Attended Visual Targets in Video
• Self-Learning with Rectification Strategy for Human Parsing
• Learning Triadic Belief Dynamics in Nonverbal Communication from Videos