Situation Recognition with Graph Neural Networks

Abstract

We address the problem of recognizing situations in images. Given an image, the task is to predict the most salient verb (action), and fill its semantic roles such as who is performing the action, what is the source and target of the action, etc. Different verbs have different roles (e.g. attacking has weapon), and each role can take on many possible values (nouns). We propose a model based on Graph Neural Networks that allows us to efficiently capture joint dependencies between roles using neural networks defined on a graph. Experiments with different graph connectivities show that our approach that propagates information between roles significantly outperforms existing work, as well as multiple baselines. We obtain roughly 3-5% improvement over previous work in predicting the full situation. We also provide a thorough qualitative analysis of our model and influence of different roles in the verbs.

ICCV2017

References in corpus (14)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Residual Learning for Image Recognition
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Spectral Networks and Locally Connected Networks on Graphs
• Conditional Random Fields as Recurrent Neural Networks
• VQA: Visual Question Answering
• Visual Semantic Role Labeling
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Places: An Image Database for Deep Scene Understanding
• Visual Relationship Detection with Language Priors
• Visual Translation Embedding Network for Visual Relation Detection
• Annotating Object Instances with a Polygon-RNN
• Teaching Machines to Describe Images via Natural Language Feedback
• Commonly Uncommon: Semantic Sparsity in Situation Recognition

Cited by in corpus (10)

• Deep Vessel Segmentation By Learning Graphical Connectivity
• An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition
• Learning Human-Object Interactions by Graph Parsing Neural Networks
• Graph Partition Neural Networks for Semi-Supervised Classification
• Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning
• V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction
• A Review on Intelligent Object Perception Methods Combining Knowledge-based Reasoning and Machine Learning
• Learning to Act Properly: Predicting and Explaining Affordances from Images
• Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks
• Context-aware short-term interest first model for session-based recommendation