Learning Human-Object Interactions by Graph Parsing Neural Networks

Abstract

This paper addresses the task of detecting and recognizing human-object interactions (HOI) in images and videos. We introduce the Graph Parsing Neural Network (GPNN), a framework that incorporates structural knowledge while being differentiable end-to-end. For a given scene, GPNN infers a parse graph that includes i) the HOI graph structure represented by an adjacency matrix, and ii) the node labels. Within a message passing inference framework, GPNN iteratively computes the adjacency matrices and node labels. We extensively evaluate our model on three HOI detection benchmarks on images and videos: HICO-DET, V-COCO, and CAD-120 datasets. Our approach significantly outperforms state-of-art methods, verifying that GPNN is scalable to large datasets and applies to spatial-temporal settings. The code is available at https://github.com/SiyuanQi/gpnn.

arXiv commentThis paper is published in ECCV 2018

References in corpus (9)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Neural Message Passing for Quantum Chemistry
• Learning Convolutional Neural Networks for Graphs
• Deformable Convolutional Networks
• Visual Semantic Role Labeling
• Scene Graph Generation by Iterative Message Passing
• Situation Recognition with Graph Neural Networks
• Temporal Dynamic Graph LSTM for Action-driven Video Object Detection

Cited by in corpus (21)

• An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition
• Temporal Graph Networks for Deep Learning on Dynamic Graphs
• HOI Analysis: Integrating and Decomposing Human-Object Interaction
• Accelerating Large Scale Real-Time GNN Inference using Channel Pruning
• Learning Human-Object Interaction Detection using Interaction Points
• PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection
• InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting
• Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval
• Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspective
• LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention
• Hierarchical Human Parsing with Typed Part-Relation Reasoning
• Cascaded Human-Object Interaction Recognition
• Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection
• Reasoning About Human-Object Interactions Through Dual Attention Networks
• Dynamic Graph Modules for Modeling Object-Object Interactions in Activity Recognition
• Symbiotic Attention with Privileged Information for Egocentric Action Recognition
• Explainable Video Action Reasoning via Prior Knowledge and State Transitions
• Spatial Priming for Detecting Human-Object Interactions
• Spatio-Temporal Interaction Graph Parsing Networks for Human-Object Interaction Recognition
• Long Short-Term Relation Networks for Video Action Detection
• Target-Tailored Source-Transformation for Scene Graph Generation