Detecting and Recognizing Human-Object Interactions

Abstract

To understand the visual world, a machine must not only recognize individual object instances but also how they interact. Humans are often at the center of such interactions and detecting human-object interactions is an important practical and scientific problem. In this paper, we address the task of detecting <human, verb, object> triplets in challenging everyday photos. We propose a novel model that is driven by a human-centric approach. Our hypothesis is that the appearance of a person -- their pose, clothing, action -- is a powerful cue for localizing the objects they are interacting with. To exploit this cue, our model learns to predict an action-specific density over target object locations based on the appearance of a detected person. Our model also jointly learns to detect people and objects, and by fusing these predictions it efficiently infers interaction triplets in a clean, jointly trained end-to-end system we call InteractNet. We validate our approach on the recently introduced Verbs in COCO (V-COCO) and HICO-DET datasets, where we show quantitatively compelling results.

References in corpus (6)

• Going Deeper with Convolutions
• Feature Pyramid Networks for Object Detection
• Visual Semantic Role Labeling
• Visual Relationship Detection with Language Priors
• Learning to Detect Human-Object Interactions
• Describing Common Human Visual Actions in Images

Cited by in corpus (13)

• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Analyzing Human-Human Interactions: A Survey
• Detailed 2D-3D Joint Representation for Human-Object Interaction
• EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• Understanding Human Hands in Contact at Internet Scale
• Grounded Objects and Interactions for Video Captioning
• Activity Driven Weakly Supervised Object Detection
• Turbo Learning Framework for Human-Object Interactions Recognition and Human Pose Estimation
• Spatial Priming for Detecting Human-Object Interactions
• Generating Videos of Zero-Shot Compositions of Actions and Objects
• Loss Guided Activation for Action Recognition in Still Images
• Meta R-CNN : Towards General Solver for Instance-level Few-shot Learning