Spatial Priming for Detecting Human-Object Interactions

Abstract

The relative spatial layout of a human and an object is an important cue for determining how they interact. However, until now, spatial layout has been used just as side-information for detecting human-object interactions (HOIs). In this paper, we present a method for exploiting this spatial layout information for detecting HOIs in images. The proposed method consists of a layout module which primes a visual module to predict the type of interaction between a human and an object. The visual and layout modules share information through lateral connections at several stages. The model uses predictions from the layout module as a prior to the visual module and the prediction from the visual module is given as the final output. It also incorporates semantic information about the object using word2vec vectors. The proposed model reaches an mAP of 24.79% for HICO-Det dataset which is about 2.8% absolute points higher than the current state-of-the-art.

References in corpus (11)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• Spatiotemporal Residual Networks for Video Action Recognition
• Beyond Skip Connections: Top-Down Modulation for Object Detection
• Visual Semantic Role Labeling
• SlowFast Networks for Video Recognition
• Semantic Autoencoder for Zero-Shot Learning
• Zero-Shot Learning -- The Good, the Bad and the Ugly
• Detecting and Recognizing Human-Object Interactions
• Learning Human-Object Interactions by Graph Parsing Neural Networks
• Learning to Detect Human-Object Interactions