PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN

Abstract

We aim to tackle a novel vision task called Weakly Supervised Visual Relation Detection (WSVRD) to detect "subject-predicate-object" relations in an image with object relation groundtruths available only at the image level. This is motivated by the fact that it is extremely expensive to label the combinatorial relations between objects at the instance level. Compared to the extensively studied problem, Weakly Supervised Object Detection (WSOD), WSVRD is more challenging as it needs to examine a large set of regions pairs, which is computationally prohibitive and more likely stuck in a local optimal solution such as those involving wrong spatial context. To this end, we present a Parallel, Pairwise Region-based, Fully Convolutional Network (PPR-FCN) for WSVRD. It uses a parallel FCN architecture that simultaneously performs pair selection and classification of single regions and region pairs for object and relation detection, while sharing almost all computation shared over the entire image. In particular, we propose a novel position-role-sensitive score map with pairwise RoI pooling to efficiently capture the crucial context associated with a pair of objects. We demonstrate the superiority of PPR-FCN over all baselines in solving the WSVRD challenge by using results of extensive experiments over two visual relation benchmarks.

arXiv commentTo appear in International Conference on Computer Vision (ICCV) 2017, Venice, Italy

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Deep Residual Learning for Image Recognition
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• YFCC100M: The New Data in Multimedia Research
• Going Deeper with Convolutions
• Fully Convolutional Networks for Semantic Segmentation
• Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Show and Tell: A Neural Image Caption Generator
• Visual Relationship Detection with Language Priors
• Detecting Visual Relationships with Deep Relational Networks
• Visual Translation Embedding Network for Visual Relation Detection
• Learning to generalize to new compositions in image understanding
• Modeling Context in Referring Expressions
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localization
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Deep Self-Taught Learning for Weakly Supervised Object Localization
• Modeling Relationships in Referential Expressions with Compositional Modular Networks
• Surveillance Video Parsing with Single Frame Supervision

Cited by in corpus (7)

• WRENCH: A Comprehensive Benchmark for Weak Supervision
• Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
• Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions
• Unbiased Scene Graph Generation via Rich and Fair Semantic Extraction
• An Interpretable Model for Scene Graph Generation
• Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance
• Human-Object Interaction Detection via Weak Supervision