ViP-CNN: Visual Phrase Guided Convolutional Neural Network
arXiv:1702.07191
Abstract
As the intermediate level task connecting image captioning and object detection, visual relationship detection started to catch researchers' attention because of its descriptive power and clear structure. It detects the objects and captures their pair-wise interactions with a subject-predicate-object triplet, e.g. person-ride-horse. In this paper, each visual relationship is considered as a phrase with three components. We formulate the visual relationship detection as three inter-connected recognition problems and propose a Visual Phrase guided Convolutional Neural Network (ViP-CNN) to address them simultaneously. In ViP-CNN, we present a Phrase-guided Message Passing Structure (PMPS) to establish the connection among relationship components and help the model consider the three problems jointly. Corresponding non-maximum suppression method and model training strategy are also proposed. Experimental results show that our ViP-CNN outperforms the state-of-art method both in speed and accuracy. We further pretrain ViP-CNN on our cleansed Visual Genome Relationship dataset, which is found to perform better than the pretraining on the ImageNet for this task.
arXiv comment10 pages, 5 figures, accepted by CVPR 2017
References in corpus (18)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Efficient Estimation of Word Representations in Vector Space
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
- VQA: Visual Question Answering
- T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videos
- Rich feature hierarchies for accurate object detection and semantic segmentation
- Image Captioning with Semantic Attention
- Learning a Recurrent Visual Representation for Image Caption Generation
- Visual Relationship Detection with Language Priors
- From Captions to Visual Concepts and Back
- DeepID-Net: Deformable Deep Convolutional Neural Networks for Object Detection
- Deformable Part Models are Convolutional Neural Networks
- Learning to generalize to new compositions in image understanding
- Dense Captioning with Joint Inference and Visual Context
- Structured Feature Learning for Pose Estimation