Target-Tailored Source-Transformation for Scene Graph Generation

Abstract

Scene graph generation aims to provide a semantic and structural description of an image, denoting the objects (with nodes) and their relationships (with edges). The best performing works to date are based on exploiting the context surrounding objects or relations,e.g., by passing information among objects. In these approaches, to transform the representation of source objects is a critical process for extracting information for the use by target objects. In this work, we argue that a source object should give what tar-get object needs and give different objects different information rather than contributing common information to all targets. To achieve this goal, we propose a Target-TailoredSource-Transformation (TTST) method to efficiently propagate information among object proposals and relations. Particularly, for a source object proposal which will contribute information to other target objects, we transform the source object feature to the target object feature domain by simultaneously taking both the source and target into account. We further explore more powerful representations by integrating language prior with the visual context in the transformation for the scene graph generation. By doing so the target object is able to extract target-specific information from the source object and source relation accordingly to refine its representation. Our framework is validated on the Visual Genome bench-mark and demonstrated its state-of-the-art performance for the scene graph generation. The experimental results show that the performance of object detection and visual relation-ship detection are promoted mutually by our method.

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• YOLO9000: Better, Faster, Stronger
• Pyramid Scene Parsing Network
• Pixels to Graphs by Associative Embedding
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Scene Graph Generation by Iterative Message Passing
• Visual Translation Embedding Network for Visual Relation Detection
• PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph
• Learning Human-Object Interactions by Graph Parsing Neural Networks
• Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
• Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation
• Natural Language Guided Visual Relationship Detection
• Large-Scale Visual Relationship Understanding