Scene Graph Generation by Iterative Message Passing

Abstract

Understanding a visual scene goes beyond recognizing individual objects in isolation. Relationships between objects also constitute rich semantic information about the scene. In this work, we explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image. We propose a novel end-to-end model that generates such structured scene representation from an input image. The model solves the scene graph inference problem using standard RNNs and learns to iteratively improves its predictions via message passing. Our joint inference model can take advantage of contextual cues to make better predictions on objects and their relationships. The experiments show that our model significantly outperforms previous methods for generating scene graphs using Visual Genome dataset and inferring support relations with NYU Depth v2 dataset.

CVPR 2017

Cited by in corpus (43)

• Pixels to Graphs by Associative Embedding
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Learning Human-Object Interactions by Graph Parsing Neural Networks
• Exploring Visual Relationship for Image Captioning
• Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
• Learning to Compose Dynamic Tree Structures for Visual Contexts
• PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection
• Auto-Encoding Scene Graphs for Image Captioning
• Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships
• Natural Language Guided Visual Relationship Detection
• Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval
• Heuristics for k-domination models of facility location problems in street networks
• Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions
• Scene Graph Generation via Conditional Random Fields
• Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• Meta Module Network for Compositional Visual Reasoning
• Graph R-CNN for Scene Graph Generation
• An Interpretable Model for Scene Graph Generation
• Dual ResGCN for Balanced Scene GraphGeneration
• Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path
• Scene Graph Parsing as Dependency Parsing
• LinkNet: Relational Embedding for Scene Graph
• SOGNet: Scene Overlap Graph Network for Panoptic Segmentation
• Introduction to the 1st Place Winning Model of OpenImages Relationship Detection Challenge
• 2nd Place Solution to the GQA Challenge 2019
• One-shot Scene Graph Generation
• Is Object Detection Necessary for Human-Object Interaction Recognition?
• Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation
• What and Where: A Context-based Recommendation System for Object Insertion
• Scenes and Surroundings: Scene Graph Generation using Relation Transformer
• ORD: Object Relationship Discovery for Visual Dialogue Generation
• Data-Efficient Graph Embedding Learning for PCB Component Detection
• Recovering the Unbiased Scene Graphs from the Biased Ones
• Semantic Image Manipulation Using Scene Graphs
• Seq-SG2SL: Inferring Semantic Layout from Scene Graph Through Sequence to Sequence Learning
• Image interpretation by iterative bottom-up top-down processing
• Learning Actor Relation Graphs for Group Activity Recognition
• Shuffle-Then-Assemble: Learning Object-Agnostic Visual Relationship Features
• Target-Tailored Source-Transformation for Scene Graph Generation
• MOC-GAN: Mixing Objects and Captions to Generate Realistic Images