Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction

Abstract

Machine understanding of complex images is a key goal of artificial intelligence. One challenge underlying this task is that visual scenes contain multiple inter-related objects, and that global context plays an important role in interpreting the scene. A natural modeling framework for capturing such effects is structured prediction, which optimizes over complex labels, while modeling within-label interactions. However, it is unclear what principles should guide the design of a structured prediction model that utilizes the power of deep learning components. Here we propose a design principle for such architectures that follows from a natural requirement of permutation invariance. We prove a necessary and sufficient characterization for architectures that follow this invariance, and discuss its implication on model design. Finally, we show that the resulting model achieves new state of the art results on the Visual Genome scene graph labeling benchmark, outperforming all recent approaches.

arXiv commentPaper is accepted for NIPS 2018 conference

Cited by in corpus (24)

• Neural Relational Inference for Interacting Systems
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• On the equivalence between graph isomorphism testing and function approximation with GNNs
• Visual Relationship Detection with Visual-Linguistic Knowledge from Multimodal Representations
• Counterfactual Critic Multi-Agent Training for Scene Graph Generation
• Bridging Knowledge Graphs to Generate Scene Graphs
• Action Genome: Actions as Composition of Spatio-temporal Scene Graphs
• The Limited Multi-Label Projection Layer
• VrR-VG: Refocusing Visually-Relevant Relationships
• Attentive Relational Networks for Mapping Images to Scene Graphs
• GPS-Net: Graph Property Sensing Network for Scene Graph Generation
• Graph Structured Network for Image-Text Matching
• Regularizing Towards Permutation Invariance in Recurrent Models
• Learning Object Detection from Captions via Textual Scene Attributes
• Compositional Video Synthesis with Action Graphs
• Relation Transformer Network
• Semantic Compositional Learning for Low-shot Scene Graph Generation
• Visual Semantic Information Pursuit: A Survey
• Semantic Image Manipulation Using Scene Graphs
• Learning Predicates as Functions to Enable Few-shot Scene Graph Prediction
• HOSE-Net: Higher Order Structure Embedded Network for Scene Graph Generation
• Graph Embedding VAE: A Permutation Invariant Model of Graph Structure
• Image-Graph-Image Translation via Auto-Encoding
• Addressing Class Imbalance in Scene Graph Parsing by Learning to Contrast and Score