Auto-Encoding Scene Graphs for Image Captioning

Abstract

We propose Scene Graph Auto-Encoder (SGAE) that incorporates the language inductive bias into the encoder-decoder image captioning framework for more human-like captions. Intuitively, we humans use the inductive bias to compose collocations and contextual inference in discourse. For example, when we see the relation `person on bike', it is natural to replace `on' with `ride' and infer `person riding bike on a road' even the `road' is not evident. Therefore, exploiting such bias as a language prior is expected to help the conventional encoder-decoder models less likely overfit to the dataset bias and focus on reasoning. Specifically, we use the scene graph --- a directed graph ($\mathcal{G}$) where an object node is connected by adjective nodes and relationship nodes --- to represent the complex structural layout of both image ($\mathcal{I}$) and sentence ($\mathcal{S}$). In the textual domain, we use SGAE to learn a dictionary ($\mathcal{D}$) that helps to reconstruct sentences in the $\mathcal{S}\to \mathcal{G}\to \mathcal{D}\to \mathcal{S}$ pipeline, where $\mathcal{D}$ encodes the desired language prior; in the vision-language domain, we use the shared $\mathcal{D}$ to guide the encoder-decoder in the $\mathcal{I}\to \mathcal{G}\to \mathcal{D}\to \mathcal{S}$ pipeline. Thanks to the scene graph representation and shared dictionary, the inductive bias is transferred across domains in principle. We validate the effectiveness of SGAE on the challenging MS-COCO image captioning benchmark, e.g., our SGAE-based single-model achieves a new state-of-the-art $127.8$ CIDEr-D on the Karpathy split, and a competitive $125.5$ CIDEr-D (c40) on the official server even compared to other ensemble models.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Sequence Level Training with Recurrent Neural Networks
• Matching Networks for One Shot Learning
• Dynamic Memory Networks for Visual and Textual Question Answering
• Gated Graph Sequence Neural Networks
• Learning to Count Objects in Natural Images for Visual Question Answering
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Scene Graph Generation by Iterative Message Passing
• Visual Translation Embedding Network for Visual Relation Detection
• Scene Graph Parsing as Dependency Parsing

Cited by in corpus (8)

• Learning to Collocate Neural Modules for Image Captioning
• Macroscopic Control of Text Generation for Image Captioning
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• Scene Graph Generation for Better Image Captioning?
• A Dataset and Benchmarks for Multimedia Social Analysis
• Relationship-Aware Spatial Perception Fusion for Realistic Scene Layout Generation
• Hidden State Guidance: Improving Image Captioning using An Image Conditioned Autoencoder
• PR Product: A Substitute for Inner Product in Neural Networks