Recurrent Topic-Transition GAN for Visual Paragraph Generation

Abstract

A natural image usually conveys rich semantic content and can be viewed from different angles. Existing image description methods are largely restricted by small sets of biased visual paragraph annotations, and fail to cover rich underlying semantics. In this paper, we investigate a semi-supervised paragraph generative framework that is able to synthesize diverse and semantically coherent paragraph descriptions by reasoning over local semantic regions and exploiting linguistic knowledge. The proposed Recurrent Topic-Transition Generative Adversarial Network (RTT-GAN) builds an adversarial framework between a structured paragraph generator and multi-level paragraph discriminators. The paragraph generator generates sentences recurrently by incorporating region-based visual and language attention mechanisms at each step. The quality of generated paragraph sentences is assessed by multi-level adversarial discriminators from two aspects, namely, plausibility at sentence level and topic-transition coherence at paragraph level. The joint adversarial training of RTT-GAN drives the model to generate realistic paragraphs with smooth logical transition between sentence topics. Extensive quantitative experiments on image and video paragraph datasets demonstrate the effectiveness of our RTT-GAN in both supervised and semi-supervised settings. Qualitative results on telling diverse stories for an image also verify the interpretability of RTT-GAN.

10 pages, 6 figures

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Image Captioning with Semantic Attention
• Coherent Multi-Sentence Video Description with Variable Level of Detail
• Describing Videos by Exploiting Temporal Structure
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Toward Controlled Generation of Text
• Visual Storytelling
• Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
• Review Networks for Caption Generation
• Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection

Cited by in corpus (29)

• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Improved Training of Wasserstein GANs
• Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
• NetGAN: Generating Graphs via Random Walks
• An Introduction to Image Synthesis with Generative Adversarial Nets
• Language Generation with Recurrent Generative Adversarial Networks without Pre-training
• Semantic-aware Grad-GAN for Virtual-to-Real Urban Scene Adaption
• Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations
• Video Storytelling: Textual Summaries for Events
• 3D Human Pose Estimation in the Wild by Adversarial Learning
• Structured Generative Adversarial Networks
• Learning Semantic Sentence Embeddings using Sequential Pair-wise Discriminator
• StoryGAN: A Sequential Conditional GAN for Story Visualization
• Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos
• From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
• Learning to Evaluate Image Captioning
• Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
• Dual Motion GAN for Future-Flow Embedded Video Prediction
• Shaping the Narrative Arc: An Information-Theoretic Approach to Collaborative Dialogue
• Automatic Article Commenting: the Task and Dataset
• InfoCatVAE: Representation Learning with Categorical Variational Autoencoders
• Recent Advances of Image Steganography with Generative Adversarial Networks
• Improving Visual Question Answering by Referring to Generated Paragraph Captions
• CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
• Unsupervised Stylish Image Description Generation via Domain Layer Norm
• Efficient Correlated Topic Modeling with Topic Embedding
• A Customizable Dynamic Scenario Modeling and Data Generation Platform for Autonomous Driving
• Enhanced Modality Transition for Image Captioning
• Off-Policy Self-Critical Training for Transformer in Visual Paragraph Generation