No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling

Abstract

Though impressive results have been achieved in visual captioning, the task of generating abstract stories from photo streams is still a little-tapped problem. Different from captions, stories have more expressive language styles and contain many imaginary concepts that do not appear in the images. Thus it poses challenges to behavioral cloning algorithms. Furthermore, due to the limitations of automatic metrics on evaluating story quality, reinforcement learning methods with hand-crafted rewards also face difficulties in gaining an overall performance boost. Therefore, we propose an Adversarial REward Learning (AREL) framework to learn an implicit reward function from human demonstrations, and then optimize policy search with the learned reward function. Though automatic eval- uation indicates slight performance boost over state-of-the-art (SOTA) methods in cloning expert behaviors, human evaluation shows that our approach achieves significant improvement in generating more human-like stories than SOTA systems.

arXiv commentACL 2018. 15 pages, 10 figures, 4 tables, with supplementary material

References in corpus (11)

• Microsoft COCO Captions: Data Collection and Evaluation Server
• A Deep Reinforced Model for Abstractive Summarization
• Convolutional Neural Networks for Sentence Classification
• Sequence Level Training with Recurrent Neural Networks
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Generative Adversarial Imitation Learning
• An Actor-Critic Algorithm for Sequence Prediction
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Visual Storytelling
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward

Cited by in corpus (5)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Learning to Generalize from Sparse and Underspecified Rewards
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Visual Storytelling via Predicting Anchor Word Embeddings in the Stories
• SDA: Improving Text Generation with Self Data Augmentation