Middle-Out Decoding
arXiv:1810.11735
Abstract
Despite being virtually ubiquitous, sequence-to-sequence models are challenged by their lack of diversity and inability to be externally controlled. In this paper, we speculate that a fundamental shortcoming of sequence generation models is that the decoding is done strictly from left-to-right, meaning that outputs values generated earlier have a profound effect on those generated later. To address this issue, we propose a novel middle-out decoder architecture that begins from an initial middle-word and simultaneously expands the sequence in both directions. To facilitate information flow and maintain consistent decoding, we introduce a dual self-attention mechanism that allows us to model complex dependencies between the outputs. We illustrate the performance of our model on the task of video captioning, as well as a synthetic sequence de-noising task. Our middle-out decoder achieves significant improvements on de-noising and competitive performance in the task of video captioning, while quantifiably improving the caption diversity. Furthermore, we perform a qualitative analysis that demonstrates our ability to effectively control the generation process of our decoder.
arXiv commentPublished as a conference paper at NIPS 2018
References in corpus (18)
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- VQA: Visual Question Answering
- Sequence to Sequence -- Video to Text
- Describing Videos by Exploiting Temporal Structure
- Long Short-Term Memory-Networks for Machine Reading
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Texygen: A Benchmarking Platform for Text Generation Models
- Language Models for Image Captioning: The Quirks and What Works
- Towards Diverse and Natural Image Descriptions via a Conditional GAN
- Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
- Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search
- Jointly Modeling Embedding and Translation to Bridge Video and Language
- Neural Program Search: Solving Programming Tasks from Description and Examples
- Generating captions without looking beyond objects
- Learning Structured Text Representations