Masked Non-Autoregressive Image Captioning
arXiv:1906.00717
Abstract
Existing captioning models often adopt the encoder-decoder architecture, where the decoder uses autoregressive decoding to generate captions, such that each token is generated sequentially given the preceding generated tokens. However, autoregressive decoding results in issues such as sequential error accumulation, slow generation, improper semantics and lack of diversity. Non-autoregressive decoding has been proposed to tackle slow generation for neural machine translation but suffers from multimodality problem due to the indirect modeling of the target distribution. In this paper, we propose masked non-autoregressive decoding to tackle the issues of both autoregressive decoding and non-autoregressive decoding. In masked non-autoregressive decoding, we mask several kinds of ratios of the input sequences during training, and generate captions parallelly in several stages from a totally masked sequence to a totally non-masked sequence in a compositional manner during inference. Experimentally our proposed model can preserve semantic content more effectively and can generate more diverse captions.
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Non-Autoregressive Neural Machine Translation
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- Towards Diverse and Natural Image Descriptions via a Conditional GAN
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- A Neural Compositional Paradigm for Image Captioning
Cited by in corpus (5)
- Automated Audio Captioning: An Overview of Recent Progress and New Challenges
- Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning
- Fast Sequence Generation with Multi-Agent Reinforcement Learning
- Semi-Autoregressive Image Captioning
- Diff-3DCap: Shape Captioning with Diffusion Models