Imputer: Sequence Modelling via Imputation and Dynamic Programming
arXiv:2002.08926
Abstract
This paper presents the Imputer, a neural sequence model that generates output sequences iteratively via imputations. The Imputer is an iterative generative model, requiring only a constant number of generation steps independent of the number of input or output tokens. The Imputer can be trained to approximately marginalize over all possible alignments between the input and output sequences, and all possible generation orders. We present a tractable dynamic programming training algorithm, which yields a lower bound on the log marginal likelihood. When applied to end-to-end speech recognition, the Imputer outperforms prior non-autoregressive models and achieves competitive results to autoregressive models. On LibriSpeech test-other, the Imputer achieves 11.1 WER, outperforming CTC at 13.0 WER and seq2seq at 12.5 WER.
References in corpus (17)
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- WaveNet: A Generative Model for Raw Audio
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Sequence Transduction with Recurrent Neural Networks
- Non-Autoregressive Neural Machine Translation
- Improved training of end-to-end attention models for speech recognition
- Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
- Insertion Transformer: Flexible Sequence Generation via Insertion Operations
- Transformers with convolutional context for ASR
- Exploring Neural Transducers for End-to-End Speech Recognition
- KERMIT: Generative Insertion-Based Modeling for Sequences
- Twin Networks: Matching the Future for Sequence Generation
- Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling
- Latent Sequence Decompositions
- Task Loss Estimation for Sequence Prediction
- Insertion-based Decoding with automatically Inferred Generation Order
Cited by in corpus (7)
- Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
- Pushing the Limits of Non-Autoregressive Speech Recognition
- Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict
- Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation
- Insertion-Based Modeling for End-to-End Automatic Speech Recognition
- Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder
- Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation