Fast Decoding in Sequence Models using Discrete Latent Variables

Abstract

Autoregressive sequence models based on deep neural networks, such as RNNs, Wavenet and the Transformer attain state-of-the-art results on many tasks. However, they are difficult to parallelize and are thus slow at processing long sequences. RNNs lack parallelism both during training and decoding, while architectures like WaveNet and Transformer are much more parallelizable during training, yet still operate sequentially during decoding. Inspired by [arxiv:1711.00937], we present a method to extend sequence models using discrete latent variables that makes decoding much more parallelizable. We first auto-encode the target sequence into a shorter sequence of discrete latent variables, which at inference time is generated autoregressively, and finally decode the output sequence from this shorter latent sequence in parallel. To this end, we introduce a novel method for constructing a sequence of discrete latent variables and compare it with previously introduced methods. Finally, we evaluate our model end-to-end on the task of neural machine translation, where it is an order of magnitude faster at decoding than comparable autoregressive models. While lower in BLEU than purely autoregressive models, our model achieves higher scores than previously proposed non-autoregressive translation models.

arXiv commentICML 2018

References in corpus (15)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Categorical Reparameterization with Gumbel-Softmax
• Convolutional Sequence to Sequence Learning
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Neural Machine Translation in Linear Time
• Depthwise Separable Convolutions for Neural Machine Translation
• Importance Weighted Autoencoders
• Neural GPUs Learn Algorithms
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Variational inference for Monte Carlo objectives
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• Discrete Autoencoders for Sequence Models
• Learning the Base Distribution in Implicit Generative Models

Cited by in corpus (69)

• CTRL: A Conditional Transformer Language Model for Controllable Generation
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• Fast Structured Decoding for Sequence Models
• Syntactically Supervised Transformers for Faster Neural Machine Translation
• Theory and Experiments on Vector Quantized Autoencoders
• Variational Transformers for Diverse Response Generation
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Discrete Flows: Invertible Generative Models of Discrete Data
• Budget-Aware Adapters for Multi-Domain Learning
• Understanding Knowledge Distillation in Non-autoregressive Machine Translation
• A Tutorial on Deep Latent Variable Models of Natural Language
• Imitation Learning for Non-Autoregressive Neural Machine Translation
• Real or Fake? Learning to Discriminate Machine from Human Generated Text
• Non-autoregressive Transformer by Position Learning
• A Comprehensive Survey of Grammar Error Correction
• Non-Autoregressive Machine Translation with Auxiliary Regularization
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
• Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• Hint-Based Training for Non-Autoregressive Machine Translation
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
• Semi-Autoregressive Neural Machine Translation
• LAVA NAT: A Non-Autoregressive Translation Model with Look-Around Decoding and Vocabulary Attention
• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
• Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models
• Information Maximizing Visual Question Generation
• Learning Discrete Structured Representations by Adversarially Maximizing Mutual Information
• ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
• (Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation
• Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation
• Dialog without Dialog Data: Learning Visual Dialog Agents from VQA Data
• Fast Sequence Generation with Multi-Agent Reinforcement Learning
• Parallel Iterative Edit Models for Local Sequence Transduction
• OCTOPUS: Overcoming Performance andPrivatization Bottlenecks in Distributed Learning
• You Look Twice: GaterNet for Dynamic Filter Selection in CNNs
• Latent Programmer: Discrete Latent Codes for Program Synthesis
• Non-Autoregressive Translation by Learning Target Categorical Codes
• Extractive Summary as Discrete Latent Variables
• Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
• Select and Attend: Towards Controllable Content Selection in Text Generation
• Fast Interleaved Bidirectional Sequence Generation
• Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder
• Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders
• A Study on the Autoregressive and non-Autoregressive Multi-label Learning
• Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraints
• Speech-to-speech Translation between Untranscribed Unknown Languages
• Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism
• Sequence-Level Training for Non-Autoregressive Neural Machine Translation
• Neural Machine Translation: A Review and Survey
• Learning Sampling in Financial Statement Audits using Vector Quantised Autoencoder Neural Networks
• Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoders
• Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation
• Towards Interlingua Neural Machine Translation
• Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval
• Interpretable Neural Network Decoupling
• Discrete Structural Planning for Neural Machine Translation
• Incorporating Symbolic Sequential Modeling for Speech Enhancement
• Unsupervised Learning of Neurosymbolic Encoders
• Supervised Vector Quantized Variational Autoencoder for Learning Interpretable Global Representations
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck
• Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation
• Sequence Generation: From Both Sides to the Middle
• Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP
• Entropy optimized semi-supervised decomposed vector-quantized variational autoencoder model based on transfer learning for multiclass text classification and generation
• Modeling Coverage for Non-Autoregressive Neural Machine Translation
• A Comparison of Discrete Latent Variable Models for Speech Representation Learning
• Non-autoregressive electron flow generation for reaction prediction
• Quantization-Based Regularization for Autoencoders
• MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation