Optimal Completion Distillation for Sequence Learning

Abstract

We present Optimal Completion Distillation (OCD), a training procedure for optimizing sequence to sequence models based on edit distance. OCD is efficient, has no hyper-parameters of its own, and does not require pretraining or joint optimization with conditional log-likelihood. Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient dynamic programming algorithm. Then, for each position of the generated sequence, we use a target distribution that puts equal probability on the first token of all the optimal suffixes. OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and Librispeech datasets, achieving $9.3\%$ WER and $4.5\%$ WER respectively.

References in corpus (26)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Achieving Human Parity on Automatic Chinese to English News Translation
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Improved training of end-to-end attention models for speech recognition
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• An Actor-Critic Algorithm for Sequence Prediction
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• Sequence-to-Sequence Learning as Beam-Search Optimization
• Learning to Search Better Than Your Teacher
• Reward Augmented Maximum Likelihood for Neural Structured Prediction
• Policy Distillation
• Twin Networks: Matching the Future for Sequence Generation
• Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling
• Latent Sequence Decompositions
• Letter-Based Speech Recognition with Gated ConvNets
• Task Loss Estimation for Sequence Prediction
• Softmax Q-Distribution Estimation for Structured Prediction: A Theoretical Interpretation for RAML
• SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation
• Convergence of Value Aggregation for Imitation Learning
• Learning Noise-Invariant Representations for Robust Speech Recognition

Cited by in corpus (4)

• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• Promising Accurate Prefix Boosting for sequence-to-sequence ASR
• Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation
• Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models