RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
arXiv:1805.05225 · doi:10.18653/v1/P18-4022
Abstract
We compare the fast training and decoding speed of RETURNN of attention models for translation, due to fast CUDA LSTM kernels, and a fast pure TensorFlow beam search decoder. We show that a layer-wise pretraining scheme for recurrent attention models gives over 1% BLEU improvement absolute and it allows to train deeper recurrent encoder networks. Promising preliminary results on max. expected BLEU training are presented. We are able to train state-of-the-art models for translation and end-to-end models for speech recognition and show results on WMT 2017 and Switchboard. The flexibility of RETURNN allows a fast research feedback loop to experiment with alternative architectures, and its generality allows to use it on a wide range of applications.
arXiv commentaccepted as demo paper on ACL 2018
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Neural Machine Translation by Jointly Learning to Align and Translate
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Focal Loss for Dense Object Detection
- Sockeye: A Toolkit for Neural Machine Translation
- A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
- Modeling Coverage for Neural Machine Translation
- Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions
- Nematus: a Toolkit for Neural Machine Translation
- Toward a full-scale neural machine translation in production: the Booking.com use case
- Machine Translation at Booking.com: Journey and Lessons Learned
Cited by in corpus (23)
- SpeechBrain: A General-Purpose Speech Toolkit
- Improved training of end-to-end attention models for speech recognition
- Language Modeling with Deep Transformers
- fairseq S2T: Fast Speech-to-Text Modeling with fairseq
- A New Training Pipeline for an Improved Neural Transducer
- Why does CTC result in peaky behavior?
- Espresso: A Fast End-to-end Neural Speech Recognition Toolkit
- CAT: CRF-based ASR Toolkit
- Context-Dependent Acoustic Modeling without Explicit Phone Clustering
- Novel Applications of Factored Neural Machine Translation
- ESPnet-ST: All-in-One Speech Translation Toolkit
- A study of latent monotonic attention variants
- Gated Recurrent Context: Softmax-free Attention for Online Encoder-Decoder Speech Recognition
- Robust Beam Search for Encoder-Decoder Attention Based Speech Recognition without Length Bias
- The 2020 ESPnet update: new features, broadened applications, performance improvements, and future plans
- A systematic comparison of grapheme-based vs. phoneme-based label units for encoder-decoder-attention models
- Towards Consistent Hybrid HMM Acoustic Modeling
- RASR2: The RWTH ASR Toolkit for Generic Sequence-to-sequence Speech Recognition
- Combining TF-GridNet and Mixture Encoder for Continuous Speech Separation for Meeting Transcription
- Improved Multi-Stage Training of Online Attention-based Encoder-Decoder Models
- CAT: A CTC-CRF based ASR Toolkit Bridging the Hybrid and the End-to-end Approaches towards Data Efficiency and Low Latency
- On Sampling-Based Training Criteria for Neural Language Modeling
- uniblock: Scoring and Filtering Corpus with Unicode Block Information