Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq
arXiv:1805.10387
Abstract
We present OpenSeq2Seq - a TensorFlow-based toolkit for training sequence-to-sequence models that features distributed and mixed-precision training. Benchmarks on machine translation and speech recognition tasks show that models built using OpenSeq2Seq give state-of-the-art performance at 1.5-3x less training time. OpenSeq2Seq currently provides building blocks for models that solve a wide range of tasks including neural machine translation, automatic speech recognition, and speech synthesis.
arXiv commentPresented at Workshop for Natural Language Processing Open Source Software (NLP-OSS), co-located with ACL2018
References in corpus (10)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Convolutional Sequence to Sequence Learning
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Mixed Precision Training
- Horovod: fast and easy distributed deep learning in TensorFlow
- Pointer Sentinel Mixture Models
- Listen, Attend and Spell
- Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
- Exploring Neural Transducers for End-to-End Speech Recognition
Cited by in corpus (13)
- NeMo: a toolkit for building AI applications using Neural Modules
- wav2letter++: The Fastest Open-source Speech Recognition System
- Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
- Mixed Precision Training With 8-bit Floating Point
- Learning Robust and Multilingual Speech Representations
- ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit
- Adaptive Loss Scaling for Mixed Precision Training
- Graph-Aware Transformer: Is Attention All Graphs Need?
- ConVoice: Real-Time Zero-Shot Voice Style Transfer with Convolutional Network
- End-to-end Anchored Speech Recognition
- Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models
- PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
- Densifying Assumed-sparse Tensors: Improving Memory Efficiency and MPI Collective Performance during Tensor Accumulation for Parallelized Training of Neural Machine Translation Models