The 2020 ESPnet update: new features, broadened applications, performance improvements, and future plans

Abstract

This paper describes the recent development of ESPnet (https://github.com/espnet/espnet), an end-to-end speech processing toolkit. This project was initiated in December 2017 to mainly deal with end-to-end speech recognition experiments based on sequence-to-sequence modeling. The project has grown rapidly and now covers a wide range of speech processing applications. Now ESPnet also includes text to speech (TTS), voice conversation (VC), speech translation (ST), and speech enhancement (SE) with support for beamforming, speech separation, denoising, and dereverberation. All applications are trained in an end-to-end manner, thanks to the generic sequence to sequence modeling properties, and they can be further integrated and jointly optimized. Also, ESPnet provides reproducible all-in-one recipes for these applications with state-of-the-art performance in various benchmarks by incorporating transformer, advanced data augmentation, and conformer. This project aims to provide up-to-date speech processing experience to the community so that researchers in academia and various industry scales can develop their technologies collaboratively.

References in corpus (22)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Sequence to Sequence Learning with Neural Networks
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Attention-Based Models for Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• A Comparative Study on Transformer vs RNN in Speech Applications
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• NeMo: a toolkit for building AI applications using Neural Modules
• CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition
• RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
• Recent Developments on ESPnet Toolkit Boosted by Conformer
• Towards a Competitive End-to-End Speech Recognition for CHiME-6 Dinner Party Transcription
• Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition
• Espresso: A Fast End-to-end Neural Speech Recognition Toolkit
• ESPnet-ST: All-in-One Speech Translation Toolkit
• End-to-End Speech Recognition: A review for the French Language
• Improved Mask-CTC for Non-Autoregressive End-to-End ASR
• ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integration
• Directional ASR: A New Paradigm for E2E Multi-Speaker Speech Recognition with Source Localization

Cited by in corpus (4)

• Dynamic Acoustic Unit Augmentation With BPE-Dropout for Low-Resource End-to-End Speech Recognition
• Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0
• Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement
• A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion