SpeechNet: A Universal Modularized Model for Speech Processing Tasks
arXiv:2105.03070
Abstract
There is a wide variety of speech processing tasks ranging from extracting content information from speech signals to generating speech signals. For different tasks, model networks are usually designed and tuned separately. If a universal model can perform multiple speech processing tasks, some tasks might be improved with the related abilities learned from other tasks. The multi-task learning of a wide variety of speech processing tasks with a universal model has not been studied. This paper proposes a universal modularized model, SpeechNet, which treats all speech processing tasks into a speech/text input and speech/text output format. We select five essential speech processing tasks for multi-task learning experiments with SpeechNet. We show that SpeechNet learns all of the above tasks, and we further analyze which tasks can be improved by other tasks. SpeechNet is modularized and flexible for incorporating more modules, tasks, or training approaches in the future. We release the code and experimental settings to facilitate the research of modularized universal models and multi-task learning of speech processing tasks.
References in corpus (18)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- An Overview of Multi-Task Learning in Deep Neural Networks
- VoxCeleb: a large-scale speaker identification dataset
- A Structured Self-attentive Sentence Embedding
- Multi-Task Learning for Dense Prediction Tasks: A Survey
- A Comparative Study on Transformer vs RNN in Speech Applications
- A Survey on Multi-Task Learning
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
- The Natural Language Decathlon: Multitask Learning as Question Answering
- One Model To Learn Them All
- AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
- Gradient Surgery for Multi-Task Learning
- The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods
- Improving Sequence-to-Sequence Acoustic Modeling by Adding Text-Supervision
- Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech
- T-vectors: Weakly Supervised Speaker Identification Using Hierarchical Transformer Model