Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
arXiv:2110.14513
Abstract
We present a neural analysis and synthesis (NANSY) framework that can manipulate voice, pitch, and speed of an arbitrary speech signal. Most of the previous works have focused on using information bottleneck to disentangle analysis features for controllable synthesis, which usually results in poor reconstruction quality. We address this issue by proposing a novel training strategy based on information perturbation. The idea is to perturb information in the original input signal (e.g., formant, pitch, and frequency response), thereby letting synthesis networks selectively take essential attributes to reconstruct the input signal. Because NANSY does not need any bottleneck structures, it enjoys both high reconstruction quality and controllability. Furthermore, NANSY does not require any labels associated with speech data such as text and speaker information, but rather uses a new set of analysis features, i.e., wav2vec feature and newly proposed pitch feature, Yingram, which allows for fully self-supervised training. Taking advantage of fully self-supervised training, NANSY can be easily extended to a multilingual setting by simply training it with a multilingual dataset. The experiments show that NANSY can achieve significant improvement in performance in several applications such as zero-shot voice conversion, pitch shift, and time-scale modification.
Neural Information Processing Systems (NeurIPS) 2021
References in corpus (10)
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Neural Discrete Representation Learning
- Unsupervised Data Augmentation for Consistency Training
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification
- AdaSpeech: Adaptive Text to Speech for Custom Voice
- DDSP: Differentiable Digital Signal Processing
- Exploring wav2vec 2.0 on speaker verification and language identification
- From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech
- What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure