WESPER: Zero-shot and Realtime Whisper to Normal Voice Conversion for Whisper-based Speech Interactions
arXiv:2303.01639 · doi:10.1145/3544548.3580706
Abstract
Recognizing whispered speech and converting it to normal speech creates many possibilities for speech interaction. Because the sound pressure of whispered speech is significantly lower than that of normal speech, it can be used as a semi-silent speech interaction in public places without being audible to others. Converting whispers to normal speech also improves the speech quality for people with speech or hearing impairments. However, conventional speech conversion techniques do not provide sufficient conversion quality or require speaker-dependent datasets consisting of pairs of whispered and normal speech utterances. To address these problems, we propose WESPER, a zero-shot, real-time whisper-to-normal speech conversion mechanism based on self-supervised learning. WESPER consists of a speech-to-unit (STU) encoder, which generates hidden speech units common to both whispered and normal speech, and a unit-to-speech (UTS) decoder, which reconstructs speech from the encoded speech units. Unlike the existing methods, this conversion is user-independent and does not require a paired dataset for whispered and normal speech. The UTS decoder can reconstruct speech in any target speaker's voice from speech units, and it requires only an unlabeled target speaker's speech data. We confirmed that the quality of the speech converted from a whisper was improved while preserving its natural prosody. Additionally, we confirmed the effectiveness of the proposed approach to perform speech reconstruction for people with speech or hearing disabilities. (project page: http://lab.rekimoto.org/projects/wesper )
arXiv commentACM CHI 2023 paper
References in corpus (20)
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
- HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
- AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
- Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
- SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks
- A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion
- Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages
- StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks
- ClearBuds: Wireless Binaural Earbuds for Learning-Based Speech Enhancement
- AudioLM: a Language Modeling Approach to Audio Generation
- Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation
- Unsupervised Cross-lingual Representation Learning for Speech Recognition
- DualVoice: Speech Interaction that Discriminates between Normal and Whispered Voice Input
- The Sequence-to-Sequence Baseline for the Voice Conversion Challenge 2020: Cascading ASR and TTS
- Attention-Guided Generative Adversarial Network for Whisper to Normal Speech Conversion
- Textless Speech Emotion Conversion using Discrete and Decomposed Representations
- Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learning
- Non-autoregressive sequence-to-sequence voice conversion
- End-to-end Whispered Speech Recognition with Frequency-weighted Approaches and Pseudo Whisper Pre-training
- Whispered-to-voiced Alaryngeal Speech Conversion with Generative Adversarial Networks