Unsupervised Speech Decomposition via Triple Information Bottleneck

Abstract

Speech information can be roughly decomposed into four components: language content, timbre, pitch, and rhythm. Obtaining disentangled representations of these components is useful in many speech analysis and generation applications. Recently, state-of-the-art voice conversion systems have led to speech representations that can disentangle speaker-dependent and independent information. However, these systems can only disentangle timbre, while information about pitch, rhythm and content is still mixed together. Further disentangling the remaining speech components is an under-determined problem in the absence of explicit annotations for each component, which are difficult and expensive to obtain. In this paper, we propose SpeechSplit, which can blindly decompose speech into its four components by introducing three carefully designed information bottlenecks. SpeechSplit is among the first algorithms that can separately perform style transfer on timbre, pitch and rhythm without text labels. Our code is publicly available at https://github.com/auspicious3000/SpeechSplit.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
• F0-consistent many-to-many non-parallel voice conversion via conditional autoencoder
• Voice Conversion Based on Cross-Domain Features Using Variational Auto Encoders
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
• ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder
• StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks
• Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation

Cited by in corpus (21)

• A Survey on Neural Speech Synthesis
• "Hello, It's Me": Deep Learning-based Speech Synthesis Attacks in the Real World
• Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
• Review of end-to-end speech synthesis technology based on deep learning
• Global Rhythm Style Transfer Without Text Transcriptions
• Learning and controlling the source-filter representation of speech with a variational autoencoder
• Recognizing Predictive Substructures with Subgraph Information Bottleneck
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• ConVoice: Real-Time Zero-Shot Voice Style Transfer with Convolutional Network
• AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion
• Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction
• Adversarially learning disentangled speech representations for robust multi-factor voice conversion
• Graph Information Bottleneck for Subgraph Recognition
• Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
• STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech
• Zero-shot Singing Technique Conversion
• Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narration
• How Speech is Recognized to Be Emotional - A Study Based on Information Decomposition
• Towards Universal Neural Vocoding with a Multi-band Excited WaveNet
• How Far Are We from Robust Voice Conversion: A Survey