Nonparallel Emotional Speech Conversion

Abstract

We propose a nonparallel data-driven emotional speech conversion method. It enables the transfer of emotion-related characteristics of a speech signal while preserving the speaker's identity and linguistic content. Most existing approaches require parallel data and time alignment, which is not available in most real applications. We achieve nonparallel training based on an unsupervised style transfer technique, which learns a translation model between two distributions instead of a deterministic one-to-one mapping between paired examples. The conversion model consists of an encoder and a decoder for each emotion domain. We assume that the speech signal can be decomposed into an emotion-invariant content code and an emotion-related style code in latent space. Emotion conversion is performed by extracting and recombining the content code of the source speech and the style code of the target emotion. We tested our method on a nonparallel corpora with four emotions. Both subjective and objective evaluations show the effectiveness of our approach.

arXiv commentPublished in INTERSPEECH 2019, 5 pages, 6 figures. Simulation available at http://www.jian-gao.org/emogan

References in corpus (3)

• WaveNet: A Generative Model for Raw Audio
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Language Modeling with Gated Convolutional Networks

Cited by in corpus (16)

• Emotion Intensity and its Control for Emotional Voice Conversion
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era
• Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Data
• Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversion
• DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignment
• Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset
• Neural network algorithm and its application in temperature control of distillation tower
• CycleTransGAN-EVC: A CycleGAN-based Emotional Voice Conversion Model with Transformer
• Emotional Voice Conversion: Theory, Databases and ESD
• StarGAN-based Emotional Voice Conversion for Japanese Phrases
• Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion
• Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training
• Research Progress of Convolutional Neural Network and its Application in Object Detection
• Expressive Voice Conversion: A Joint Framework for Speaker Identity and Emotional Style Transfer
• Study on the simulation control of neural network algorithm in thermally coupled distillation
• Application of Neural Network in Optimization of Chemical Process