Perception of prosodic variation for speech synthesis using an unsupervised discrete representation of F0

Abstract

In English, prosody adds a broad range of information to segment sequences, from information structure (e.g. contrast) to stylistic variation (e.g. expression of emotion). However, when learning to control prosody in text-to-speech voices, it is not clear what exactly the control is modifying. Existing research on discrete representation learning for prosody has demonstrated high naturalness, but no analysis has been performed on what these representations capture, or if they can generate meaningfully-distinct variants of an utterance. We present a phrase-level variational autoencoder with a multi-modal prior, using the mode centres as "intonation codes". Our evaluation establishes which intonation codes are perceptually distinct, finding that the intonation codes from our multi-modal latent model were significantly more distinct than a baseline using k-means clustering. We carry out a follow-up qualitative study to determine what information the codes are carrying. Most commonly, listeners commented on the intonation codes having a statement or question style. However, many other affect-related styles were also reported, including: emotional, uncertain, surprised, sarcastic, passive aggressive, and upset.

arXiv commentPublished to the 10th ISCA International Conference on Speech Prosody (SP2020)

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven Acoustic Embedding Selection

Cited by in corpus (4)

• Expressive TTS Training with Frame and Style Reconstruction Loss
• Double Articulation Analyzer with Prosody for Unsupervised Word and Phoneme Discovery
• It's not what you said, it's how you said it: discriminative perception of speech as a multichannel communication system
• Location, Location: Enhancing the Evaluation of Text-to-Speech Synthesis Using the Rapid Prosody Transcription Paradigm