Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis

Abstract

In this work, we propose "global style tokens" (GSTs), a bank of embeddings that are jointly trained within Tacotron, a state-of-the-art end-to-end speech synthesis system. The embeddings are trained with no explicit labels, yet learn to model a large range of acoustic expressiveness. GSTs lead to a rich set of significant results. The soft interpretable "labels" they generate can be used to control synthesis in novel ways, such as varying speed and speaking style - independently of the text content. They can also be used for style transfer, replicating the speaking style of a single audio clip across an entire long-form text corpus. When trained on noisy, unlabeled found data, GSTs learn to factorize noise and speaker identity, providing a path towards highly scalable but robust speech synthesis.

References in corpus (7)

• WaveNet: A Generative Model for Raw Audio
• Neural Discrete Representation Learning
• A note on the evaluation of generative models
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning

Cited by in corpus (128)

• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• Emotion Intensity and its Control for Emotional Voice Conversion
• Spoken Language Interaction with Robots: Research Issues and Recommendations, Report from the NSF Future Directions Workshop
• DurIAN: Duration Informed Attention Network For Multimodal Synthesis
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era
• AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• Multi-reference Tacotron by Intercross Training for Style Disentangling,Transfer and Control in Speech Synthesis
• Training Neural Speech Recognition Systems with Synthetic Speech Augmentation
• Data Augmentation for End-to-end Code-switching Speech Recognition
• Modeling Multi-speaker Latent Space to Improve Neural TTS: Quick Enrolling New Speaker and Enhancing Premium Voice
• Deep MOS Predictor for Synthetic Speech Using Cluster-Based Modeling
• Modeling Prosodic Phrasing with Multi-Task Learning in Tacotron-based TTS
• Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement
• Parallel Tacotron: Non-Autoregressive and Controllable TTS
• Review of end-to-end speech synthesis technology based on deep learning
• Expressive TTS Training with Frame and Style Reconstruction Loss
• One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization
• Expressive Neural Voice Cloning
• Perception of prosodic variation for speech synthesis using an unsupervised discrete representation of F0
• Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data
• Hierarchical Emotion Prediction and Control in Text-to-Speech Synthesis
• Semi-Supervised Generative Modeling for Controllable Speech Synthesis
• Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation
• Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features
• Rich Prosody Diversity Modelling with Phone-level Mixture Density Network
• Maximizing Mutual Information for Tacotron
• Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset
• ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit
• Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech
• Multi-Target Emotional Voice Conversion With Neural Vocoders
• Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling
• Emotional Prosody Control for Speech Generation
• Phonetic Posteriorgrams based Many-to-Many Singing Voice Conversion via Adversarial Training
• MuSE-SVS: Multi-Singer Emotional Singing Voice Synthesizer that Controls Emotional Intensity
• Exploring emotional prototypes in a high dimensional TTS latent space
• Controllable neural text-to-speech synthesis using intuitive prosodic features
• Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model
• Noise Robust TTS for Low Resource Speakers using Pre-trained Model and Speech Enhancement
• OverFlow: Putting flows on top of neural transducers for better TTS
• SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker text-to-speech
• Effective parameter estimation methods for an ExcitNet model in generative text-to-speech systems
• The 2020 ESPnet update: new features, broadened applications, performance improvements, and future plans
• On granularity of prosodic representations in expressive text-to-speech
• Controllable speech synthesis by learning discrete phoneme-level prosodic representations
• Interactive Text-to-Speech System via Joint Style Analysis
• Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis
• Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis
• PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS
• Spectrogram Inpainting for Interactive Generation of Instrument Sounds
• Word-Level Style Control for Expressive, Non-attentive Speech Synthesis
• An Exploratory Study on Perceptual Spaces of the Singing Voice
• Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability
• Learning latent representations for style control and transfer in end-to-end speech synthesis
• A Variational Prosody Model for Mapping the Context-Sensitive Variation of Functional Prosodic Prototypes
• Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition
• Robust and fine-grained prosody control of end-to-end speech synthesis
• fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
• Attention Forcing for Sequence-to-sequence Model Training
• CUHK-EE Voice Cloning System for ICASSP 2021 M2VoC Challenge
• Towards Fine-Grained Prosody Control for Voice Conversion
• Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis
• M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention
• Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
• Investigation of learning abilities on linguistic features in sequence-to-sequence text-to-speech synthesis
• Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency
• Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck
• Adversarially learning disentangled speech representations for robust multi-factor voice conversion
• Noise Tokens: Learning Neural Noise Templates for Environment-Aware Speech Enhancement
• NWT: Towards natural audio-to-video generation with representation learning
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021
• Forward-Backward Decoding for Regularizing End-to-End TTS
• GraphPB: Graphical Representations of Prosody Boundary in Speech Synthesis
• Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis
• Pitchtron: Towards audiobook generation from ordinary people's voices
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation
• Speech Rhythm-Based Speaker Embeddings Extraction from Phonemes and Phoneme Duration for Multi-Speaker Speech Synthesis
• Attention Forcing for Machine Translation
• STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech
• Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control
• Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesis
• End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning
• MASS: Multi-task Anthropomorphic Speech Synthesis Framework
• Transferring Source Style in Non-Parallel Voice Conversion
• A unified sequence-to-sequence front-end model for Mandarin text-to-speech synthesis
• Audiovisual Speech Synthesis using Tacotron2
• A Neural Network-Based Linguistic Similarity Measure for Entrainment in Conversations
• Half-Truth: A Partially Fake Audio Detection Dataset
• In Other News: A Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data
• Learn2Sing: Target Speaker Singing Voice Synthesis by learning from a Singing Teacher
• Whispered and Lombard Neural Speech Synthesis
• Rhythm-controllable Attention with High Robustness for Long Sentence Speech Synthesis
• PPSpeech: Phrase based Parallel End-to-End TTS System
• Variation Network: Learning High-level Attributes for Controlled Input Manipulation
• On Investigation of Unsupervised Speech Factorization Based on Normalization Flow
• Handwriting styles: benchmarks and evaluation metrics
• Wasserstein GAN and Waveform Loss-based Acoustic Model Training for Multi-speaker Text-to-Speech Synthesis Systems Using a WaveNet Vocoder
• Incorporating Symbolic Sequential Modeling for Speech Enhancement
• Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions
• Model See Model Do: Speech-Driven Facial Animation with Style Control
• Deep generative factorization for speech signal
• Neural voice cloning with a few low-quality samples
• Speech BERT Embedding For Improving Prosody in Neural TTS
• GANtron: Emotional Speech Synthesis with Generative Adversarial Networks
• Low-Latency Incremental Text-to-Speech Synthesis with Distilled Context Prediction Network
• Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech
• Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representations
• Automatic Documentation of ICD Codes with Far-Field Speech Recognition
• An investigation of phrase break prediction in an End-to-End TTS system
• Investigating context features hidden in End-to-End TTS
• Emotional speech synthesis with rich and granularized control
• Enhancing audio quality for expressive Neural Text-to-Speech
• A Fully Time-domain Neural Model for Subband-based Speech Synthesizer
• Learning from Multiple Sources for Data-to-Text and Text-to-Data
• Mathematical Vocoder Algorithm : Modified Spectral Inversion for Efficient Neural Speech Synthesis
• A Neural Text-to-Speech Model Utilizing Broadcast Data Mixed with Background Music
• Learning Speaker Embedding from Text-to-Speech
• Improving Accent Conversion with Reference Encoder and End-To-End Text-To-Speech
• Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis
• Enhancing Low-Quality Voice Recordings Using Disentangled Channel Factor and Neural Waveform Model
• The IQIYI System for Voice Conversion Challenge 2020
• Domain-adversarial training of multi-speaker TTS
• Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences