Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
arXiv:2110.02891
Abstract
Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, under an unsupervised-style setting, typical training algorithms for controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but unpaired samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. The proposed method is simple yet effective, where we use a style transformation module to transfer target style information into an unrelated style input. This method enables training using unpaired content and style samples and thereby mitigate the training-inference mismatch. We apply style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. We conduct thorough evaluation, including both quantitative and qualitative user studies. Our results show that by mitigating the training-inference mismatch with the proposed style equalization, we achieve style replication scores comparable to real data in our user studies.
arXiv commentICML 2022
References in corpus (15)
- Auto-Encoding Variational Bayes
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Gaussian Error Linear Units (GELUs)
- VoxCeleb2: Deep Speaker Recognition
- A Recurrent Latent Variable Model for Sequential Data
- Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
- Making Convolutional Networks Shift-Invariant Again
- AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
- Tandem Assessment of Spoofing Countermeasures and Automatic Speaker Verification: Fundamentals
- Jukebox: A Generative Model for Music
- AdaSpeech: Adaptive Text to Speech for Custom Voice
- Unsupervised Speech Decomposition via Triple Information Bottleneck
- End-to-End Adversarial Text-to-Speech
- Improved Autoregressive Modeling with Distribution Smoothing