Unsupervised Audiovisual Synthesis via Exemplar Autoencoders
arXiv:2001.04463
Abstract
We present an unsupervised approach that converts the input speech of any individual into audiovisual streams of potentially-infinitely many output speakers. Our approach builds on simple autoencoders that project out-of-sample data onto the distribution of the training set. We use Exemplar Autoencoders to learn the voice, stylistic prosody, and visual appearance of a specific target exemplar speech. In contrast to existing methods, the proposed approach can be easily extended to an arbitrarily large number of speakers and styles using only 3 minutes of target audio-video data, without requiring {\em any} training data for the input speaker. To do so, we learn audiovisual bottleneck representations that capture the structured linguistic content of speech. We outperform prior approaches on both audio and video synthesis, and provide extensive qualitative analysis on our project page -- https://www.cs.cmu.edu/~exemplar-ae/.
ICLR 2021; Project page -- https://www.cs.cmu.edu/~exemplar-ae/
References in corpus (9)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Sequence to Sequence Learning with Neural Networks
- WaveNet: A Generative Model for Raw Audio
- Neural Discrete Representation Learning
- VoxCeleb2: Deep Speaker Recognition
- AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
- Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
- ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder
- One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization