Latent Translation: Crossing Modalities by Bridging Generative Models

Abstract

End-to-end optimization has achieved state-of-the-art performance on many specific problems, but there is no straight-forward way to combine pretrained models for new problems. Here, we explore improving modularity by learning a post-hoc interface between two existing models to solve a new task. Specifically, we take inspiration from neural machine translation, and cast the challenging problem of cross-modal domain transfer as unsupervised translation between the latent spaces of pretrained deep generative models. By abstracting away the data representation, we demonstrate that it is possible to transfer across different modalities (e.g., image-to-audio) and even different types of generative models (e.g., VAE-to-GAN). We compare to state-of-the-art techniques and find that a straight-forward variational autoencoder is able to best bridge the two generative models through learning a shared latent space. We can further impose supervised alignment of attributes in both domains with a classifier in the shared latent space. Through qualitative and quantitative evaluations, we demonstrate that locality and semantic alignment are preserved through the transfer process, as indicated by high transfer accuracies and smooth interpolations within a class. Finally, we show this modular structure speeds up training of new interface models by several orders of magnitude by decoupling it from expensive retraining of base generative models.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Unsupervised Image-to-Image Translation Networks
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• GANSynth: Adversarial Neural Audio Synthesis
• Phrase-Based & Neural Unsupervised Machine Translation
• Demystifying MMD GANs
• CycleGAN, a Master of Steganography
• Video-to-Video Synthesis
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• A Universal Music Translation Network
• Twin-GAN -- Unpaired Cross-Domain Image Translation with Weight-Sharing GANs

Cited by in corpus (5)

• Multimodal Few-Shot Learning with Frozen Language Models
• Optimal Unsupervised Domain Translation
• Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Experts
• Learning Audio-Visual Correlations from Variational Cross-Modal Generation
• Face-to-Music Translation Using a Distance-Preserving Generative Adversarial Network with an Auxiliary Discriminator