Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models

Abstract

Deep generative neural networks have proven effective at both conditional and unconditional modeling of complex data distributions. Conditional generation enables interactive control, but creating new controls often requires expensive retraining. In this paper, we develop a method to condition generation without retraining the model. By post-hoc learning latent constraints, value functions that identify regions in latent space that generate outputs with desired attributes, we can conditionally sample from these regions with gradient-based optimization or amortized actor functions. Combining attribute constraints with a universal "realism" constraint, which enforces similarity to the data distribution, we generate realistic conditional images from an unconditional variational autoencoder. Further, using gradient-based optimization, we demonstrate identity-preserving transformations that make the minimal adjustment in latent space to modify the attributes of an image. Finally, with discrete sequences of musical notes, we demonstrate zero-shot conditional generation, learning latent constraints in the absence of labeled data or a differentiable reward function. Code with dedicated cloud instance has been made publicly available (https://goo.gl/STGMGx).

References in corpus (10)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Improved Techniques for Training GANs
• Adversarial Feature Learning
• Adversarially Learned Inference
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Adversarial Autoencoders
• Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
• Improving Variational Inference with Inverse Autoregressive Flow
• VAE with a VampPrior

Cited by in corpus (12)

• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• Generative Models for Automatic Chemical Design
• Simple and Effective VAE Training with Calibrated Decoders
• What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
• Scalable Unbalanced Optimal Transport using Generative Adversarial Networks
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
• Learning via social awareness: Improving a deep generative sketching model with facial feedback
• Encoding Musical Style with Transformer Autoencoders
• Cross Domain Image Generation through Latent Space Exploration with Adversarial Loss
• Face-to-Music Translation Using a Distance-Preserving Generative Adversarial Network with an Auxiliary Discriminator