Compositional Visual Generation and Inference with Energy Based Models
arXiv:2004.06030
Abstract
A vital aspect of human intelligence is the ability to compose increasingly complex concepts out of simpler ideas, enabling both rapid learning and adaptation of knowledge. In this paper we show that energy-based models can exhibit this ability by directly combining probability distributions. Samples from the combined distribution correspond to compositions of concepts. For example, given a distribution for smiling faces, and another for male faces, we can combine them to generate smiling male faces. This allows us to generate natural images that simultaneously satisfy conjunctions, disjunctions, and negations of concepts. We evaluate compositional generation abilities of our model on the CelebA dataset of natural faces and synthetic 3D scene images. We also demonstrate other unique advantages of our model, such as the ability to continually learn and incorporate new concepts, or infer compositions of concept properties underlying an image.
arXiv commentNeurIPS 2020 Spotlight; Website at https://energy-based-model.github.io/compositional-generation-inference/
References in corpus (19)
- Overcoming catastrophic forgetting in neural networks
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Pixel Recurrent Neural Networks
- DRAW: A Recurrent Neural Network For Image Generation
- Searching for Activation Functions
- Deep Convolutional Inverse Graphics Network
- Multi-Object Representation Learning with Iterative Variational Inference
- A Theory of Generative ConvNet
- Implicit Generation and Generalization in Energy-Based Models
- Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
- SCAN: Learning Hierarchical Compositional Visual Concepts
- Deep Directed Generative Models with Energy-Based Probability Estimation
- Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions
- Investigating Object Compositionality in Generative Adversarial Networks
- Real or Fake? Learning to Discriminate Machine from Human Generated Text
- Improved Contrastive Divergence Training of Energy Based Models
- Measuring Compositionality in Representation Learning
- Meta-Learning Deep Energy-Based Memory Models
- Emerging Disentanglement in Auto-Encoder Based Unsupervised Image Content Transfer