Investigating Object Compositionality in Generative Adversarial Networks

Abstract

Deep generative models seek to recover the process with which the observed data was generated. They may be used to synthesize new samples or to subsequently extract representations. Successful approaches in the domain of images are driven by several core inductive biases. However, a bias to account for the compositional way in which humans structure a visual scene in terms of objects has frequently been overlooked. In this work, we investigate object compositionality as an inductive bias for Generative Adversarial Networks (GANs). We present a minimal modification of a standard generator to incorporate this inductive bias and find that it reliably learns to generate images as compositions of objects. Using this general design as a backbone, we then propose two useful extensions to incorporate dependencies among objects and background. We extensively evaluate our approach on several multi-object image datasets and highlight the merits of incorporating structure for representation learning purposes. In particular, we find that our structured GANs are better at generating multi-object images that are more faithful to the reference distribution. More so, we demonstrate how, by leveraging the structure of the learned generative process, one can `invert' the learned generative model to perform unsupervised instance segmentation. On the challenging CLEVR dataset, it is shown how our approach is able to improve over other recent purely unsupervised object-centric approaches to image generation.

arXiv commentA preliminary version of this work (arXiv v1) appeared under the title "A Case for Object Compositionality in Deep Generative Models of Images" as a workshop paper at the NeurIPS2018 workshop on "Modeling the Physical World: Perception, Learning, and Control", and at the NeurIPS2018 workshop on "Relational Representation Learning"

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Neural Message Passing for Quantum Chemistry
• Spectral Normalization for Generative Adversarial Networks
• Pixel Recurrent Neural Networks
• Adversarially Learned Inference
• Layer Normalization
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Multi-Object Representation Learning with Iterative Variational Inference
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
• Relational Deep Reinforcement Learning
• Learning Latent Permutations with Gumbel-Sinkhorn Networks
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Unsupervised Object Segmentation by Redrawing
• Neural Expectation Maximization
• Object Discovery with a Copy-Pasting GAN
• Generating Images Part by Part with Composite Generative Adversarial Networks
• A Large-Scale Study on Regularization and Normalization in GANs
• Generating Multiple Objects at Spatially Distinct Locations
• Relational recurrent neural networks
• Deep Structured Generative Models
• Generative Adversarial Networks are Special Cases of Artificial Curiosity (1990) and also Closely Related to Predictability Minimization (1991)

Cited by in corpus (13)

• Object-Centric Learning with Slot Attention
• Unsupervised Object Segmentation by Redrawing
• On the Binding Problem in Artificial Neural Networks
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• Generative Adversarial Transformers
• Emergence of Object Segmentation in Perturbed Generative Models
• Unsupervised Learning of Compositional Energy Concepts
• Compositional Transformers for Scene Generation
• Compositional Visual Generation and Inference with Energy Based Models
• Generalization and Robustness Implications in Object-Centric Learning
• Unsupervised object-centric video generation and decomposition in 3D
• ORGAN: Object-Centric Representation Learning using Cycle Consistent Generative Adversarial Networks
• Illiterate DALL-E Learns to Compose