Emergence of Object Segmentation in Perturbed Generative Models

Abstract

We introduce a novel framework to build a model that can learn how to segment objects from a collection of images without any human annotation. Our method builds on the observation that the location of object segments can be perturbed locally relative to a given background without affecting the realism of a scene. Our approach is to first train a generative model of a layered scene. The layered representation consists of a background image, a foreground image and the mask of the foreground. A composite image is then obtained by overlaying the masked foreground image onto the background. The generative model is trained in an adversarial fashion against a discriminator, which forces the generative model to produce realistic composite images. To force the generator to learn a representation where the foreground layer corresponds to an object, we perturb the output of the generative model by introducing a random shift of both the foreground image and mask relative to the background. Because the generator is unaware of the shift before computing its output, it must produce layered representations that are realistic for any such random perturbation. Finally, we learn to segment an image by defining an autoencoder consisting of an encoder, which we train, and the pre-trained generator as the decoder, which we freeze. The encoder maps an image to a feature vector, which is fed as input to the generator to give a composite image matching the original input image. Because the generator outputs an explicit layered representation of the scene, the encoder learns to detect and segment objects. We demonstrate this framework on real images of several object categories.

arXiv comment33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Spotlight presentation

References in corpus (9)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Multi-Object Representation Learning with Iterative Variational Inference
• W-Net: A Deep Model for Fully Unsupervised Image Segmentation
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
• MONet: Unsupervised Scene Decomposition and Representation
• Generating Images Part by Part with Composite Generative Adversarial Networks
• Investigating Object Compositionality in Generative Adversarial Networks
• SEIGAN: Towards Compositional Image Generation by Simultaneously Learning to Segment, Enhance, and Inpaint

Cited by in corpus (5)

• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• Reconstruction Bottlenecks in Object-Centric Generative Models
• ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation
• Deformation-aware Unpaired Image Translation for Pose Estimation on Laboratory Animals
• SS-CPGAN: Self-Supervised Cut-and-Pasting Generative Adversarial Network for Object Segmentation