BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

Abstract

We present BlockGAN, an image generative model that learns object-aware 3D scene representations directly from unlabelled 2D images. Current work on scene representation learning either ignores scene background or treats the whole scene as one object. Meanwhile, work that considers scene compositionality treats scene objects only as image patches or 2D layers with alpha maps. Inspired by the computer graphics pipeline, we design BlockGAN to learn to first generate 3D features of background and foreground objects, then combine them into 3D features for the wholes cene, and finally render them into realistic images. This allows BlockGAN to reason over occlusion and interaction between objects' appearance, such as shadow and lighting, and provides control over each object's 3D pose and identity, while maintaining image realism. BlockGAN is trained end-to-end, using only unlabelled single images, without the need for 3D geometry, pose labels, object masks, or multiple views of the same scene. Our experiments show that using explicit 3D features to represent objects allows BlockGAN to learn disentangled representations both in terms of objects (foreground and background) and their properties (pose and identity).

arXiv commentFor project page, see https://www.monkeyoverflow.com/#/blockgan/ Accepted to Conference on Neural Information Processing Systemsm, NeurIPS 2020

References in corpus (20)

• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Adversarially Learned Inference
• A simple neural network module for relational reasoning
• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• On Self Modulation for Generative Adversarial Networks
• PhotoShape: Photorealistic Materials for Large-Scale Shape Collections
• Visual Object Networks: Image Generation with Disentangled 3D Representation
• 3D-Aware Scene Manipulation via Inverse Graphics
• Generating Images Part by Part with Composite Generative Adversarial Networks
• Object-Centric Image Generation with Factored Depths, Locations, and Appearances
• How to make a pizza: Learning a compositional layer-based GAN model

Cited by in corpus (18)

• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
• SofGAN: A Portrait Image Generator with Dynamic Styling
• Object-Centric Neural Scene Rendering
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
• Generative Adversarial Transformers
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• DOVE: Learning Deformable 3D Objects by Watching Videos
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors
• Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis
• Reconstruction Bottlenecks in Object-Centric Generative Models
• Unsupervised object-centric video generation and decomposition in 3D
• Template NeRF: Towards Modeling Dense Shape Correspondences from Category-Specific Object Images
• Unsupervised Object Learning via Common Fate
• Towards a Neural Graphics Pipeline for Controllable Image Generation
• Illiterate DALL-E Learns to Compose