Authoring image decompositions with generative models

Abstract

We show how to extend traditional intrinsic image decompositions to incorporate further layers above albedo and shading. It is hard to obtain data to learn a multi-layer decomposition. Instead, we can learn to decompose an image into layers that are "like this" by authoring generative models for each layer using proxy examples that capture the Platonic ideal (Mondrian images for albedo; rendered 3D primitives for shading; material swatches for shading detail). Our method then generates image layers, one from each model, that explain the image. Our approach rests on innovation in generative models for images. We introduce a Convolutional Variational Auto Encoder (conv-VAE), a novel VAE architecture that can reconstruct high fidelity images. The approach is general, and does not require that layers admit a physical interpretation.

References in corpus (10)

• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Tutorial on Variational Autoencoders
• Fully Convolutional Networks for Semantic Segmentation
• Autoencoding beyond pixels using a learned similarity metric
• Texture Synthesis Using Convolutional Neural Networks
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
• Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
• Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression