Deep Convolutional Inverse Graphics Network

Abstract

This paper presents the Deep Convolution Inverse Graphics Network (DC-IGN), a model that learns an interpretable representation of images. This representation is disentangled with respect to transformations such as out-of-plane rotations and lighting variations. The DC-IGN model is composed of multiple layers of convolution and de-convolution operators and is trained using the Stochastic Gradient Variational Bayes (SGVB) algorithm. We propose a training procedure to encourage neurons in the graphics code layer to represent a specific transformation (e.g. pose or light). Given a single input image, our model can generate new images of the same object with variations in pose and lighting. We present qualitative and quantitative results of the model's efficacy at learning a 3D rendering engine.

First two authors contributed equally

References in corpus (5)

• Auto-Encoding Variational Bayes
• Disentangling Factors of Variation via Generative Entangling
• Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs
• Deep Lambertian Networks
• Inverse Graphics with Probabilistic CAD Models

Cited by in corpus (257)

• An Introduction to Variational Autoencoders
• Tutorial on Variational Autoencoders
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Disentangling by Factorising
• Autoencoding beyond pixels using a learned similarity metric
• Deep Reinforcement Learning: An Overview
• GANSpace: Discovering Interpretable GAN Controls
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Recent Advances in Autoencoder-Based Representation Learning
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• A Survey on Deep Learning Techniques for Stereo-based Depth Estimation
• Deep Appearance Models for Face Rendering
• Fader Networks: Manipulating Images by Sliding Attributes
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• Disentangling factors of variation in deep representations using adversarial training
• Learning Factorized Multimodal Representations
• Unsupervised Learning of Depth and Ego-Motion from Video
• Learning What and Where to Draw
• Harnessing Deep Neural Networks with Logic Rules
• MONet: Unsupervised Scene Decomposition and Representation
• Representation Learning by Rotating Your Faces
• Flexible Neural Representation for Physics Prediction
• Deep Learning and Model Predictive Control for Self-Tuning Mode-Locked Lasers
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
• Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images
• Toward Controlled Generation of Text
• Dense 3D Object Reconstruction from a Single Depth View
• Deep Identity-aware Transfer of Facial Attributes
• Deep Shading: Convolutional Neural Networks for Screen-Space Shading
• Isolating Sources of Disentanglement in Variational Autoencoders
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Deep Learning and Density Functional Theory
• Early Visual Concept Learning with Unsupervised Deep Learning
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• Self-supervised learning of a facial attribute embedding from video
• Unsupervised Learning of Object Keypoints for Perception and Control
• Weakly-Supervised Disentanglement Without Compromises
• Self-Supervised Intrinsic Image Decomposition
• Attribute2Image: Conditional Image Generation from Visual Attributes
• Deep Structural Causal Models for Tractable Counterfactual Inference
• On the Origin of Deep Learning
• Deep Successor Reinforcement Learning
• DeepStereo: Learning to Predict New Views from the World's Imagery
• Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
• Unsupervised Learning of Visual Structure using Predictive Generative Networks
• Learning to Infer and Execute 3D Shape Programs
• Video Frame Interpolation via Adaptive Separable Convolution
• DNA-GAN: Learning Disentangled Representations from Multi-Attribute Images
• Weakly Supervised Disentanglement with Guarantees
• RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
• Learning Disentangled Joint Continuous and Discrete Representations
• Small Sample Learning in Big Data Era
• OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer
• Multiple-Attribute Text Style Transfer
• Deferred Neural Rendering: Image Synthesis using Neural Textures
• Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
• Head2Head++: Deep Facial Attributes Re-Targeting
• GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data
• Convolutional Network for Attribute-driven and Identity-preserving Human Face Generation
• Disentangling Factors of Variation Using Few Labels
• Tackling Over-pruning in Variational Autoencoders
• Multi-View Image Generation from a Single-View
• Interpretable neural networks based on continuous-valued logic and multicriteria decision operators
• Emergence of Exploratory Look-Around Behaviors through Active Observation Completion
• Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
• A Survey of Inductive Biases for Factorial Representation-Learning
• Neural Face Editing with Intrinsic Image Disentangling
• Semi-Latent GAN: Learning to generate and modify facial images from attributes
• A Comprehensive Survey of Machine Learning Applied to Radar Signal Processing
• Video Frame Interpolation via Adaptive Convolution
• Understanding Visual Concepts with Continuation Learning
• From A to Z: Supervised Transfer of Style and Content Using Deep Neural Network Generators
• Efficient inference in occlusion-aware generative models of images
• Adversarial examples for generative models
• Towards Open-Set Identity Preserving Face Synthesis
• A Deep Learning Approach for Joint Video Frame and Reward Prediction in Atari Games
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Semantically Decomposing the Latent Spaces of Generative Adversarial Networks
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• Unsupervised Visual Attribute Transfer with Reconfigurable Generative Adversarial Networks
• The Variational Homoencoder: Learning to learn high capacity generative models from few examples
• Counterfactuals uncover the modular structure of deep generative models
• View Synthesis by Appearance Flow
• A Survey on Deep Learning Architectures for Image-based Depth Reconstruction
• Learning FRAME Models Using CNN Filters
• Learning to Generate Chairs, Tables and Cars with Convolutional Networks
• Relevance Factor VAE: Learning and Identifying Disentangled Factors
• Active Long Term Memory Networks
• Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry
• Learning Disentangled Representations with Reference-Based Variational Autoencoders
• 3D Shape Induction from 2D Views of Multiple Objects
• Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning
• Gaussian Process Prior Variational Autoencoders
• Disentangling Space and Time in Video with Hierarchical Variational Auto-encoders
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder
• JADE: Joint Autoencoders for Dis-Entanglement
• Every Smile is Unique: Landmark-Guided Diverse Smile Generation
• Generative causal explanations of black-box classifiers
• Ranking CGANs: Subjective Control over Semantic Image Attributes
• Explicit Disentanglement of Appearance and Perspective in Generative Models
• Dual Swap Disentangling
• Geometry-aware Deep Network for Single-Image Novel View Synthesis
• Semi-Supervised StyleGAN for Disentanglement Learning
• 3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators
• Theory and Evaluation Metrics for Learning Disentangled Representations
• Disentangling Latent Hands for Image Synthesis and Pose Estimation
• Deep Outdoor Illumination Estimation
• Denoising without access to clean data using a partitioned autoencoder
• ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Driving
• Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction
• Disentangling Factors of Variation by Mixing Them
• Jointly Deep Multi-View Learning for Clustering Analysis
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
• Bounded Information Rate Variational Autoencoders
• InverseFaceNet: Deep Monocular Inverse Face Rendering
• Learning to Write Stylized Chinese Characters by Reading a Handful of Examples
• Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop
• Language Style Transfer from Sentences with Arbitrary Unknown Styles
• DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation
• DeepContext: Context-Encoding Neural Pathways for 3D Holistic Scene Understanding
• Disentangling Style and Content in Anime Illustrations
• On the Road with 16 Neurons: Mental Imagery with Bio-inspired Deep Neural Networks
• Natural Language Generation with Neural Variational Models
• Disentangling Controllable and Uncontrollable Factors of Variation by Interacting with the World
• Towards Automatic Image Editing: Learning to See another You
• Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing
• Disentangled Representations in Neural Models
• Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image
• Learning Diverse Image Colorization
• InverseRenderNet: Learning single image inverse rendering
• On the Fairness of Disentangled Representations
• Analyzing and Improving Neural Networks by Generating Semantic Counterexamples through Differentiable Rendering
• Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations
• CAGAN: Text-To-Image Generation with Combined Attention GANs
• Recapture as You Want
• Compositional Visual Generation and Inference with Energy Based Models
• Inducing Interpretable Representations with Variational Autoencoders
• Label Denoising Adversarial Network (LDAN) for Inverse Lighting of Face Images
• Learning Variations in Human Motion via Mix-and-Match Perturbation
• Learning Compositional Visual Concepts with Mutual Consistency
• Factorized Discriminant Analysis for Genetic Signatures of Neuronal Phenotypes
• Semi-supervised Viewpoint Estimation with Geometry-aware Conditional Generation
• Investigation of Using Disentangled and Interpretable Representations for One-shot Cross-lingual Voice Conversion
• Deep Variational Inference Without Pixel-Wise Reconstruction
• Shadow Transfer: Single Image Relighting For Urban Road Scenes
• Robust Ordinal VAE: Employing Noisy Pairwise Comparisons for Disentanglement
• Automatic Relevance Determination For Deep Generative Models
• CoDeGAN: Contrastive Disentanglement for Generative Adversarial Network
• Material Editing Using a Physically Based Rendering Network
• Commutative Lie Group VAE for Disentanglement Learning
• CDFI: Compression-Driven Network Design for Frame Interpolation
• PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes
• Visuomotor Understanding for Representation Learning of Driving Scenes
• Cerberus: A Multi-headed Derenderer
• Improving VAEs' Robustness to Adversarial Attack
• A Lightweight Neural Network for Monocular View Generation with Occlusion Handling
• Unpaired Pose Guided Human Image Generation
• BézierSketch: A generative model for scalable vector sketches
• Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackers
• A Neural Rendering Framework for Free-Viewpoint Relighting
• Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders
• Measuring the Biases and Effectiveness of Content-Style Disentanglement
• Independence Promoted Graph Disentangled Networks
• Attentive Action and Context Factorization
• dMelodies: A Music Dataset for Disentanglement Learning
• Data Augmentation for Enhancing EEG-based Emotion Recognition with Deep Generative Models
• A Neural-Symbolic Architecture for Inverse Graphics Improved by Lifelong Meta-Learning
• An Uncertain Future: Forecasting from Static Images using Variational Autoencoders
• DISSECT: Disentangled Simultaneous Explanations via Concept Traversals
• Hyperprior Induced Unsupervised Disentanglement of Latent Representations
• Towards Disentangled Representations for Human Retargeting by Multi-view Learning
• Latent feature disentanglement for 3D meshes
• 3D Shape Reconstruction from a Single 2D Image via 2D-3D Self-Consistency
• Learning Neural Light Transport
• Class-Distinct and Class-Mutual Image Generation with GANs
• Capsule network with shortcut routing
• DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability
• Soft Constraints for Inference with Declarative Knowledge
• Cloud2Curve: Generation and Vectorization of Parametric Sketches
• Adversarial Disentanglement with Grouped Observations
• Improving Bi-directional Generation between Different Modalities with Variational Autoencoders
• On Qualitative Shape Inferences: a journey from geometry to topology
• CDVAE: Co-embedding Deep Variational Auto Encoder for Conditional Variational Generation
• LAVAE: Disentangling Location and Appearance
• Controllable Top-down Feature Transformer
• Gotta Adapt 'Em All: Joint Pixel and Feature-Level Domain Adaptation for Recognition in the Wild
• Affine Variational Autoencoders: An Efficient Approach for Improving Generalization and Robustness to Distribution Shift
• Deep Structure for end-to-end inverse rendering
• Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth
• Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
• Learning disentangled representations via product manifold projection
• Rotation-Invariant Autoencoders for Signals on Spheres
• Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval
• Learning to Manipulate Individual Objects in an Image
• A Variational Feature Encoding Method of 3D Object for Probabilistic Semantic SLAM
• WeLa-VAE: Learning Alternative Disentangled Representations Using Weak Labels
• Deep Anomaly Detection by Residual Adaptation
• Learning Controllable Disentangled Representations with Decorrelation Regularization
• Thinking Outside the Pool: Active Training Image Creation for Relative Attributes
• DeepCoder: Semi-parametric Variational Autoencoders for Automatic Facial Action Coding
• High-dimensional Assisted Generative Model for Color Image Restoration
• Deep Reflectance Maps
• Harnessing value from data science in business: ensuring explainability and fairness of solutions
• Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks
• Disentanglement Learning via Topology
• Deep generative demixing: Recovering Lipschitz signals from noisy subgaussian mixtures
• Cognitive computational neuroscience
• Wasserstein Introspective Neural Networks
• Learning to Recognize Objects by Retaining other Factors of Variation
• Neural Articulated Radiance Field
• Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning
• PI-GAN: Learning Pose Independent representations for multiple profile face synthesis
• Deep disentangled representations for volumetric reconstruction
• Interpretable Intuitive Physics Model
• Learning a Representation Map for Robot Navigation using Deep Variational Autoencoder
• Joint Learning of Portrait Intrinsic Decomposition and Relighting
• Open-Ended Content-Style Recombination Via Leakage Filtering
• Inferring 3D Shapes from Image Collections using Adversarial Networks
• Disentangling Video with Independent Prediction
• Invariance-based Multi-Clustering of Latent Space Embeddings for Equivariant Learning
• VUNet: Dynamic Scene View Synthesis for Traversability Estimation using an RGB Camera
• Deep Generative Model with Beta Bernoulli Process for Modeling and Learning Confounding Factors
• Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patterns
• Learning to Reconstruct and Segment 3D Objects
• De-rendering the World's Revolutionary Artefacts
• Disentangling Representations of Text by Masking Transformers
• Illiterate DALL-E Learns to Compose
• Quantised Transforming Auto-Encoders: Achieving Equivariance to Arbitrary Transformations in Deep Networks
• Video Autoencoder: self-supervised disentanglement of static 3D structure and motion
• An inverse modeling method to estimate undertain spatial configurations from 2d information and time-based visual discriminations
• Automatic generation of object shapes with desired functionalities
• Variational learning across domains with triplet information
• Face representation by deep learning: a linear encoding in a parameter space?
• SimVAE: Simulator-Assisted Training forInterpretable Generative Models
• Latent Transformations for Object View Points Synthesis
• Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrieval
• Inner Space Preserving Generative Pose Machine
• Learning to Infer User Interface Attributes from Images
• Deep Unsupervised Common Representation Learning for LiDAR and Camera Data using Double Siamese Networks
• MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
• From abstract items to latent spaces to observed data and back: Compositional Variational Auto-Encoder
• Multiple View Generation and Classification of Mid-wave Infrared Images using Deep Learning
• Cross-Identity Motion Transfer for Arbitrary Objects through Pose-Attentive Video Reassembling
• Incremental Scene Synthesis
• Decoder-free Robustness Disentanglement without (Additional) Supervision
• Traversing Latent Space using Decision Ferns
• Games for Fairness and Interpretability
• Not Only Look But Observe: Variational Observation Model of Scene-Level 3D Multi-Object Understanding for Probabilistic SLAM
• Dual Encoder-Decoder based Generative Adversarial Networks for Disentangled Facial Representation Learning
• Adaptive Multiscale Illumination-Invariant Feature Representation for Undersampled Face Recognition
• Attribute-controlled face photo synthesis from simple line drawing
• Outdoor inverse rendering from a single image using multiview self-supervision
• Learning Equivariant Representations
• Difference-in-Differences: Bridging Normalization and Disentanglement in PG-GAN