Discovering Hidden Factors of Variation in Deep Networks

Abstract

Deep learning has enjoyed a great deal of success because of its ability to learn useful features for tasks such as classification. But there has been less exploration in learning the factors of variation apart from the classification signal. By augmenting autoencoders with simple regularization terms during training, we demonstrate that standard deep architectures can discover and explicitly represent factors of variation beyond those relevant for categorization. We introduce a cross-covariance penalty (XCov) as a method to disentangle factors like handwriting style for digits and subject identity in faces. We demonstrate this on the MNIST handwritten digit database, the Toronto Faces Database (TFD) and the Multi-PIE dataset by generating manipulated instances of the data. Furthermore, we demonstrate these deep networks can extrapolate `hidden' variation in the supervised signal.

arXiv commentPresented at International Conference on Learning Representations 2015 Workshop

References in corpus (6)

• ADADELTA: An Adaptive Learning Rate Method
• Going Deeper with Convolutions
• Deep Generative Stochastic Networks Trainable by Backprop
• Better Mixing via Deep Representations
• Pylearn2: a machine learning research library
• Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods

Cited by in corpus (57)

• Visual Reinforcement Learning with Imagined Goals
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• Disentangling factors of variation in deep representations using adversarial training
• Learning Factorized Multimodal Representations
• DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
• Unsupervised Learning via Meta-Learning
• Isolating Sources of Disentanglement in Variational Autoencoders
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Early Visual Concept Learning with Unsupervised Deep Learning
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Learning Disentangled Representations in the Imaging Domain
• Weakly-Supervised Disentanglement Without Compromises
• Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research
• Disentangling Disentanglement in Variational Autoencoders
• Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
• GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data
• Disentangling Factors of Variation Using Few Labels
• Disentangled Representations for Short-Term and Long-Term Person Re-Identification
• A Survey of Inductive Biases for Factorial Representation-Learning
• Efficient inference in occlusion-aware generative models of images
• Challenges in Disentangling Independent Factors of Variation
• View Synthesis by Appearance Flow
• Controllable Data Generation by Deep Learning: A Review
• Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness
• Disentangled Person Image Generation
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• RG-Flow: A hierarchical and explainable flow model based on renormalization group and sparse prior
• Learning Disentangled Representations for Timber and Pitch in Music Audio
• Denoising without access to clean data using a partitioned autoencoder
• Effect of The Latent Structure on Clustering with GANs
• VGAN-Based Image Representation Learning for Privacy-Preserving Facial Expression Recognition
• An Identifiable Double VAE For Disentangled Representations
• Attribute Guided Unpaired Image-to-Image Translation with Semi-supervised Learning
• On the Fairness of Disentangled Representations
• All-In-One: Facial Expression Transfer, Editing and Recognition Using A Single Network
• Image Generation from Layout
• Automatic Relevance Determination For Deep Generative Models
• Conditional Adversarial Generative Flow for Controllable Image Synthesis
• Pose-Guided High-Resolution Appearance Transfer via Progressive Training
• Factorized Adversarial Networks for Unsupervised Domain Adaptation
• An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wild
• Improving Content-Invariance in Gated Autoencoders for 2D and 3D Object Rotation
• Tuning-Free Disentanglement via Projection
• Multi-Label Robust Factorization Autoencoder and its Application in Predicting Drug-Drug Interactions
• Mode matching in GANs through latent space learning and inversion
• Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval
• Generalization to Novel Objects using Prior Relational Knowledge
• Scalable and Effective Deep CCA via Soft Decorrelation
• DLGAN: Disentangling Label-Specific Fine-Grained Features for Image Manipulation
• Disentangling images with Lie group transformations and sparse coding
• Forced to Learn: Discovering Disentangled Representations Without Exhaustive Labels
• Novel View Synthesis via Depth-guided Skip Connections
• Toward Privacy and Utility Preserving Image Representation
• Efficient decorrelation of features using Gramian in Reinforcement Learning
• Attending Category Disentangled Global Context for Image Classification
• Quantifying the Effects of Enforcing Disentanglement on Variational Autoencoders
• Attribute-controlled face photo synthesis from simple line drawing