Disentangling factors of variation in deep representations using adversarial training

Abstract

We introduce a conditional generative model for learning to disentangle the hidden factors of variation within a set of labeled observations, and separate them into complementary codes. One code summarizes the specified factors of variation associated with the labels. The other summarizes the remaining unspecified variability. During training, the only available source of supervision comes from our ability to distinguish among different observations belonging to the same class. Examples of such observations include images of a set of labeled objects captured at different viewpoints, or recordings of set of speakers dictating multiple phrases. In both instances, the intra-class diversity is the source of the unspecified factors of variation: each object is observed at multiple viewpoints, and each speaker dictates multiple phrases. Learning to disentangle the specified factors from the unspecified ones becomes easier when strong supervision is possible. Suppose that during training, we have access to pairs of images, where each pair shows two different objects captured from the same viewpoint. This source of alignment allows us to solve our task using existing methods. However, labels for the unspecified factors are usually unavailable in realistic scenarios where data acquisition is not strictly controlled. We address the problem of disentanglement in this more general setting by combining deep convolutional autoencoders with a form of adversarial training. Both factors of variation are implicitly captured in the organization of the learned embedding space, and can be used for solving single-image analogies. Experimental results on synthetic and real datasets show that the proposed method is capable of generalizing to unseen classes and intra-class variabilities.

arXiv commentConference paper in NIPS 2016

References in corpus (8)

• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Deep Convolutional Inverse Graphics Network
• Autoencoding beyond pixels using a learned similarity metric
• Deep multi-scale video prediction beyond mean square error
• Discovering Hidden Factors of Variation in Deep Networks

Cited by in corpus (111)

• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• Recent Advances in Autoencoder-Based Representation Learning
• Fader Networks: Manipulating Images by Sliding Attributes
• Disentangled Representation Learning in Cardiac Image Analysis
• Unsupervised Learning via Meta-Learning
• Federated Adversarial Domain Adaptation
• Swapping Autoencoder for Deep Image Manipulation
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Weakly-Supervised Disentanglement Without Compromises
• Compositional Fairness Constraints for Graph Embeddings
• Unsupervised Object Segmentation by Redrawing
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• Disentangling Disentanglement in Variational Autoencoders
• Disentangling Factors of Variation Using Few Labels
• Distribution Matching in Variational Inference
• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• Semi-Latent GAN: Learning to generate and modify facial images from attributes
• Feature Alignment and Restoration for Domain Generalization and Adaptation
• Domain-invariant Similarity Activation Map Contrastive Learning for Retrieval-based Long-term Visual Localization
• Counterfactuals uncover the modular structure of deep generative models
• AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation
• Music Style Transfer: A Position Paper
• Sill-Net: Feature Augmentation with Separated Illumination Representation
• Attacks on State-of-the-Art Face Recognition using Attentional Adversarial Attack Generative Network
• Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness
• Learning Interpretable Representation for Controllable Polyphonic Music Generation
• Finding an Unsupervised Image Segmenter in Each of Your Deep Generative Models
• Joint Disentangling and Adaptation for Cross-Domain Person Re-Identification
• Deep Generative Video Compression
• Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry
• Learning Disentangled Representations with Reference-Based Variational Autoencoders
• JADE: Joint Autoencoders for Dis-Entanglement
• Semi-Supervised Generative Modeling for Controllable Speech Synthesis
• Elastic-InfoGAN: Unsupervised Disentangled Representation Learning in Class-Imbalanced Data
• Theory and Evaluation Metrics for Learning Disentangled Representations
• Invariant Representations from Adversarially Censored Autoencoders
• Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
• Attribute Guided Unpaired Image-to-Image Translation with Semi-supervised Learning
• Disentangling Style and Content in Anime Illustrations
• StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval
• Guided Variational Autoencoder for Disentanglement Learning
• Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders
• Multi-View Data Generation Without View Supervision
• On the Fairness of Disentangled Representations
• Disentangled Representation Learning with Wasserstein Total Correlation
• Transformers with Competitive Ensembles of Independent Mechanisms
• Learning Cross-domain Generalizable Features by Representation Disentanglement
• Product of Orthogonal Spheres Parameterization for Disentangled Representation Learning
• Improving ClusterGAN Using Self-Augmented Information Maximization of Disentangling Latent Spaces
• Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis
• Image Generation from Layout
• Dual Gaussian-based Variational Subspace Disentanglement for Visible-Infrared Person Re-Identification
• Conditional Adversarial Generative Flow for Controllable Image Synthesis
• Disentangled Item Representation for Recommender Systems
• Adversarially Approximated Autoencoder for Image Generation and Manipulation
• Unsupervised Domain Adaptation via Disentangled Representations: Application to Cross-Modality Liver Segmentation
• Factorized Adversarial Networks for Unsupervised Domain Adaptation
• DynamicVAE: Decoupling Reconstruction Error and Disentangled Representation Learning
• An Improved Semi-Supervised VAE for Learning Disentangled Representations
• Unsupervised 3D Shape Learning from Image Collections in the Wild
• Probabilistic Video Generation using Holistic Attribute Control
• DualDis: Dual-Branch Disentangling with Adversarial Learning
• Latent feature disentanglement for 3D meshes
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• A Unified Conditional Disentanglement Framework for Multimodal Brain MR Image Translation
• Adversarial Disentanglement with Grouped Observations
• Unsupervised Shape and Pose Disentanglement for 3D Meshes
• Video Reenactment as Inductive Bias for Content-Motion Disentanglement
• Representation Disentanglement for Multi-task Learning with application to Fetal Ultrasound
• A Cyclically-Trained Adversarial Network for Invariant Representation Learning
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling
• Future Urban Scenes Generation Through Vehicles Synthesis
• Learning Independently-Obtainable Reward Functions
• Mix and match networks: cross-modal alignment for zero-pair image-to-image translation
• Disentangling Dynamics and Content for Control and Planning
• Generative and Discriminative Learning for Distorted Image Restoration
• Evolvement Constrained Adversarial Learning for Video Style Transfer
• Identity-aware Facial Expression Recognition in Compressed Video
• Bayes-Factor-VAE: Hierarchical Bayesian Deep Auto-Encoder Models for Factor Disentanglement
• Dilated Temporal Relational Adversarial Network for Generic Video Summarization
• Double cycle-consistent generative adversarial network for unsupervised conditional generation
• DLGAN: Disentangling Label-Specific Fine-Grained Features for Image Manipulation
• Conditional Adversarial Camera Model Anonymization
• Inverse Learning of Symmetries
• Decoupling Global and Local Representations via Invertible Generative Flows
• Rethinking Content and Style: Exploring Bias for Unsupervised Disentanglement
• Image Style Transfer and Content-Style Disentanglement
• Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS
• StyleFusion: A Generative Model for Disentangling Spatial Segments
• Inference-InfoGAN: Inference Independence via Embedding Orthogonal Basis Expansion
• Domain2Vec: Domain Embedding for Unsupervised Domain Adaptation
• Excavate Condition-invariant Space by Intrinsic Encoder
• Unsupervised Domain Alignment to Mitigate Low Level Dataset Biases
• An Interactive Insight Identification and Annotation Framework for Power Grid Pixel Maps using DenseU-Hierarchical VAE
• Towards Purely Unsupervised Disentanglement of Appearance and Shape for Person Images Generation
• Group-based Learning of Disentangled Representations with Generalizability for Novel Contents
• Learning Invariant Representation of Tasks for Robust Surgical State Estimation
• Human Annotations Improve GAN Performances
• Quantifying point cloud realism through adversarially learned latent representations
• Attending Category Disentangled Global Context for Image Classification
• Mutual Information-based Disentangled Neural Networks for Classifying Unseen Categories in Different Domains: Application to Fetal Ultrasound Imaging
• Multi-class Novelty Detection Using Mix-up Technique
• Towards Better Understanding of Disentangled Representations via Mutual Information
• Cross-Domain Image Manipulation by Demonstration
• A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations
• Toward Privacy and Utility Preserving Image Representation
• Disentangled Adversarial Transfer Learning for Physiological Biosignals
• Decoder-free Robustness Disentanglement without (Additional) Supervision
• A Two-Step Disentanglement Method
• Disentangling Latent Space for VAE by Label Relevant/Irrelevant Dimensions