Learning Hierarchical Features from Generative Models

Abstract

Deep neural networks have been shown to be very successful at learning feature hierarchies in supervised learning tasks. Generative models, on the other hand, have benefited less from hierarchical models with multiple layers of latent variables. In this paper, we prove that hierarchical latent variable models do not take advantage of the hierarchical structure when trained with existing variational methods, and provide some limitations on the kind of features existing models can learn. Finally we propose an alternative architecture that do not suffer from these limitations. Our model is able to learn highly interpretable and disentangled hierarchical features on several natural image datasets with no task specific regularization or prior knowledge.

ICML'2017

References in corpus (3)

• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• NICE: Non-linear Independent Components Estimation
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap

Cited by in corpus (12)

• InfoVAE: Information Maximizing Variational Autoencoders
• Deep Neural Generative Model of Functional MRI Images for Psychiatric Disorder Diagnosis
• A Tutorial on Deep Latent Variable Models of Natural Language
• Auto-Encoding Total Correlation Explanation
• Learning Interpretable Representations of Entanglement in Quantum Optics Experiments using Deep Generative Models
• Degeneration in VAE: in the Light of Fisher Information Loss
• An Explicit Local and Global Representation Disentanglement Framework with Applications in Deep Clustering and Unsupervised Object Detection
• Commutative Lie Group VAE for Disentanglement Learning
• Learning Disentangled Representations with Latent Variation Predictability
• An Interpretable Generative Model for Handwritten Digit Image Synthesis
• There and Back Again: Unraveling the Variational Auto-Encoder
• Multi-Task Neural Processes