Degeneration in VAE: in the Light of Fisher Information Loss

Abstract

While enormous progress has been made to Variational Autoencoder (VAE) in recent years, similar to other deep networks, VAE with deep networks suffers from the problem of degeneration, which seriously weakens the correlation between the input and the corresponding latent codes, deviating from the goal of the representation learning. To investigate how degeneration affects VAE from a theoretical perspective, we illustrate the information transmission in VAE and analyze the intermediate layers of the encoders/decoders. Specifically, we propose a Fisher Information measure for the layer-wise analysis. With such measure, we demonstrate that information loss is ineluctable in feed-forward networks and causes the degeneration in VAE. We show that skip connections in VAE enable the preservation of information without changing the model architecture. We call this class of VAE equipped with skip connections as SCVAE and perform a range of experiments to show its advantages in information preservation and degeneration mitigation.

References in corpus (10)

• Densely Connected Convolutional Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• DRAW: A Recurrent Neural Network For Image Generation
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• InfoVAE: Information Maximizing Variational Autoencoders
• MADE: Masked Autoencoder for Distribution Estimation
• Importance Weighted Autoencoders
• Improving Variational Inference with Inverse Autoregressive Flow
• VAE with a VampPrior
• Learning Hierarchical Features from Generative Models

Cited by in corpus (7)

• Learning on Attribute-Missing Graphs
• Information Bottleneck and its Applications in Deep Learning
• Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems
• Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
• Perturbation theory approach to study the latent space degeneracy of Variational Autoencoders
• Learning Audio-Visual Correlations from Variational Cross-Modal Generation
• Decoupled Variational Embedding for Signed Directed Networks