What Regularized Auto-Encoders Learn from the Data Generating Distribution

Abstract

What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments.

References in corpus (3)

• Better Mixing via Deep Representations
• Representation Learning: A Review and New Perspectives
• Implicit Density Estimation by Local Moment Matching to Sample from Auto-Encoders

Cited by in corpus (17)

• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Representation Learning: A Review and New Perspectives
• Bayesian MRI Reconstruction with Joint Uncertainty Estimation using Diffusion Models
• Overparameterized Neural Networks Implement Associative Memory
• Linear and Nonlinear Dimensionality Reduction from Fluid Mechanics to Machine Learning
• Score Function Features for Discriminative Learning: Matrix and Tensor Framework
• Deeply Coupled Auto-encoder Networks for Cross-view Classification
• Spectral Learning on Matrices and Tensors
• Saturating Auto-Encoders
• Bayes-Optimal Unsupervised Learning for Channel Estimation in Near-Field Holographic MIMO
• Training Input-Output Recurrent Neural Networks through Spectral Methods
• Provable Methods for Training Neural Networks with Sparse Connectivity
• Provable Tensor Methods for Learning Mixtures of Generalized Linear Models
• De-noising of galaxy optical spectra with autoencoders
• Visual Time Series Forecasting: An Image-driven Approach
• Learning Discriminative Features with Class Encoder