Memorization in Overparameterized Autoencoders

Abstract

The ability of deep neural networks to generalize well in the overparameterized regime has become a subject of significant research interest. We show that overparameterized autoencoders exhibit memorization, a form of inductive bias that constrains the functions learned through the optimization process to concentrate around the training examples, although the network could in principle represent a much larger function class. In particular, we prove that single-layer fully-connected autoencoders project data onto the (nonlinear) span of the training examples. In addition, we show that deep fully-connected autoencoders learn a map that is locally contractive at the training examples, and hence iterating the autoencoder results in convergence to the training examples. Finally, we prove that depth is necessary and provide empirical evidence that it is also sufficient for memorization in convolutional autoencoders. Understanding this inductive bias may shed light on the generalization properties of overparametrized deep neural networks that are currently unexplained by classical statistical theory.

References in corpus (9)

• Scikit-learn: Machine Learning in Python
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Deep Image Prior
• A Closer Look at Memorization in Deep Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• How do infinite width bounded norm networks look in function space?

Cited by in corpus (9)

• Shallow Neural Networks for Fluid Flow Reconstruction with Limited Sensors
• DarkneTZ: Towards Model Privacy at the Edge using Trusted Execution Environments
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• Analysis of feature learning in weight-tied autoencoders via the mean field lens
• Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networks
• Fixing Bias in Reconstruction-based Anomaly Detection with Lipschitz Discriminators
• Rate-Regularization and Generalization in VAEs
• On Alignment in Deep Linear Neural Networks
• A Mechanism for Producing Aligned Latent Spaces with Autoencoders