Regularized linear autoencoders recover the principal components, eventually
arXiv:2007.06731
Abstract
Our understanding of learning input-output relationships with neural nets has improved rapidly in recent years, but little is known about the convergence of the underlying representations, even in the simple case of linear autoencoders (LAEs). We show that when trained with proper regularization, LAEs can directly learn the optimal representation -- ordered, axis-aligned principal components. We analyze two such regularization schemes: non-uniform regularization and a deterministic variant of nested dropout [Rippel et al, ICML' 2014]. Though both regularization schemes converge to the optimal representation, we show that this convergence is slow due to ill-conditioning that worsens with increasing latent dimension. We show that the inefficiency of learning the optimal representation is not inevitable -- we present a simple modification to the gradient descent update that greatly speeds up convergence empirically.
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A mathematical theory of semantic development in deep neural networks
- Learning Ordered Representations with Nested Dropout
- Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks
- Loss Landscapes of Regularized Linear Autoencoders
- On Implicit Regularization in -VAEs
- EigenGame: PCA as a Nash Equilibrium
- Improving Optimization for Models With Continuous Symmetry Breaking