Loss Landscapes of Regularized Linear Autoencoders

Abstract

Autoencoders are a deep learning model for representation learning. When trained to minimize the distance between the data and its reconstruction, linear autoencoders (LAEs) learn the subspace spanned by the top principal directions but cannot learn the principal directions themselves. In this paper, we prove that $L_2$-regularized LAEs are symmetric at all critical points and learn the principal directions as the left singular vectors of the decoder. We smoothly parameterize the critical manifold and relate the minima to the MAP estimate of probabilistic PCA. We illustrate these results empirically and consider implications for PCA algorithms, computational neuroscience, and the algebraic topology of learning.

arXiv comment12 pages, 8 figures. ICML 2019

References in corpus (7)

• The Loss Surfaces of Multilayer Networks
• Global Optimality in Low-rank Matrix Optimization
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• From Principal Subspaces to Principal Components with Linear Autoencoders
• Deep Learning without Weight Transport
• On the Implicit Bias of Dropout
• The combinatorics of Morse theory with boundary

Cited by in corpus (5)

• Deep Learning without Weight Transport
• Two Routes to Scalable Credit Assignment without Weight Symmetry
• Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
• Regularized linear autoencoders recover the principal components, eventually
• The ELBO of Variational Autoencoders Converges to a Sum of Three Entropies