On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers

Abstract

A deep equilibrium model uses implicit layers, which are implicitly defined through an equilibrium point of an infinite sequence of computation. It avoids any explicit computation of the infinite sequence by finding an equilibrium point directly via root-finding and by computing gradients via implicit differentiation. In this paper, we analyze the gradient dynamics of deep equilibrium models with nonlinearity only on weight matrices and non-convex objective functions of weights for regression and classification. Despite non-convexity, convergence to global optimum at a linear rate is guaranteed without any assumption on the width of the models, allowing the width to be smaller than the output dimension and the number of data points. Moreover, we prove a relation between the gradient dynamics of the deep implicit layer and the dynamics of trust region Newton method of a shallow explicit layer. This mathematically proven relation along with our numerical observation suggests the importance of understanding implicit bias of implicit layers and an open problem on the topic. Our proofs deal with implicit layers, weight tying and nonlinearity on weights, and differ from those in the related literature.

ICLR 2021. Selected for ICLR Spotlight (top 6% submissions)

References in corpus (17)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On Lazy Training in Differentiable Programming
• Deep Equilibrium Models
• Identity Matters in Deep Learning
• Depth with Nonlinearity Creates No Bad Local Minima in ResNets
• Kernel and Rich Regimes in Overparametrized Models
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• Adding One Neuron Can Eliminate All Bad Local Minima
• Effect of Depth and Width on Local Minima in Deep Learning
• Are ResNets Provably Better than Linear Predictors?
• On the Global Convergence of Training Deep Linear ResNets
• A Note on Connectivity of Sublevel Sets in Deep Learning

Cited by in corpus (8)

• Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology
• When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
• On Training Implicit Models
• A global convergence theory for deep ReLU implicit networks via over-parameterization
• A Recipe for Global Convergence Guarantee in Deep Neural Networks
• Stable Invariant Models via Koopman Spectra
• Understanding Dynamics of Nonlinear Representation Learning and Its Application
• Robust Implicit Networks via Non-Euclidean Contractions