On Training Implicit Models
arXiv:2111.05177
Abstract
This paper focuses on training implicit models of infinite layers. Specifically, previous works employ implicit differentiation and solve the exact gradient for the backward propagation. However, is it necessary to compute such an exact but expensive gradient for training? In this work, we propose a novel gradient estimate for implicit models, named phantom gradient, that 1) forgoes the costly computation of the exact gradient; and 2) provides an update direction empirically preferable to the implicit model training. We theoretically analyze the condition under which an ascent direction of the loss landscape could be found, and provide two specific instantiations of the phantom gradient based on the damped unrolling and Neumann series. Experiments on large-scale tasks demonstrate that these lightweight phantom gradients significantly accelerate the backward passes in training implicit models by roughly 1.7 times, and even boost the performance over approaches based on the exact gradient on ImageNet.
arXiv comment24 pages, 4 figures, in The 35th Conference on Neural Information Processing Systems (NeurIPS 2021)
References in corpus (26)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Improving neural networks by preventing co-adaptation of feature detectors
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Spectral Normalization for Generative Adversarial Networks
- DARTS: Differentiable Architecture Search
- Shake-Shake regularization
- Deep Equilibrium Models
- Meta-Learning with Implicit Gradients
- Residual Flows for Invertible Generative Modeling
- Training Neural Networks Without Gradients: A Scalable ADMM Approach
- Hyperparameter optimization with approximate gradient
- How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
- Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
- The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
- End to end learning and optimization on graphs
- Implicit Graph Neural Networks
- Is Attention Better Than Matrix Decomposition?
- On the Noisy Gradient Descent that Generalizes as SGD
- Understanding Synthetic Gradients and Decoupled Neural Interfaces
- Learning to solve the credit assignment problem
- Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training
- Implicit Feature Pyramid Network for Object Detection
- On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
- Stabilizing Equilibrium Models by Jacobian Regularization
- Implicit Normalizing Flows
- Optimization Induced Equilibrium Networks