Robust Implicit Backpropagation

Abstract

Arguably the biggest challenge in applying neural networks is tuning the hyperparameters, in particular the learning rate. The sensitivity to the learning rate is due to the reliance on backpropagation to train the network. In this paper we present the first application of Implicit Stochastic Gradient Descent (ISGD) to train neural networks, a method known in convex optimization to be unconditionally stable and robust to the learning rate. Our key contribution is a novel layer-wise approximation of ISGD which makes its updates tractable for neural networks. Experiments show that our method is more robust to high learning rates and generally outperforms standard backpropagation on a variety of tasks.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Self-Normalizing Neural Networks
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• YellowFin and the Art of Momentum Tuning
• Incremental Aggregated Proximal and Augmented Lagrangian Algorithms
• Catalyst Acceleration for Gradient-Based Non-Convex Optimization
• Stochastic Methods for Composite and Weakly Convex Optimization Problems
• Catalyst Acceleration for First-order Convex Optimization: from Theory to Practice
• Nonasymptotic convergence of stochastic proximal point algorithms for constrained convex optimization
• Implicit Temporal Differences