Multilayer Lookahead: a Nested Version of Lookahead

Abstract

In recent years, SGD and its variants have become the standard tool to train Deep Neural Networks. In this paper, we focus on the recently proposed variant Lookahead, which improves upon SGD in a wide range of applications. Following this success, we study an extension of this algorithm, the \emph{Multilayer Lookahead} optimizer, which recursively wraps Lookahead around itself. We prove the convergence of Multilayer Lookahead with two layers to a stationary point of smooth non-convex functions with $O(\frac{1}{\sqrt{T}})$ rate. We also justify the improved generalization of both Lookahead over SGD, and of Multilayer Lookahead over Lookahead, by showing how they amplify the implicit regularization effect of SGD. We empirically verify our results and show that Multilayer Lookahead outperforms Lookahead on CIFAR-10 and CIFAR-100 classification tasks, and on GANs training on the MNIST dataset.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• On First-Order Meta-Learning Algorithms
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Lookahead Optimizer: k steps forward, 1 step back
• Local SGD Converges Fast and Communicates Little
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• The large learning rate phase of deep learning: the catapult mechanism
• Fluctuation-dissipation relations for stochastic gradient descent
• Implicit Gradient Regularization
• Taming GANs with Lookahead-Minmax