Lookahead Optimizer: k steps forward, 1 step back

Abstract

The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categorized into two approaches: (1) adaptive learning rate schemes, such as AdaGrad and Adam, and (2) accelerated schemes, such as heavy-ball and Nesterov momentum. In this paper, we propose a new optimization algorithm, Lookahead, that is orthogonal to these previous approaches and iteratively updates two sets of weights. Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of fast weights generated by another optimizer. We show that Lookahead improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost. We empirically demonstrate Lookahead can significantly improve the performance of SGD and Adam, even with their default hyperparameter settings on ImageNet, CIFAR-10/100, neural machine translation, and Penn Treebank.

arXiv commentAccepted to Neural Information Processing Systems 2019. Code available at: https://github.com/michaelrzhang/lookahead

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Decoupled Weight Decay Regularization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Regularizing and Optimizing LSTM Language Models
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• New insights and perspectives on the natural gradient method
• No More Pesky Learning Rates
• Averaging Weights Leads to Wider Optima and Better Generalization
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• The Unusual Effectiveness of Averaging in GAN Training
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization

Cited by in corpus (75)

• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Physics-informed neural networks for the shallow-water equations on the sphere
• A New Dataset, Poisson GAN and AquaNet for Underwater Object Grabbing
• Review: Deep Learning in Electron Microscopy
• CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
• Audio Retrieval with Natural Language Queries: A Benchmark Study
• Applications and Techniques for Fast Machine Learning in Science
• SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
• Generalized Wasserstein Dice Score, Distributionally Robust Deep Learning, and Ranger for brain tumor segmentation: BraTS 2020 challenge
• Cross-dimensional transfer learning in medical image segmentation with deep learning
• Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
• Neural Networks Versus Conventional Filters for Inertial-Sensor-based Attitude Estimation
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Personalized Federated Learning using Hypernetworks
• PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTML
• Entropic gradient descent algorithms and wide flat minima
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• Deep Volumetric Ambient Occlusion
• Hierarchical forecasting with a top-down alignment of independent level forecasts
• Connecting optical morphology, environment, and HI mass fraction for low-redshift galaxies using deep learning
• LaProp: Separating Momentum and Adaptivity in Adam
• Surgical Visual Domain Adaptation: Results from the MICCAI 2020 SurgVisDom Challenge
• Drum-Aware Ensemble Architecture for Improved Joint Musical Beat and Downbeat Tracking
• Learning spectro-temporal representations of complex sounds with parameterized neural networks
• Phonetic Posteriorgrams based Many-to-Many Singing Voice Conversion via Adversarial Training
• Self6D: Self-Supervised Monocular 6D Object Pose Estimation
• Learning Curves for Analysis of Deep Networks
• Domain-Specific Pre-training Improves Confidence in Whole Slide Image Classification
• Image segmentation of treated and untreated tumor spheroids by Fully Convolutional Networks
• Predicting galaxy spectra from images with hybrid convolutional neural networks
• Measuring and Harnessing Transference in Multi-Task Learning
• PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Table Image Recognition to Latex
• On Accelerating Distributed Convex Optimizations
• 1st Place Solution for ICDAR 2021 Competition on Mathematical Formula Detection
• Multimodal Indoor Localisation in Parkinson's Disease for Detecting Medication Use: Observational Pilot Study in a Free-Living Setting
• ReFlow6D: Refraction-Guided Transparent Object 6D Pose Estimation via Intermediate Representation Learning
• Training Stronger Baselines for Learning to Optimize
• Neuromorphic computing for attitude estimation onboard quadrotors
• CRNNs for Urban Sound Tagging with spatiotemporal context
• Leveraging Human Selective Attention for Medical Image Analysis with Limited Training Data
• Taming GANs with Lookahead-Minmax
• Stochastic Anderson Mixing for Nonconvex Stochastic Optimization
• Ranger21: a synergistic deep learning optimizer
• Bilevel Continual Learning
• Jet energy calibration with deep learning as a Kubeflow pipeline
• Multi-view Self-Constructing Graph Convolutional Networks with Adaptive Class Weighting Loss for Semantic Segmentation
• Improving End-to-End Speech-to-Intent Classification with Reptile
• SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches
• RIANN -- A Robust Neural Network Outperforms Attitude Estimation Filters
• Monocular pose estimation of articulated open surgery tools -- in the wild
• Weakly Supervised Multi-Task Learning for Cell Detection and Segmentation
• VidFace: A Full-Transformer Solver for Video FaceHallucination with Unaligned Tiny Snapshots
• Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization
• Enhanced 3D Human Pose Estimation from Videos by using Attention-Based Neural Network with Dilated Convolutions
• Targeted Adversarial Training for Natural Language Understanding
• Multi-Plateau Ensemble for Endoscopic Artefact Segmentation and Detection
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Hierarchical Attention-based Age Estimation and Bias Estimation
• Beyond Individual Input for Deep Anomaly Detection on Tabular Data
• Training With Data Dependent Dynamic Learning Rates
• Reconstructing hadronically decaying tau leptons with a jet foundation model
• Colored Noise Injection for Training Adversarially Robust Neural Networks
• Image interpretation by iterative bottom-up top-down processing
• Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model
• ISCL: Interdependent Self-Cooperative Learning for Unpaired Image Denoising
• Delta Sampling R-BERT for limited data and low-light action recognition
• The Efficiency Misnomer
• Multilayer Lookahead: a Nested Version of Lookahead
• Conflicting Bundles: Adapting Architectures Towards the Improved Training of Deep Neural Networks
• Knowledge Generation -- Variational Bayes on Knowledge Graphs
• Bilevel Online Deep Learning in Non-stationary Environment
• Complex Momentum for Optimization in Games