Exploring loss function topology with cyclical learning rates

Abstract

We present observations and discussion of previously unreported phenomena discovered while training residual networks. The goal of this work is to better understand the nature of neural networks through the examination of these new empirical results. These behaviors were identified through the application of Cyclical Learning Rates (CLR) and linear network interpolation. Among these behaviors are counterintuitive increases and decreases in training loss and instances of rapid training. For example, we demonstrate how CLR can produce greater testing accuracy than traditional training despite using large learning rates. Files to replicate these results are available at https://github.com/lnsmith54/exploring-loss

Submitted as an ICLR 2017 Workshop paper

References in corpus (2)

• Qualitatively characterizing neural network optimization problems
• An empirical analysis of the optimization of deep network loss surfaces

Cited by in corpus (9)

• Visualizing the Loss Landscape of Neural Nets
• Averaging Weights Leads to Wider Optima and Better Generalization
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Errors for Deep Neural Networks
• Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Data efficiency and extrapolation trends in neural network interatomic potentials
• A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training of DNNs
• SuperNet -- An efficient method of neural networks ensembling