A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay

Abstract

Although deep learning has produced dazzling successes for applications of image, speech, and video processing in the past few years, most trainings are with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyper-parameters remains a black art that requires years of experience to acquire. This report proposes several efficient ways to set the hyper-parameters that significantly reduce training time and improves performance. Specifically, this report shows how to examine the training validation/test loss function for subtle clues of underfitting and overfitting and suggests guidelines for moving toward the optimal balance point. Then it discusses how to increase/decrease the learning rate/momentum to speed up training. Our experiments show that it is crucial to balance every manner of regularization for each dataset and architecture. Weight decay is used as a sample regularizer to show how its optimal value is tightly coupled with the learning rates and momentums. Files to help replicate the results reported here are available.

Files to help replicate the results reported here are available on Github

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Don't Decay the Learning Rate, Increase the Batch Size
• Three Factors Influencing Minima in SGD
• Regularization for Deep Learning: A Taxonomy
• Stochastic Hyperparameter Optimization through Hypernetworks
• Residual Connections Encourage Iterative Inference
• A Walk with SGD
• Do deep nets really need weight decay and dropout?
• A Diffusion Approximation Theory of Momentum SGD in Nonconvex Optimization

Cited by in corpus (129)

• ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data
• fastai: A Layered API for Deep Learning
• Atomistic Line Graph Neural Network for Improved Materials Property Predictions
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• ParticleNet: Jet Tagging via Particle Clouds
• Automated Pavement Crack Segmentation Using U-Net-based Convolutional Neural Network
• Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning
• SuperNNova: an open-source framework for Bayesian, Neural Network based supernova classification
• On Empirical Comparisons of Optimizers for Deep Learning
• Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifiers
• Query2Label: A Simple Transformer Way to Multi-Label Classification
• Adversarial Robustness Against the Union of Multiple Perturbation Models
• Rethinking the Hyperparameters for Fine-tuning
• Asymmetric Loss For Multi-Label Classification
• Automatic Vision-Based Parking Slot Detection and Occupancy Classification
• Multi-scale Convolutional Neural Networks for Inverse Problems
• ktrain: A Low-Code Library for Augmented Machine Learning
• AR-Net: A simple Auto-Regressive Neural Network for time-series
• Using deep learning to understand and mitigate the qubit noise environment
• Vertex and Energy Reconstruction in JUNO with Machine Learning Methods
• NeuralTailor: Reconstructing Sewing Pattern Structures from 3D Point Clouds of Garments
• Skin Lesions Classification Using Convolutional Neural Networks in Clinical Images
• The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics
• One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision
• Understanding Batch Normalization
• Multiclass Language Identification using Deep Learning on Spectral Images of Audio Signals
• Variational Bayes survival analysis for unemployment modelling
• Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training
• Multiplicative noise and heavy tails in stochastic optimization
• Toward a Next Generation Particle Precipitation Model: Mesoscale Prediction Through Machine Learning (a Case Study and Framework for Progress)
• Machine Learning in High Energy Physics: A review of heavy-flavor jet tagging at the LHC
• Training Deep Spiking Neural Networks
• Regularization Learning Networks: Deep Learning for Tabular Datasets
• Exploiting the ConvLSTM: Human Action Recognition using Raw Depth Video-Based Recurrent Neural Networks
• Connecting optical morphology, environment, and HI mass fraction for low-redshift galaxies using deep learning
• Topology-Informed Machine Learning for Efficient Prediction of Solid Oxide Fuel Cell Electrode Polarization
• BenthicNet: A global compilation of seafloor images for deep learning applications
• Fast robust peg-in-hole insertion with continuous visual servoing
• Image Based Identification of Ghanaian Timbers Using the XyloTron: Opportunities, Risks and Challenges
• Statistical process monitoring of artificial neural networks
• Artificial and beneficial -- Exploiting artificial images for aerial vehicle detection
• Influence Functions in Deep Learning Are Fragile
• Frustratingly Easy Natural Question Answering
• TomOpt: Differential optimisation for task- and constraint-aware design of particle detectors in the context of muon tomography
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• FLERT: Document-Level Features for Named Entity Recognition
• The Fishnet Open Images Database: A Dataset for Fish Detection and Fine-Grained Categorization in Fisheries
• The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews
• Learning to play the Chess Variant Crazyhouse above World Champion Level with Deep Neural Networks and Human Data
• Embedded hyper-parameter tuning by Simulated Annealing
• On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case
• Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
• MXR-U-Nets for Real Time Hyperspectral Reconstruction
• Estimating oil and gas recovery factors via machine learning: Database-dependent accuracy and reliability
• Towards recognizing the light facet of the Higgs Boson
• TAME: Attention Mechanism Based Feature Fusion for Generating Explanation Maps of Convolutional Neural Networks
• Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification
• Predicting brain-age from raw T 1 -weighted Magnetic Resonance Imaging data using 3D Convolutional Neural Networks
• T-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers
• Improving the HardNet Descriptor
• Seismic data denoising and deblending using deep learning
• Variational Quantum Circuit Design for Quantum Reinforcement Learning on Continuous Environments
• Image segmentation of treated and untreated tumor spheroids by Fully Convolutional Networks
• CFO: A Framework for Building Production NLP Systems
• An Evolutionary Deep Learning Method for Short-term Wind Speed Prediction: A Case Study of the Lillgrund Offshore Wind Farm
• 3D Human Pose Estimation with 2D Marginal Heatmaps
• aSAGA: Automatic Sleep Analysis with Gray Areas
• MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
• Composer Style Classification of Piano Sheet Music Images Using Language Model Pretraining
• MBAPose: Mask and Bounding-Box Aware Pose Estimation of Surgical Instruments with Photorealistic Domain Randomization
• Task Embedding Temporal Convolution Networks for Transfer Learning Problems in Renewable Power Time-Series Forecast
• AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly
• Echofilter: A Deep Learning Segmentation Model Improves the Automation, Standardization, and Timeliness for Post-Processing Echosounder Data in Tidal Energy Streams
• REX: Revisiting Budgeted Training with an Improved Schedule
• Applying a Pre-trained Language Model to Spanish Twitter Humor Prediction
• More Than Meets The Eye: Semi-supervised Learning Under Non-IID Data
• Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding
• Learning Architectures for Binary Networks
• Skin lesion segmentation using U-Net and good training strategies
• Too Many Claims to Fact-Check: Prioritizing Political Claims Based on Check-Worthiness
• Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
• Inferring Javascript types using Graph Neural Networks
• ContamiNet: Detecting Contamination in Municipal Solid Waste
• ImageNet-21K Pretraining for the Masses
• Super-convergence and Differential Privacy: Training faster with better privacy guarantees
• AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks through Accuracy Gradient
• ConvNets for Counting: Object Detection of Transient Phenomena in Steelpan Drums
• Exploiting Unsupervised Pre-training and Automated Feature Engineering for Low-resource Hate Speech Detection in Polish
• Muddling Label Regularization: Deep Learning for Tabular Datasets
• Hierarchical Multi-Label Classification with Missing Information for Benthic Habitat Imagery
• Is Attention always needed? A Case Study on Language Identification from Speech
• Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck
• Bayesian Optimization for Iterative Learning
• Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results
• 2kenize: Tying Subword Sequences for Chinese Script Conversion
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements
• InfoFocus: 3D Object Detection for Autonomous Driving with Dynamic Information Modeling
• ArXiving Before Submission Helps Everyone
• Neural Model-based Optimization with Right-Censored Observations
• Automated Multiclass Cardiac Volume Segmentation and Model Generation
• Analysis of skin lesion images with deep learning
• Learning scale-variant features for robust iris authentication with deep learning based ensemble framework
• Instrument Classification of Solo Sheet Music Images
• AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop
• CProp: Adaptive Learning Rate Scaling from Past Gradient Conformity
• Unconstrained Text Detection in Manga
• Muddling Labels for Regularization, a novel approach to generalization
• NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image
• Deep-learning in the bioimaging wild: Handling ambiguous data with deepflash2
• Discover the Mysteries of the Maya: Selected Contributions from the Machine Learning Challenge & The Discovery Challenge Workshop at ECML PKDD 2021
• Implicit Equivariance in Convolutional Networks
• Sequence-to-sequence Singing Voice Synthesis with Perceptual Entropy Loss
• (Un)Masked COVID-19 Trends from Social Media
• Evolutionary NAS with Gene Expression Programming of Cellular Encoding
• Implicit Regularization of Bregman Proximal Point Algorithm and Mirror Descent on Separable Data
• Mining for Potent Inhibitors through Artificial Intelligence and Physics: A Unified Methodology for Ligand Based and Structure Based Drug Design
• Automatic travel pattern extraction from visa page stamps using CNN models
• Label-free Monitoring of Self-Supervised Learning Progress
• BOBBY2: Buffer Based Robust High-Speed Object Tracking
• Subword Language Model for Query Auto-Completion
• Generalization by design: Shortcuts to Generalization in Deep Learning
• Delta Sampling R-BERT for limited data and low-light action recognition
• Deep Learning for Estimating Synaptic Health of Primary Neuronal Cell Culture
• Hybrid Backpropagation Parallel Reservoir Networks
• Hyper-optimization with Gaussian Process and Differential Evolution Algorithm
• Optimizing generalization on the train set: a novel gradient-based framework to train parameters and hyperparameters simultaneously
• Mastering Large Scale Multi-label Image Recognition with high efficiency overCamera trap images
• Drawing with AI -- Exploring Collaborative Inking Experiences Based on Mid-air Pointing and Reinforcement Learning
• How many images do I need? Understanding how sample size per class affects deep learning model performance metrics for balanced designs in autonomous wildlife monitoring