Parameter Re-Initialization through Cyclical Batch Size Schedules

Abstract

Optimal parameter initialization remains a crucial problem for neural network training. A poor weight initialization may take longer to train and/or converge to sub-optimal solutions. Here, we propose a method of weight re-initialization by repeated annealing and injection of noise in the training process. We implement this through a cyclical batch size schedule motivated by a Bayesian perspective of neural network training. We evaluate our methods through extensive experiments on tasks in language modeling, natural language inference, and image classification. We demonstrate the ability of our method to improve language modeling performance by up to 7.91 perplexity and reduce training iterations by up to $61\%$, in addition to its flexibility in enabling snapshot ensembling and use with adversarial training.

arXiv commentPresented in Systems for Machine Learning Workshop at NeurIPS'18 conference

References in corpus (17)

• Explaining and Harnessing Adversarial Examples
• Recurrent Neural Network Regularization
• Wide Residual Networks
• Enhanced LSTM for Natural Language Inference
• One weird trick for parallelizing convolutional neural networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Don't Decay the Learning Rate, Increase the Batch Size
• Pointer Sentinel Mixture Models
• A large annotated corpus for learning natural language inference
• Three Factors Influencing Minima in SGD
• All you need is a good init
• Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
• Large batch size training of neural networks with adversarial training and second-order information
• Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior

Cited by in corpus (2)

• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Inefficiency of K-FAC for Large Batch Size Training