Hessian-based Analysis of Large Batch Training and Robustness to Adversaries

Abstract

Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, we perform a Hessian based study to analyze exactly how the landscape of the loss function changes when training with large batch size. We compute the true Hessian spectrum, without approximation, by back-propagating the second derivative. Extensive experiments on multiple networks show that saddle-points are not the cause for generalization gap of large batch size training, and the results consistently show that large batch converges to points with noticeably higher Hessian spectrum. Furthermore, we show that robust training allows one to favor flat areas, as points with large Hessian spectrum show poor robustness to adversarial perturbation. We further study this relationship, and provide empirical and theoretical proof that the inner loop for robust training is a saddle-free optimization problem \textit{almost everywhere}. We present detailed experiments with five different network architectures, including a residual network, tested on MNIST, CIFAR-10, and CIFAR-100 datasets. We have open sourced our method which can be accessed at [1].

arXiv commentPresented in NeurIPS'18 conference

Cited by in corpus (37)

• Understanding and Robustifying Differentiable Architecture Search
• Don't Use Large Mini-Batches, Use Local SGD
• How Does Learning Rate Decay Help Modern Neural Networks?
• Characterizing possible failure modes in physics-informed neural networks
• HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• Logit Pairing Methods Can Fool Gradient-Based Attacks
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
• Large batch size training of neural networks with adversarial training and second-order information
• PyHessian: Neural Networks Through the Lens of the Hessian
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• The Implicit and Explicit Regularization Effects of Dropout
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning
• Taxonomizing local versus global structure in neural network loss landscapes
• The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
• Noisy Recurrent Neural Networks
• Parameter Re-Initialization through Cyclical Batch Size Schedules
• Learning with Gradient Descent and Weakly Convex Losses
• Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
• Deep Curvature Suite
• Why flatness does and does not correlate with generalization for deep neural networks
• Inefficiency of K-FAC for Large Batch Size Training
• Flatness is a False Friend
• Self-paced Data Augmentation for Training Neural Networks
• Analysis of Random Perturbations for Robust Convolutional Neural Networks
• Training few-shot classification via the perspective of minibatch and pretraining
• Relative Flatness and Generalization
• Regularization in ResNet with Stochastic Depth
• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II
• On tuning deep learning models: a data mining perspective