PyHessian: Neural Networks Through the Lens of the Hessian

Abstract

We present PYHESSIAN, a new scalable framework that enables fast computation of Hessian (i.e., second-order derivative) information for deep neural networks. PYHESSIAN enables fast computations of the top Hessian eigenvalues, the Hessian trace, and the full Hessian eigenvalue/spectral density, and it supports distributed-memory execution on cloud/supercomputer systems and is available as open source. This general framework can be used to analyze neural network models, including the topology of the loss landscape (i.e., curvature information) to gain insight into the behavior of different models/optimizers. To illustrate this, we analyze the effect of residual connections and Batch Normalization layers on the trainability of neural networks. One recent claim, based on simpler first-order analysis, is that residual connections and Batch Normalization make the loss landscape smoother, thus making it easier for Stochastic Gradient Descent to converge to a good solution. Our extensive analysis shows new finer-scale insights, demonstrating that, while conventional wisdom is sometimes validated, in other cases it is simply incorrect. In particular, we find that Batch Normalization does not necessarily make the loss landscape smoother, especially for shallower networks.

References in corpus (16)

• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Randomized algorithms for matrices and data
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
• Sub-sampled Newton Methods with Non-uniform Sampling
• Sub-Sampled Newton Methods I: Globally Convergent Algorithms
• A Progressive Batching L-BFGS Method for Machine Learning
• HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
• Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study
• Large batch size training of neural networks with adversarial training and second-order information
• Inexact Non-Convex Newton-Type Methods
• Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization

Cited by in corpus (22)

• Generative Data Augmentation for Commonsense Reasoning
• Applications and Techniques for Fast Machine Learning in Science
• Lipschitz Recurrent Neural Networks
• Large batch size training of neural networks with adversarial training and second-order information
• CPR: Classifier-Projection Regularization for Continual Learning
• Deep Learning Through the Lens of Example Difficulty
• HAWQV3: Dyadic Neural Network Quantization
• PowerNorm: Rethinking Batch Normalization in Transformers
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Network Quantization with Element-wise Gradient Scaling
• Hessian Eigenspectra of More Realistic Nonlinear Models
• Constraint-Based Regularization of Neural Networks
• Hessian-Aware Pruning and Optimal Neural Implant
• LocalNewton: Reducing Communication Bottleneck for Distributed Learning
• Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information
• Learnable Uncertainty under Laplace Approximations
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
• BEAR: Sketching BFGS Algorithm for Ultra-High Dimensional Feature Selection in Sublinear Memory
• Sparse sketches with small inversion bias
• RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions
• Observations on K-image Expansion of Image-Mixing Augmentation for Classification