Deep Curvature Suite
arXiv:1912.09656
Abstract
We present MLRG Deep Curvature suite, a PyTorch-based, open-source package for analysis and visualisation of neural network curvature and loss landscape. Despite of providing rich information into properties of neural network and useful for a various designed tasks, curvature information is still not made sufficient use for various reasons, and our method aims to bridge this gap. We present a primer, including its main practical desiderata and common misconceptions, of \textit{Lanczos algorithm}, the theoretical backbone of our package, and present a series of examples based on synthetic toy examples and realistic modern neural networks tested on CIFAR datasets, and show the superiority of our package against existing competing approaches for the similar purposes.
11 pages, 11 figures
References in corpus (18)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- The Loss Surfaces of Multilayer Networks
- Visualizing the Loss Landscape of Neural Nets
- GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- A Simple Baseline for Bayesian Uncertainty in Deep Learning
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
- Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
- Krylov Subspace Descent for Deep Learning
- Asymmetric Valleys: Beyond Sharp and Flat Local Minima
- The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
- Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
- An information and field theoretic approach to the grand canonical ensemble
- BackPACK: Packing more into backprop
- Entropic Spectral Learning for Large-Scale Graphs
- VBALD - Variational Bayesian Approximation of Log Determinants