Fisher-Rao Metric, Geometry, and Complexity of Neural Networks

Abstract

We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity --- the Fisher-Rao norm --- that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks.

arXiv commentTo appear in the proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS) 2019

References in corpus (8)

• Understanding deep learning requires rethinking generalization
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Exploring Generalization in Deep Learning
• Spectrally-normalized margin bounds for neural networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Norm-Based Capacity Control in Neural Networks
• Failures of Gradient-Based Deep Learning
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach

Cited by in corpus (60)

• Quantum Natural Gradient
• Optimization Problems for Machine Learning: A Survey
• A Study on Overfitting in Deep Reinforcement Learning
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Fantastic Generalization Measures and Where to Find Them
• The Modern Mathematics of Deep Learning
• A Theoretical Analysis of Deep Q-Learning
• Are All Layers Created Equal?
• To understand deep learning we need to understand kernel learning
• Stronger generalization bounds for deep nets via a compression approach
• Pruning neural networks without any data by iteratively conserving synaptic flow
• On Warm-Starting Neural Network Training
• Three Mechanisms of Weight Decay Regularization
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• Identifying Mislabeled Data using the Area Under the Margin Ranking
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• A Priori Estimates of the Population Risk for Residual Networks
• Dimensionality compression and expansion in Deep Neural Networks
• On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning
• ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks
• On the Optimal Weighted ${\ell }_2$ Regularization in Overparameterized Linear Regression
• Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness
• Training Algorithm Matters for the Performance of Neural Network Potential: A Case Study of Adam and the Kalman Filter Optimizers
• A numerical approximation method for the Fisher-Rao distance between multivariate normal distributions
• Similarity of Neural Networks with Gradients
• PAC-Bayes Information Bottleneck
• A hybrid quantum algorithm to detect conical intersections
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Stationary Points of Shallow Neural Networks with Quadratic Activation Function
• Geometric Insights into the Convergence of Nonlinear TD Learning
• The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
• On the interplay between noise and curvature and its effect on optimization and generalization
• Geometrization of deep networks for the interpretability of deep learning systems
• Evaluation of Complexity Measures for Deep Learning Generalization in Medical Image Analysis
• Model Complexity of Deep Learning: A Survey
• Implicit Regularization via Neural Feature Alignment
• Neural Complexity Measures
• Why ResNet Works? Residuals Generalize
• Theory III: Dynamics and Generalization in Deep Networks
• In Search of Robust Measures of Generalization
• Information-Theoretic Local Minima Characterization and Regularization
• Approximation and bounding techniques for the Fisher-Rao distances between parametric statistical models
• An Information-Geometric Distance on the Space of Tasks
• Self-Regularity of Non-Negative Output Weights for Overparameterized Two-Layer Neural Networks
• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• CoNES: Convex Natural Evolutionary Strategies
• Detecting Memorization in ReLU Networks
• Understanding over-parameterized deep networks by geometrization
• Interpreting and Disentangling Feature Components of Various Complexity from DNNs
• Extreme Memorization via Scale of Initialization
• Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples
• Deep network as memory space: complexity, generalization, disentangled representation and interpretability
• The role of invariance in spectral complexity-based generalization bounds
• Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks
• Distribution-Dependent Analysis of Gibbs-ERM Principle
• Relative Flatness and Generalization
• Theory of Generative Deep Learning : Probe Landscape of Empirical Error via Norm Based Capacity Control
• The duality structure gradient descent algorithm: analysis and applications to neural networks
• Think Global, Act Local: Relating DNN generalisation and node-level SNR