A Scale Invariant Flatness Measure for Deep Network Minima

Abstract

It has been empirically observed that the flatness of minima obtained from training deep networks seems to correlate with better generalization. However, for deep networks with positively homogeneous activations, most measures of sharpness/flatness are not invariant to rescaling of the network parameters, corresponding to the same function. This means that the measure of flatness/sharpness can be made as small or as large as possible through rescaling, rendering the quantitative measures meaningless. In this paper we show that for deep networks with positively homogenous activations, these rescalings constitute equivalence relations, and that these equivalence relations induce a quotient manifold structure in the parameter space. Using this manifold structure and an appropriate metric, we propose a Hessian-based measure for flatness that is invariant to rescaling. We use this new measure to confirm the proposition that Large-Batch SGD minima are indeed sharper than Small-Batch SGD minima.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Sequence to Sequence Learning with Neural Networks
• Visualizing the Loss Landscape of Neural Nets
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Identifying Generalization Properties in Neural Networks
• Projection Based Weight Normalization for Deep Neural Networks

Cited by in corpus (9)

• On the interplay between noise and curvature and its effect on optimization and generalization
• SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
• Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
• Flatness is a False Friend
• Why flatness does and does not correlate with generalization for deep neural networks
• A Reparameterization-Invariant Flatness Measure for Deep Neural Networks
• Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation
• Relative Flatness and Generalization
• Minimum sharpness: Scale-invariant parameter-robustness of neural networks