Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis

Abstract

The notion of flat minima has played a key role in the generalization studies of deep learning models. However, existing definitions of the flatness are known to be sensitive to the rescaling of parameters. The issue suggests that the previous definitions of the flatness might not be a good measure of generalization, because generalization is invariant to such rescalings. In this paper, from the PAC-Bayesian perspective, we scrutinize the discussion concerning the flat minima and introduce the notion of normalized flat minima, which is free from the known scale dependence issues. Additionally, we highlight the scale dependence of existing matrix-norm based generalization error bounds similar to the existing flat minima definitions. Our modified notion of the flatness does not suffer from the insufficiency, either, suggesting it might provide better hierarchy in the hypothesis class.

References in corpus (7)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Decoupled Weight Decay Regularization
• Variational Dropout and the Local Reparameterization Trick
• Sharp Minima Can Generalize For Deep Nets
• Functional Variational Bayesian Neural Networks
• On the properties of variational approximations of Gibbs posteriors
• Identifying Generalization Properties in Neural Networks

Cited by in corpus (6)

• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• A Generalization Bound for Online Variational Inference
• Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
• Flatness is a False Friend
• Why flatness does and does not correlate with generalization for deep neural networks
• Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation