Minimum sharpness: Scale-invariant parameter-robustness of neural networks

Abstract

Toward achieving robust and defensive neural networks, the robustness against the weight parameters perturbations, i.e., sharpness, attracts attention in recent years (Sun et al., 2020). However, sharpness is known to remain a critical issue, "scale-sensitivity." In this paper, we propose a novel sharpness measure, Minimum Sharpness. It is known that NNs have a specific scale transformation that constitutes equivalent classes where functional properties are completely identical, and at the same time, their sharpness could change unlimitedly. We define our sharpness through a minimization problem over the equivalent NNs being invariant to the scale transformation. We also develop an efficient and exact technique to make the sharpness tractable, which reduces the heavy computational costs involved with Hessian. In the experiment, we observed that our sharpness has a valid correlation with the generalization of NNs and runs with less computational cost than existing sharpness measures.

arXiv comment9 pages, accepted to ICML 2021 Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI

References in corpus (11)

• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks
• Fantastic Generalization Measures and Where to Find Them
• Sharp Minima Can Generalize For Deep Nets
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Identifying Generalization Properties in Neural Networks
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• A Scale Invariant Flatness Measure for Deep Network Minima
• Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption
• Formalizing Generalization and Robustness of Neural Networks to Weight Perturbations