Positively Scale-Invariant Flatness of ReLU Neural Networks

Mingyang Yi, Qi Meng, Wei Chen +2

It was empirically confirmed by Keskar et al.\cite{SharpMinima} that flatter minima generalize better. However, for the popular ReLU network, sharp minimum can also generalize well…