Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training

Rie Johnson, Tong Zhang

As deep neural networks are highly expressive, it is important to find solutions with small generalization gap (the difference between the performance on the training data and unse…