The effect of Leaky ReLUs on the training and generalization of overparameterized networks

Yinglong Guo, Shaohan Li, Gilad Lerman

We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specificall…