Implicit Regularization in Over-parameterized Neural Networks

Abstract

Over-parameterized neural networks generalize well in practice without any explicit regularization. Although it has not been proven yet, empirical evidence suggests that implicit regularization plays a crucial role in deep learning and prevents the network from overfitting. In this work, we introduce the gradient gap deviation and the gradient deflection as statistical measures corresponding to the network curvature and the Hessian matrix to analyze variations of network derivatives with respect to input parameters, and investigate how implicit regularization works in ReLU neural networks from both theoretical and empirical perspectives. Our result reveals that the network output between each pair of input samples is properly controlled by random initialization and stochastic gradient descent to keep interpolating between samples almost straight, which results in low complexity of over-parameterized neural networks.

References in corpus (3)

• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• A Surprising Linear Relationship Predicts Test Performance in Deep Networks

Cited by in corpus (4)

• Exact expressions for double descent and implicit regularization via surrogate random design
• Implicit competitive regularization in GANs
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer
• Explicit Regularization of Stochastic Gradient Methods through Duality