Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem

Abstract

Empirical evidence suggests that neural networks with ReLU activations generalize better with over-parameterization. However, there is currently no theoretical analysis that explains this observation. In this work, we provide theoretical and empirical evidence that, in certain cases, overparameterized convolutional networks generalize better than small networks because of an interplay between weight clustering and feature exploration at initialization. We demonstrate this theoretically for a 3-layer convolutional neural network with max-pooling, in a novel setting which extends the XOR problem. We show that this interplay implies that with overparamterization, gradient descent converges to global minima with better generalization performance compared to global minima of small networks. Empirically, we demonstrate these phenomena for a 3-layer convolutional neural network in the MNIST task.

Cited by in corpus (14)

• Knowledge Distillation: A Survey
• A Survey of Human-in-the-loop for Machine Learning
• Finite Versus Infinite Neural Networks: an Empirical Study
• Recent advances in deep learning theory
• Network Pruning That Matters: A Case Study on Retraining Variants
• When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
• Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention
• When does gradient descent with logistic loss find interpolating two-layer networks?
• An Optimization and Generalization Analysis for Max-Pooling Networks
• Learning Boolean Circuits with Neural Networks
• Towards Understanding Learning in Neural Networks with Linear Teachers
• Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum
• Understanding How Over-Parametrization Leads to Acceleration: A case of learning a single teacher neuron
• DNNs as Layers of Cooperating Classifiers