Towards Understanding the Generalization Bias of Two Layer Convolutional Linear Classifiers with Gradient Descent
arXiv:1802.04420
Abstract
A major challenge in understanding the generalization of deep learning is to explain why (stochastic) gradient descent can exploit the network architecture to find solutions that have good generalization performance when using high capacity models. We find simple but realistic examples showing that this phenomenon exists even when learning linear classifiers --- between two linear networks with the same capacity, the one with a convolutional layer can generalize better than the other when the data distribution has some underlying spatial structure. We argue that this difference results from a combination of the convolution architecture, data distribution and gradient descent, all of which are necessary to be included in a meaningful analysis. We provide a general analysis of the generalization performance as a function of data distribution and convolutional filter size, given gradient descent as the optimization algorithm, then interpret the results using concrete examples. Experimental results show that our analysis is able to explain what happens in our introduced examples.
arXiv commentThe 22nd International Conference on Artificial Intelligence and Statistics (AISTATS 2019)
References in corpus (10)
- Playing Atari with Deep Reinforcement Learning
- Understanding deep learning requires rethinking generalization
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Train faster, generalize better: Stability of stochastic gradient descent
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Spectrally-normalized margin bounds for neural networks
- When is a Convolutional Filter Easy To Learn?
- SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
- Implicit Bias of Gradient Descent on Linear Convolutional Networks
- The Implicit Bias of Gradient Descent on Separable Data