An Optimization and Generalization Analysis for Max-Pooling Networks
arXiv:2002.09781
Abstract
Max-Pooling operations are a core component of deep learning architectures. In particular, they are part of most convolutional architectures used in machine vision, since pooling is a natural approach to pattern detection problems. However, these architectures are not well understood from a theoretical perspective. For example, we do not understand when they can be globally optimized, and what is the effect of over-parameterization on generalization. Here we perform a theoretical analysis of a convolutional max-pooling architecture, proving that it can be globally optimized, and can generalize well even for highly over-parameterized models. Our analysis focuses on a data generating distribution inspired by pattern detection problem, where a "discriminative" pattern needs to be detected among "spurious" patterns. We empirically validate that CNNs significantly outperform fully connected networks in our setting, as predicted by our theoretical results.
References in corpus (22)
- Adam: A Method for Stochastic Optimization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- On Exact Computation with an Infinitely Wide Neural Net
- On Lazy Training in Differentiable Programming
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Gradient descent aligns the layers of deep linear networks
- Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
- Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
- Enhanced Convolutional Neural Tangent Kernels
- Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
- Risk and parameter convergence of logistic regression
- Generalization bounds for deep convolutional neural networks
- Implicit Bias of Gradient Descent on Linear Convolutional Networks
- Understanding Generalization and Optimization Performance of Deep CNNs
- On the Power and Limitations of Random Features for Understanding Neural Networks
- Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem
- Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?
- On the Expressive Power of Overlapping Architectures of Deep Learning
- Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
- Decoupling Gating from Linearity
- Computational Separation Between Convolutional and Fully-Connected Networks
- On the Learning Dynamics of Two-layer Nonlinear Convolutional Neural Networks