Implicit Rugosity Regularization via Data Augmentation
arXiv:1905.11639
Abstract
Deep (neural) networks have been applied productively in a wide range of supervised and unsupervised learning tasks. Unlike classical machine learning algorithms, deep networks typically operate in the \emph{overparameterized} regime, where the number of parameters is larger than the number of training data points. Consequently, understanding the generalization properties and the role of (explicit or implicit) regularization in these networks is of great importance. In this work, we explore how the oft-used heuristic of \emph{data augmentation} imposes an {\em implicit regularization} penalty of a novel measure of the \emph{rugosity} or "roughness" based on the tangent Hessian of the function fit to the training data.
15 pages, 12 figures
References in corpus (21)
- The Effectiveness of Data Augmentation in Image Classification using Deep Learning
- Wide Residual Networks
- Understanding deep learning requires rethinking generalization
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Deep Neural Networks as Gaussian Processes
- On Exact Computation with an Infinitely Wide Neural Net
- Train faster, generalize better: Stability of stochastic gradient descent
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Spectrally-normalized margin bounds for neural networks
- Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
- Approximating Continuous Functions by ReLU Nets of Minimal Width
- Representation Benefits of Deep Feedforward Networks
- Nonlinear Approximation and (Deep) ReLU Networks
- A Kernel Theory of Modern Data Augmentation
- Stronger generalization bounds for deep nets via a compression approach
- A Group-Theoretic Framework for Data Augmentation
- Complexity of Linear Regions in Deep Networks
- Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
- How do infinite width bounded norm networks look in function space?