In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Abstract

We present experiments demonstrating that some other form of capacity control, different from network size, plays a central role in learning multilayer feed-forward networks. We argue, partially through analogy to matrix factorization, that this is an inductive bias that can help shed light on deep learning.

arXiv comment9 pages, 2 figures

Cited by in corpus (41)

• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Train faster, generalize better: Stability of stochastic gradient descent
• Exploring Generalization in Deep Learning
• Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization
• The Role of Neural Network Activation Functions
• The Modern Mathematics of Deep Learning
• Just Interpolate: Kernel "Ridgeless" Regression Can Generalize
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Understanding the Failure Modes of Out-of-Distribution Generalization
• What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory
• Topology of Learning in Artificial Neural Networks
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• Exact expressions for double descent and implicit regularization via surrogate random design
• Bad Global Minima Exist and SGD Can Reach Them
• Deep Learning Meets Sparse Regularization: A Signal Processing Perspective
• Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime
• Memorization in Overparameterized Autoencoders
• SINR: Deconvolving Circular SAS Images Using Implicit Neural Representations
• Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness
• Distance-Based Learning from Errors for Confidence Calibration
• The Low-Rank Simplicity Bias in Deep Networks
• Should attention be all we need? The epistemic and ethical implications of unification in machine learning
• Probing the Purview of Neural Networks via Gradient Analysis
• The Role of Information Complexity and Randomization in Representation Learning
• Exploring Weight Importance and Hessian Bias in Model Pruning
• Data-Dependent Path Normalization in Neural Networks
• Regression as Classification: Influence of Task Formulation on Neural Network Features
• A Learning Theoretic Perspective on Local Explainability
• On the approximation of functions by tanh neural networks
• Pre-interpolation loss behaviour in neural networks
• Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscape
• On Alignment in Deep Linear Neural Networks
• Rethinking Gauss-Newton for learning over-parameterized models
• Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows
• Informative regularization for a multi-layer perceptron RR Lyrae classifier under data shift
• Implicit regularization of deep residual networks towards neural ODEs
• Inherent Noise in Gradient Based Methods
• On the Regularization of Autoencoders
• A Functional Perspective on Learning Symmetric Functions with Neural Networks
• Painless step size adaptation for SGD