Implicit Regularization in Deep Learning

Abstract

In an attempt to better understand generalization in deep learning, we study several possible explanations. We show that implicit regularization induced by the optimization method is playing a key role in generalization and success of deep learning models. Motivated by this view, we study how different complexity measures can ensure generalization and explain how optimization algorithms can implicitly regularize complexity measures. We empirically investigate the ability of these measures to explain different observed phenomena in deep learning. We further study the invariances in neural networks, suggest complexity measures and optimization algorithms that have similar invariances to those in neural networks and evaluate them on a number of learning tasks.

PhD Thesis

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Breaking the Curse of Dimensionality with Convex Neural Networks
• No More Pesky Learning Rates
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Unitary Evolution Recurrent Neural Networks
• Spectrally-normalized margin bounds for neural networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Natural Neural Networks
• Generalization Error of Invariant Classifiers
• Regularization and nonlinearities for neural language models: when are they needed?

Cited by in corpus (43)

• Spectrally-normalized margin bounds for neural networks
• Data augmentation instead of explicit regularization
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• Implicit Geometric Regularization for Learning Shapes
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• Exact expressions for double descent and implicit regularization via surrogate random design
• Bad Global Minima Exist and SGD Can Reach Them
• Transport Analysis of Infinitely Deep Neural Network
• Consensus Control for Decentralized Deep Learning
• NeurIPS 2020 Competition: Predicting Generalization in Deep Learning
• Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime
• Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks
• Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization
• On the Benefits of Invariance in Neural Networks
• Observational Overfitting in Reinforcement Learning
• Implicit competitive regularization in GANs
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Understanding Adversarial Robustness Through Loss Landscape Geometries
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• On Mixup Regularization
• Gradient-Free Learning Based on the Kernel and the Range Space
• On the interplay between noise and curvature and its effect on optimization and generalization
• Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
• Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case Study
• Analytic Network Learning
• Model Complexity of Deep Learning: A Survey
• Robust and On-the-fly Dataset Denoising for Image Classification
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer
• Deep Learning for Inverse Problems: Bounds and Regularizers
• Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum
• The Dynamics of Gradient Descent for Overparametrized Neural Networks
• On the Bias-Variance Tradeoff: Textbooks Need an Update
• Extrapolation for Large-batch Training in Deep Learning
• Non-Asymptotic Analysis for Two Time-scale TDC with General Smooth Function Approximation
• Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Predictive Model Degrees of Freedom in Linear Regression
• On the potential for open-endedness in neural networks
• Extracting representations of cognition across neuroimaging studies improves brain decoding
• Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space