Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Abstract

At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function $f_{\theta }$ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We prove the positive-definiteness of the limiting NTK when the data is supported on the sphere and the non-linearity is non-polynomial. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function $f_{\theta }$ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

References in corpus (1)

• Strictly and non-strictly positive definite functions on spheres

Cited by in corpus (715)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Scaling Laws for Neural Language Models
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Benign Overfitting in Linear Regression
• Conservative Q-Learning for Offline Reinforcement Learning
• Artificial neural networks for neuroscientists: A primer
• On Exact Computation with an Infinitely Wide Neural Net
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
• Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• The Principles of Deep Learning Theory
• Kolmogorov Arnold Informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov Arnold Networks
• Deep Network Approximation for Smooth Functions
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Scaling description of generalization with number of parameters in deep learning
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Scaling Laws for Autoregressive Generative Modeling
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Explaining Neural Scaling Laws
• Optimization for deep learning: theory and algorithms
• The Random Feature Model for Input-Output Maps between Banach Spaces
• Neural Network Approximation: Three Hidden Layers Are Enough
• The Modern Mathematics of Deep Learning
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
• A Theoretical Analysis of Deep Q-Learning
• When and why PINNs fail to train: A neural tangent kernel perspective
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• Are All Layers Created Equal?
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Self-Distillation Amplifies Regularization in Hilbert Space
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Towards understanding the power of quantum kernels in the NISQ era
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Physics-informed radial basis network (PIRBN): A local approximating neural network for solving nonlinear PDEs
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Understanding the Role of Training Regimes in Continual Learning
• Biology and medicine in the landscape of quantum advantages
• Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Coresets via Bilevel Optimization for Continual Learning and Streaming
• Generalisation error in learning with random features and the hidden manifold model
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
• Enhanced Convolutional Neural Tangent Kernels
• Representation Learning via Quantum Neural Tangent Kernels
• The Global Landscape of Neural Networks: An Overview
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Neural Networks and Quantum Field Theory
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Differentially Private Learning Needs Better Features (or Much More Data)
• Towards Understanding the Spectral Bias of Deep Learning
• Overparameterized Neural Networks Implement Associative Memory
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Towards Characterizing Divergence in Deep Q-Learning
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• The large learning rate phase of deep learning: the catapult mechanism
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• Kernel and Rich Regimes in Overparametrized Models
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Continual Deep Learning by Functional Regularisation of Memorable Past
• Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems
• Convergence of Adversarial Training in Overparametrized Neural Networks
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• On Information Plane Analyses of Neural Network Classifiers -- A Review
• Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Stiffness: A New Perspective on Generalization in Neural Networks
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
• The Gaussian equivalence of generative models for learning with shallow neural networks
• Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
• Quantum Machine Learning for Finance
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• An Improved Analysis of Training Over-parameterized Deep Neural Networks
• On Interpretability of Artificial Neural Networks: A Survey
• Asymptotics of Wide Networks from Feynman Diagrams
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Neural Architecture Search without Training
• Graphon Neural Networks and the Transferability of Graph Neural Networks
• Scalable Uncertainty Quantification for Deep Operator Networks using Randomized Priors
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Reliability and Interpretability in Science and Deep Learning
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Fast and Sample-Efficient Interatomic Neural Network Potentials for Molecules and Materials Based on Gaussian Moments
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• A Selective Overview of Deep Learning
• Spectral-wise Implicit Neural Representation for Hyperspectral Image Reconstruction
• Plasma Surrogate Modelling using Fourier Neural Operators
• Finite Versus Infinite Neural Networks: an Empirical Study
• Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
• A mean-field limit for certain deep neural networks
• A Fine-Grained Spectral Perspective on Neural Networks
• Explicit Regularisation in Gaussian Noise Injections
• Aero-Nef: Neural Fields for Rapid Aircraft Aerodynamics Simulations
• Gradient Dynamics of Shallow Univariate ReLU Networks
• Mean-field inference methods for neural networks
• Learning Specialized Activation Functions for Physics-informed Neural Networks
• Generalization Bounds for Sparse Random Feature Expansions
• Do Wider Neural Networks Really Help Adversarial Robustness?
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance
• Understanding Self-supervised Learning with Dual Deep Networks
• Neural Kernels Without Tangents
• The Functional Neural Process
• Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks
• Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective
• Meta-learning Pseudo-differential Operators with Deep Neural Networks
• Neural Thompson Sampling
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Uncertainty Quantification and Deep Ensembles
• Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
• Rethinking Positional Encoding
• FreeREA: Training-Free Evolution-based Architecture Search
• Thermally Averaged Magnetic Anisotropy Tensors via Machine Learning Based on Gaussian Moments
• A Non-Asymptotic Analysis for Stein Variational Gradient Descent
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• On the different regimes of Stochastic Gradient Descent
• Learning through atypical "phase transitions" in overparameterized neural networks
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• Kernel methods through the roof: handling billions of points efficiently
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Feature Space Singularity for Out-of-Distribution Detection
• Mathematical Models of Overparameterized Neural Networks
• On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks
• Asymptotics of representation learning in finite Bayesian neural networks
• On the training dynamics of deep networks with $L_2$ regularization
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• Learning with invariances in random features and kernel models
• Deep Learning on a Data Diet: Finding Important Examples Early in Training
• Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation
• Kernel Alignment Risk Estimator: Risk Prediction from Training Data
• The Quantum Path Kernel: a Generalized Quantum Neural Tangent Kernel for Deep Quantum Machine Learning
• Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks
• Neural Parametric Fokker-Planck Equations
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• Universality Laws for High-Dimensional Learning with Random Features
• Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows
• Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?
• Recent advances in deep learning theory
• Learning Physical Models that Can Respect Conservation Laws
• Finite-time Lyapunov exponents of deep neural networks
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Solving and Learning Nonlinear PDEs with Gaussian Processes
• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equations
• Wide and Deep Neural Networks Achieve Optimality for Classification
• Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• Asymptotics of Ridge (less) Regression under General Source Condition
• Feature Learning in Infinite-Width Neural Networks
• Network size and weights size for memorization with two-layers neural networks
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
• CoIL: Coordinate-based Internal Learning for Imaging Inverse Problems
• Emergent properties of the local geometry of neural loss landscapes
• Tensor Programs III: Neural Matrix Laws
• Deep Q-learning: a robust control approach
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent
• Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process
• Finite size corrections for neural network Gaussian processes
• Neural Contextual Bandits with Deep Representation and Shallow Exploration
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• Structure Probing Neural Network Deflation
• Observational Overfitting in Reinforcement Learning
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
• Neural Networks Learning and Memorization with (almost) no Over-Parameterization
• Gaussian Universality of Perceptrons with Random Labels
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
• On Power Laws in Deep Ensembles
• A Corrective View of Neural Networks: Representation, Memorization and Learning
• A Spectral-based Physics-informed Finite Operator Learning for Prediction of Mechanical Behavior of Microstructures
• AEVA: Black-box Backdoor Detection Using Adversarial Extreme Value Analysis
• Repulsive Deep Ensembles are Bayesian
• Selective Amnesia: On Efficient, High-Fidelity and Blind Suppression of Backdoor Effects in Trojaned Machine Learning Models
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
• A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix
• k-space Physics-informed Neural Network (k-PINN) for Compressed Spectral Mapping and Efficient Inversion of Vibrations in Thin Composite Laminates
• Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network
• A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning
• Neural Projected Quantum Dynamics: a systematic study
• Towards Understanding Theoretical Advantages of Complex-Reaction Networks
• Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
• On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spaces
• Phase diagram for two-layer ReLU neural networks at infinite-width limit
• Interpolating Classifiers Make Few Mistakes
• A Finite-Time Analysis of Q-Learning with Neural Network Function Approximation
• Neural-BO: A Black-box Optimization Algorithm using Deep Neural Networks
• Towards NNGP-guided Neural Architecture Search
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
• Directional convergence and alignment in deep learning
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint
• Mean-Field Neural ODEs via Relaxed Optimal Control
• On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
• What can linearized neural networks actually say about generalization?
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing
• Asymptotics of Wide Convolutional Neural Networks
• On Convergence and Generalization of Dropout Training
• Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently
• On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks
• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
• Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• Temperature check: theory and practice for training models with softmax-cross-entropy losses
• Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction
• A Study of Gradient Variance in Deep Learning
• On the infinite width limit of neural networks with a standard parameterization
• On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels
• Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
• Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
• Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
• Similarity of Neural Networks with Gradients
• Robust Pruning at Initialization
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
• On the geometry of generalization and memorization in deep neural networks
• Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
• A Unifying View on Implicit Bias in Training Linear Neural Networks
• Effect of Activation Functions on the Training of Overparametrized Neural Nets
• NN2Poly: A polynomial representation for deep feed-forward artificial neural networks
• Reservoir Computing meets Recurrent Kernels and Structured Transforms
• On the Similarity between the Laplace and Neural Tangent Kernels
• Deep Graph Similarity Learning: A Survey
• Linear Mode Connectivity in Multitask and Continual Learning
• Approximate Inference Turns Deep Networks into Gaussian Processes
• Generalization bounds for deep learning
• Non-Gaussian processes and neural networks at finite widths
• Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity
• The Low-Rank Simplicity Bias in Deep Networks
• Regularized linear autoencoders recover the principal components, eventually
• An Unconstrained Layer-Peeled Perspective on Neural Collapse
• Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits
• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
• The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
• Gradients as Features for Deep Representation Learning
• Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime
• Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
• Loss landscapes and optimization in over-parameterized non-linear systems and neural networks
• Learning Parities with Neural Networks
• Reproducing Activation Function for Deep Learning
• On the expected behaviour of noise regularised deep neural networks as Gaussian processes
• Improving predictions of Bayesian neural nets via local linearization
• A linearized framework and a new benchmark for model selection for fine-tuning
• EE-Net: Exploitation-Exploration Neural Networks in Contextual Bandits
• Laziness, Barren Plateau, and Noise in Machine Learning
• Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
• Neural tangent kernels, transportation mappings, and universal approximation
• On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths
• Stable behaviour of infinitely wide deep neural networks
• Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
• Interference and Generalization in Temporal Difference Learning
• Graph Neural Bandits
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• Learning Deep Graph Representations via Convolutional Neural Networks
• Deep Equals Shallow for ReLU Networks in Kernel Regimes
• Spatiotemporal Implicit Neural Representation as a Generalized Traffic Data Learner
• Metric Flows with Neural Networks
• Kernelized Heterogeneous Risk Minimization
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• A Bayesian Perspective on Training Speed and Model Selection
• The Local Elasticity of Neural Networks
• Batch Normalization Orthogonalizes Representations in Deep Random Networks
• Early-stopped neural networks are consistent
• Towards Understanding Hierarchical Learning: Benefits of Neural Representations
• A Unified Paths Perspective for Pruning at Initialization
• Convex Geometry and Duality of Over-parameterized Neural Networks
• Quantum-enhanced neural networks in the neural tangent kernel framework
• On neural network kernels and the storage capacity problem
• Towards Understanding the Condensation of Neural Networks at Initial Training
• NASI: Label- and Data-agnostic Neural Architecture Search at Initialization
• Deep Active Learning by Leveraging Training Dynamics
• The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
• Infinite Neural Network Quantum States: Entanglement and Training Dynamics
• Is SGD a Bayesian sampler? Well, almost
• On the Global Convergence of Training Deep Linear ResNets
• Deep Learning is Singular, and That's Good
• Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs
• Approximate is Good Enough: Probabilistic Variants of Dimensional and Margin Complexity
• Implicit Regularization of Random Feature Models
• Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
• Information-Theoretic Generalization Bounds for Stochastic Gradient Descent
• Geometric Insights into the Convergence of Nonlinear TD Learning
• Information in Infinite Ensembles of Infinitely-Wide Neural Networks
• How Powerful are Shallow Neural Networks with Bandlimited Random Weights?
• On the asymptotics of wide networks with polynomial activations
• Revisiting BFloat16 Training
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• On Sequential Bayesian Inference for Continual Learning
• Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality
• Generalized Leverage Score Sampling for Neural Networks
• Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
• Early Stopping in Deep Networks: Double Descent and How to Eliminate it
• On Sparsity in Overparametrised Shallow ReLU Networks
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent
• Confidence Adaptive Regularization for Deep Learning with Noisy Labels
• Framing RNN as a kernel method: A neural ODE approach
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Solving PDEs on Unknown Manifolds with Machine Learning
• Label Noise SGD Provably Prefers Flat Global Minimizers
• Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
• On the interplay between data structure and loss function in classification problems
• Regularizing Neural Networks via Adversarial Model Perturbation
• On the Estimation of Derivatives Using Plug-in Kernel Ridge Regression Estimators
• Hardness of Learning Neural Networks with Natural Weights
• Memorizing Gaussians with no over-parameterizaion via gradient decent on neural networks
• Symmetry & critical points for a model shallow neural network
• On Infinite-Width Hypernetworks
• Characterizing Structural Regularities of Labeled Data in Overparameterized Models
• Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Scaling Limit of Neural Networks with the Xavier Initialization and Convergence to a Global Minimum
• Information Geometry of Orthogonal Initializations and Training
• Stochastic Training is Not Necessary for Generalization
• To Each Optimizer a Norm, To Each Norm its Generalization
• Implicit Regularization via Neural Feature Alignment
• Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels
• Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation
• Relative stability toward diffeomorphisms indicates performance in deep nets
• The Recurrent Neural Tangent Kernel
• Reinforcement Learning with Function Approximation: From Linear to Nonlinear
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
• Exact posterior distributions of wide Bayesian neural networks
• Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK
• The asymptotic spectrum of the Hessian of DNN throughout training
• Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
• Global Convergence of Second-order Dynamics in Two-layer Neural Networks
• Posterior Consistency of Semi-Supervised Regression on Graphs
• Dataset Meta-Learning from Kernel Ridge-Regression
• Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics
• When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
• Convex Regularization Behind Neural Reconstruction
• The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
• Wide Neural Networks with Bottlenecks are Deep Gaussian Processes
• Predicting Training Time Without Training
• GradSign: Model Performance Inference with Theoretical Insights
• Shift Invariance Can Reduce Adversarial Robustness
• Path Integral Sampler: a stochastic control approach for sampling
• Benign Overfitting and Noisy Features
• Spline Positional Encoding for Learning 3D Implicit Signed Distance Fields
• Learning Deep ReLU Networks Is Fixed-Parameter Tractable
• Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networks
• Precise characterization of the prior predictive distribution of deep ReLU networks
• Deep frequency principle towards understanding why deeper learning is faster
• Hierarchical Learning to Solve Partial Differential Equations Using Physics-Informed Neural Networks
• On Energy-Based Models with Overparametrized Shallow Neural Networks
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Noether: The More Things Change, the More Stay the Same
• Why bigger is not always better: on finite and infinite neural networks
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• Implicit Regularization and Convergence for Weight Normalization
• A global convergence theory for deep ReLU implicit networks via over-parameterization
• Neural Fields in Visual Computing and Beyond
• Non-asymptotic approximations of neural networks by Gaussian processes
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer
• Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time
• Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
• A Spectral Analysis of Dot-product Kernels
• Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy
• Computing on Functions Using Randomized Vector Representations
• Characteristics of Monte Carlo Dropout in Wide Neural Networks
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• On generalization bounds for deep networks based on loss surface implicit regularization
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• Optimization and Generalization Analysis of Transduction through Gradient Boosting and Application to Multi-scale Graph Neural Networks
• Fast Learning in Reproducing Kernel Krein Spaces via Signed Measures
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory
• Neural Bandit with Arm Group Graph
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach
• The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation
• The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks
• Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)
• Meta-Learning with Neural Tangent Kernels
• On Random Kernels of Residual Architectures
• Infinitely Wide Tensor Networks as Gaussian Process
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Scalable Training of Inference Networks for Gaussian-Process Models
• Wide Neural Networks Forget Less Catastrophically
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Finding trainable sparse networks through Neural Tangent Transfer
• A Deep Conditioning Treatment of Neural Networks
• Fooling Adversarial Training with Inducing Noise
• Implicit Gradient Regularization
• Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks
• Learning with Gradient Descent and Weakly Convex Losses
• Sparse Quantized Spectral Clustering
• Kernel Stein Generative Modeling
• Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
• Consistency of Empirical Bayes And Kernel Flow For Hierarchical Parameter Estimation
• When does gradient descent with logistic loss find interpolating two-layer networks?
• Neural Tangent Kernel Maximum Mean Discrepancy
• On the Universality of the Double Descent Peak in Ridgeless Regression
• Universal scaling laws in the gradient descent training of neural networks
• On the Double Descent of Random Features Models Trained with SGD
• Norm-based generalisation bounds for multi-class convolutional neural networks
• The critical locus of overparameterized neural networks
• Global Convergence of Three-layer Neural Networks in the Mean Field Regime
• Towards Evaluating the Robustness of Neural Networks Learned by Transduction
• A Recipe for Global Convergence Guarantee in Deep Neural Networks
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
• Learning time-scales in two-layers neural networks
• Meta-learning with negative learning rates
• Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models
• Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines
• On the benefit of overparameterization in state reconstruction
• A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks
• Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
• Multi-facet Contextual Bandits: A Neural Network Perspective
• Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions
• Efficient Online Bayesian Inference for Neural Bandits
• Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
• On a Sparse Shortcut Topology of Artificial Neural Networks
• Analysis of Knowledge Transfer in Kernel Regime
• $\alpha$-Stable convergence of heavy-tailed infinitely-wide neural networks
• Scale Mixtures of Neural Network Gaussian Processes
• BINet: Learning to Solve Partial Differential Equations with Boundary Integral Networks
• Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective
• Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity
• Holdout SGD: Byzantine Tolerant Federated Learning
• Distinguishing rule- and exemplar-based generalization in learning systems
• Pretrained Embeddings for E-commerce Machine Learning: When it Fails and Why?
• Analysis of Structured Deep Kernel Networks
• Credit Assignment Through Broadcasting a Global Error Vector
• Gradient Descent in RKHS with Importance Labeling
• Convergence rates for gradient descent in the training of overparameterized artificial neural networks with piecewise affine activation
• Implicit bias of deep linear networks in the large learning rate phase
• Learning with convolution and pooling operations in kernel methods
• Kernel-Based Smoothness Analysis of Residual Networks
• Improved generalization by noise enhancement
• Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms
• FEAR: A Simple Lightweight Method to Rank Architectures
• LNPT: Label-free Network Pruning and Training
• Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology
• Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• Rate-Distortion Analysis of Minimum Excess Risk in Bayesian Learning
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets
• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance
• Generic Neural Architecture Search via Regression
• Which Minimizer Does My Neural Network Converge To?
• Training Two-Layer ReLU Networks with Gradient Descent is Inconsistent
• Train-by-Reconnect: Decoupling Locations of Weights from their Values
• Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimes
• Learning to Pool in Graph Neural Networks for Extrapolation
• Dynamically Stable Infinite-Width Limits of Neural Classifiers
• DNN-Based Topology Optimisation: Spatial Invariance and Neural Tangent Kernel
• Wide stochastic networks: Gaussian limit and PAC-Bayesian training
• Scaling Neural Tangent Kernels via Sketching and Random Features
• The Dynamics of Gradient Descent for Overparametrized Neural Networks
• SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning
• Embedding Principle of Loss Landscape of Deep Neural Networks
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks
• RATT: Leveraging Unlabeled Data to Guarantee Generalization
• Analysis of One-Hidden-Layer Neural Networks via the Resolvent Method
• A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection
• Offline Neural Contextual Bandits: Pessimism, Optimization and Generalization
• On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models
• Approximating How Single Head Attention Learns
• Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces
• On the Approximation Power of Two-Layer Networks of Random ReLUs
• The Three Stages of Learning Dynamics in High-Dimensional Kernel Methods
• Online Limited Memory Neural-Linear Bandits with Likelihood Matching
• Implicit Regularization in Tensor Factorization
• Contrastive Embeddings for Neural Architectures
• On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime
• Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?
• Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis
• Trap of Feature Diversity in the Learning of MLPs
• Are wider nets better given the same number of parameters?
• Uncertainty Quantification From Scaling Laws in Deep Neural Networks
• When is Importance Weighting Correction Needed for Covariate Shift Adaptation?
• Subaging in underparametrized Deep Neural Networks
• Classification Protocols with Minimal Disclosure
• Extreme Memorization via Scale of Initialization
• A finite sample analysis of the benign overfitting phenomenon for ridge function estimation
• Dropout as a Regularizer of Interaction Effects
• The Care Label Concept: A Certification Suite for Trustworthy and Resource-Aware Machine Learning
• Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum
• An analytic theory of shallow networks dynamics for hinge loss classification
• The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization
• Towards Adversarial Robustness via Transductive Learning
• Nonasymptotic Guarantees for Spiked Matrix Recovery with Generative Priors
• Optimal Gradient-based Algorithms for Non-concave Bandit Optimization
• Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums
• Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs
• Adversarial Robustness Guarantees for Random Deep Neural Networks
• Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics
• Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
• Subquadratic Overparameterization for Shallow Neural Networks
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• On the Convergence of Gradient Descent in GANs: MMD GAN As a Gradient Flow
• Estimating informativeness of samples with Smooth Unique Information
• Fundamental Tradeoffs in Distributionally Adversarial Training
• The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods
• Mean field theory for deep dropout networks: digging up gradient backpropagation deeply
• Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension
• Generalization Error of Generalized Linear Models in High Dimensions
• Collegial Ensembles
• On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architectures
• Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
• General Probabilistic Surface Optimization and Log Density Estimation
• Provable Regret Bounds for Deep Online Learning and Control
• From the Greene--Wu Convolution to Gradient Estimation over Riemannian Manifolds
• Uniform Convergence, Adversarial Spheres and a Simple Remedy
• Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations
• Effect of the initial configuration of weights on the training and function of artificial neural networks
• MixCon: Adjusting the Separability of Data Representations for Harder Data Recovery
• A Temporal Kernel Approach for Deep Learning with Continuous-time Information
• Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks
• Why is AI hard and Physics simple?
• Explaining generalization in deep learning: progress and fundamental limits
• Fast Approximation and Estimation Bounds of Kernel Quadrature for Infinitely Wide Models
• Appearance of Random Matrix Theory in Deep Learning
• Properties of the After Kernel
• Two-Level K-FAC Preconditioning for Deep Learning
• On the validity of kernel approximations for orthogonally-initialized neural networks
• Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networks
• Making Method of Moments Great Again? -- How can GANs learn distributions
• Deep Neural Networks as Point Estimates for Deep Gaussian Processes
• Deep limits and cut-off phenomena for neural networks
• Achieving Small Test Error in Mildly Overparameterized Neural Networks
• Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Space
• Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks
• Learning curves for Gaussian process regression with power-law priors and targets
• Pure Exploration in Kernel and Neural Bandits
• The equivalence between Stein variational gradient descent and black-box variational inference
• Towards a General Theory of Infinite-Width Limits of Neural Classifiers
• Robust Generalization of Quadratic Neural Networks via Function Identification
• Privacy-Preserving Constrained Domain Generalization via Gradient Alignment
• Model Repair: Robust Recovery of Over-Parameterized Statistical Models
• Quantifying Epistemic Uncertainty in Deep Learning
• A Chain Graph Interpretation of Real-World Neural Networks
• Provably Training Overparameterized Neural Network Classifiers with Non-convex Constraints
• Rethinking Gauss-Newton for learning over-parameterized models
• Implicit Bias of Linear RNNs
• Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stopping
• Neural Tangent Kernel Empowered Federated Learning
• On the Provable Generalization of Recurrent Neural Networks
• Generalization error of minimum weighted norm and kernel interpolation
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• The Elastic Lottery Ticket Hypothesis
• Beyond Lazy Training for Over-parameterized Tensor Decomposition
• Phase Diagram of Initial Condensation for Two-layer Neural Networks
• Inference of dynamic systems from noisy and sparse data via manifold-constrained Gaussian processes
• Learning Boolean Circuits with Neural Networks
• MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization
• Learning and Generalization in RNNs
• Generalisation under gradient descent via deterministic PAC-Bayes
• Summary statistics of learning link changing neural representations to behavior
• The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization
• MUSO: Achieving Exact Machine Unlearning in Over-Parameterized Regimes
• Minimal Sufficient Views: A DNN model making predictions with more evidence has higher accuracy
• The Directional Bias Helps Stochastic Gradient Descent to Generalize in Kernel Regression Models
• Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay
• Experiments with Rich Regime Training for Deep Learning
• Convolutional Spectral Kernel Learning
• Out-of-Distribution Generalization in Kernel Regression
• Dynamical transition in controllable quantum neural networks with large depth
• Fast Adaptation with Linearized Neural Networks
• Stochastic Thermodynamics of Learning Parametric Probabilistic Models
• Revisiting minimum description length complexity in overparameterized models
• Deep Networks Provably Classify Data on Curves
• The Renyi Gaussian Process: Towards Improved Generalization
• An Even More Optimal Stochastic Optimization Algorithm: Minibatching and Interpolation Learning
• Understanding Deflation Process in Over-parametrized Tensor Decomposition
• A priori guarantees of finite-time convergence for Deep Neural Networks
• Multi-Level Fine-Tuning: Closing Generalization Gaps in Approximation of Solution Maps under a Limited Budget for Training Data
• Constrained Policy Gradient Method for Safe and Fast Reinforcement Learning: a Neural Tangent Kernel Based Approach
• Neuron Coverage-Guided Domain Generalization
• The Probabilistic Fault Tolerance of Neural Networks in the Continuous Limit
• Learning with Neural Tangent Kernels in Near Input Sparsity Time
• Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations
• Asymptotics of Ridge Regression in Convolutional Models
• One-pass Stochastic Gradient Descent in Overparametrized Two-layer Neural Networks
• Student-Teacher Learning from Clean Inputs to Noisy Inputs
• Comments on Leo Breiman's paper 'Statistical Modeling: The Two Cultures' (Statistical Science, 2001, 16(3), 199-231)
• Understanding Layer-wise Contributions in Deep Neural Networks through Spectral Analysis
• On Symmetry and Initialization for Neural Networks
• One Network Fits All? Modular versus Monolithic Task Formulations in Neural Networks
• Random Features for the Neural Tangent Kernel
• Large-Dimensional Random Matrix Theory and Its Applications in Deep Learning and Wireless Communications
• Gradient Kernel Regression
• Generalization Properties of hyper-RKHS and its Applications
• Towards Unbiased Random Features with Lower Variance For Stationary Indefinite Kernels
• An Exploration into why Output Regularization Mitigates Label Noise
• Evolution of Q Values for Deep Q Learning in Stable Baselines
• Random matrices in service of ML footprint: ternary random features with no performance loss
• New Insights into Graph Convolutional Networks using Neural Tangent Kernels
• Learning the gravitational force law and other analytic functions
• AdjointBackMapV2: Precise Reconstruction of Arbitrary CNN Unit's Activation via Adjoint Operators
• Directional Convergence Analysis under Spherically Symmetric Distribution
• Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks
• Provable Lifelong Learning of Representations
• Training Efficiency and Robustness in Deep Learning
• Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization
• A Functional Perspective on Learning Symmetric Functions with Neural Networks
• Efficient classical computation of the neural tangent kernel of quantum neural networks
• Neural Optimization Kernel: Towards Robust Deep Learning
• A Probabilistic Approach to Neural Network Pruning
• From deep to Shallow: Equivalent Forms of Deep Networks in Reproducing Kernel Krein Space and Indefinite Support Vector Machines
• Transforming physics-informed machine learning to convex optimization
• Provably Strict Generalisation Benefit for Invariance in Kernel Methods
• Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics
• A note on regularised NTK dynamics with an application to PAC-Bayesian training
• Fast convergence to an approximate solution by message-passing for complex optimizations
• A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network
• Not all parameters are born equal: Attention is mostly what you need
• A General Framework for Analyzing Stochastic Dynamics in Learning Algorithms
• Masked Training of Neural Networks with Partial Gradients
• The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutions
• Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks
• Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks
• Revisiting the Effects of Stochasticity for Hamiltonian Samplers
• Statistical Mechanical Analysis of Neural Network Pruning
• Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning
• Quasi-Bayesian Dual Instrumental Variable Regression
• Fundamental tradeoffs between memorization and robustness in random features and neural tangent regimes
• Good Classifiers are Abundant in the Interpolating Regime
• Regularization in ResNet with Stochastic Depth
• Submodular + Concave
• Learning and Generalization in Overparameterized Normalizing Flows
• Catapult Dynamics and Phase Transitions in Quadratic Nets
• Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum
• Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime
• How isotropic kernels perform on simple invariants
• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II
• Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Critic
• Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
• Generalization by design: Shortcuts to Generalization in Deep Learning
• Towards Understanding Generalization via Decomposing Excess Risk Dynamics
• Concurrent Training Improves the Performance of Behavioral Cloning from Observation
• Normalization effects on shallow neural networks and related asymptotic expansions
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Approximate Last Iterate Convergence in Overparameterized GANs
• Enhanced Recurrent Neural Tangent Kernels for Non-Time-Series Data
• M-ar-K-Fast Independent Component Analysis
• Stationary Density Estimation of Itô Diffusions Using Deep Learning
• Neural Contextual Bandits without Regret
• A Revision of Neural Tangent Kernel-based Approaches for Neural Networks
• Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective
• Forward Operator Estimation in Generative Models with Kernel Transfer Operators
• On the Impact of Stable Ranks in Deep Nets
• Disentangling deep neural networks with rectified linear units using duality
• An Empirical Study on Compressed Decentralized Stochastic Gradient Algorithms with Overparameterized Models
• Uniform Generalization Bounds for Overparameterized Neural Networks
• Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
• Analytic expressions for the output evolution of a deep neural network
• On the Regularization of Autoencoders
• Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning
• On the Parameter Combinations That Matter and on Those That do Not
• Towards Understanding Learning in Neural Networks with Linear Teachers
• On the Frequency Bias of Generative Models
• A Neural Tangent Kernel Perspective of Infinite Tree Ensembles
• Over-parametrized neural networks as under-determined linear systems
• DebiNet: Debiasing Linear Models with Nonlinear Overparameterized Neural Networks
• Depth Without the Magic: Inductive Bias of Natural Gradient Descent
• Algorithms and Hardness for Linear Algebra on Geometric Graphs
• Covariate Shift in High-Dimensional Random Feature Regression
• Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space
• WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning
• Deep Interactive Denoiser (DID) for X-Ray Computed Tomography
• Notes on Deep Learning Theory
• SGD Through the Lens of Kolmogorov Complexity
• Going Beyond Linear RL: Sample Efficient Neural Function Approximation
• Recent Advances in Large Margin Learning
• Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
• How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorization