Understanding deep learning requires rethinking generalization

Abstract

Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family, or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization, and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models.

Published in ICLR 2017

References in corpus (2)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• The Loss Surfaces of Multilayer Networks

Cited by in corpus (485)

• Confounding variables can degrade generalization performance of radiological deep learning models
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• Reconciling modern machine learning practice and the bias-variance trade-off
• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• A trans-disciplinary review of deep learning research for water resources scientists
• Opening the Black Box of Deep Neural Networks via Information
• Deep Image Prior
• Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification
• Don't Decay the Learning Rate, Increase the Batch Size
• Benign Overfitting in Linear Regression
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Generalisation in humans and deep neural networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Deep Neural Networks for Estimation and Inference
• Variational Dropout Sparsifies Deep Neural Networks
• Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation
• Training and Inference with Integers in Deep Neural Networks
• Denoising Implicit Feedback for Recommendation
• Neural Lander: Stable Drone Landing Control using Learned Dynamics
• Shallow Neural Networks for Fluid Flow Reconstruction with Limited Sensors
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Model compression via distillation and quantization
• Var-CNN: A Data-Efficient Website Fingerprinting Attack Based on Deep Learning
• Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering
• Three Factors Influencing Minima in SGD
• Topology optimization of 2D structures with nonlinearities using deep learning
• An empirical study on evaluation metrics of generative adversarial networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Optimizing the Latent Space of Generative Networks
• DarkneTZ: Towards Model Privacy at the Edge using Trusted Execution Environments
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• Unsupervised Domain Adaptation through Self-Supervision
• Learning Sparse Neural Networks through $L_0$ Regularization
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Transferable Clean-Label Poisoning Attacks on Deep Neural Nets
• Optimization for deep learning: theory and algorithms
• Intelligent Inverse Treatment Planning via Deep Reinforcement Learning, a Proof-of-Principle Study in High Dose-rate Brachytherapy for Cervical Cancer
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring
• Rademacher Complexity for Adversarially Robust Generalization
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• LongReMix: Robust Learning with High Confidence Samples in a Noisy Label Environment
• Overfitting Mechanism and Avoidance in Deep Neural Networks
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Kernel Flows: from learning kernels from data into the abyss
• Weak lensing cosmology with convolutional neural networks on noisy data
• The Conditional Entropy Bottleneck
• To understand deep learning we need to understand kernel learning
• Just Interpolate: Kernel "Ridgeless" Regression Can Generalize
• Label-Only Membership Inference Attacks
• Self-Distillation Amplifies Regularization in Hilbert Space
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Shape and Margin-Aware Lung Nodule Classification in Low-dose CT Images via Soft Activation Mapping
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• The no-free-lunch theorems of supervised learning
• Self-Adaptive Training: beyond Empirical Risk Minimization
• On the Origin of Deep Learning
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Neural Models for Information Retrieval
• Deep Learning Methods for Solving Linear Inverse Problems: Research Directions and Paradigms
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• Combating noisy labels by agreement: A joint training method with co-regularization
• Training Data Influence Analysis and Estimation: A Survey
• Overparameterized Neural Networks Implement Associative Memory
• Real-Time Well Log Prediction From Drilling Data Using Deep Learning
• Analysis and Optimization of Convolutional Neural Network Architectures
• Artificial Intelligence in Glioma Imaging: Challenges and Advances
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Regression and Classification for Direction-of-Arrival Estimation with Convolutional Recurrent Neural Networks
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
• Stiffness: A New Perspective on Generalization in Neural Networks
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Reservoir Computing as a Tool for Climate Predictability Studies
• Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification
• Scalable Neural Network Decoders for Higher Dimensional Quantum Codes
• Training behavior of deep neural network in frequency domain
• An Improved Analysis of Training Over-parameterized Deep Neural Networks
• Superposition of many models into one
• A neural network walks into a lab: towards using deep nets as models for human behavior
• Overfitting in adversarially robust deep learning
• Classification vs regression in overparameterized regimes: Does the loss function matter?
• Theory II: Landscape of the Empirical Risk in Deep Learning
• Weakly Supervised Vessel Segmentation in X-ray Angiograms by Self-Paced Learning from Noisy Labels with Suggestive Annotation
• Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey
• An Entropic Optimal Transport Loss for Learning Deep Neural Networks under Label Noise in Remote Sensing Images
• Position, Padding and Predictions: A Deeper Look at Position Information in CNNs
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Learning the Travelling Salesperson Problem Requires Rethinking Generalization
• One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision
• Understanding Batch Normalization
• A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks
• Label Noise Types and Their Effects on Deep Learning
• Topology of Learning in Artificial Neural Networks
• Fixing a Broken ELBO
• Provable limitations of deep learning
• A mean-field limit for certain deep neural networks
• Banach Space Representer Theorems for Neural Networks and Ridge Splines
• Understanding training and generalization in deep learning by Fourier analysis
• On the Robustness of Convolutional Neural Networks to Internal Architecture and Weight Perturbations
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• The loss landscape of overparameterized neural networks
• AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
• Is Deep Learning a Renormalization Group Flow?
• Gradient Centralization: A New Optimization Technique for Deep Neural Networks
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Uncertainty Quantification and Deep Ensembles
• Classification regions of deep neural networks
• On the Decision Boundary of Deep Neural Networks
• Tropical Geometry of Deep Neural Networks
• Bad Global Minima Exist and SGD Can Reach Them
• Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
• Modeling Generalization in Machine Learning: A Methodological and Computational Study
• FSDR: Frequency Space Domain Randomization for Domain Generalization
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• Automated Architecture Design for Deep Neural Networks
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Deep Neural Networks for Marine Debris Detection in Sonar Images
• Learning with Weak Supervision for Email Intent Detection
• Theoretical properties of the global optimizer of two layer neural network
• On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
• Learning with Instance-Dependent Label Noise: A Sample Sieve Approach
• Out-of-Distribution Detection using Multiple Semantic Label Representations
• On the training dynamics of deep networks with $L_2$ regularization
• Rethinking Feature Distribution for Loss Functions in Image Classification
• Porcupine Neural Networks: (Almost) All Local Optima are Global
• Lipschitz regularized Deep Neural Networks generalize and are adversarially robust
• A multiscale neural network based on hierarchical matrices
• Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron
• The Spectral Bias of the Deep Image Prior
• Intriguing Properties of Adversarial Examples
• Negative eigenvalues of the Hessian in deep neural networks
• On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition
• Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks
• General stochastic separation theorems with optimal bounds
• Meta-Learning Deep Energy-Based Memory Models
• A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-${\ell }_1$-Norm Interpolated Classifiers
• GraphCrop: Subgraph Cropping for Graph Classification
• How Does Mixup Help With Robustness and Generalization?
• Constrained Learning with Non-Convex Losses
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization
• MeerCRAB: MeerLICHT Classification of Real and Bogus Transients using Deep Learning
• Wide and Deep Neural Networks Achieve Optimality for Classification
• On Generalization Bounds of a Family of Recurrent Neural Networks
• Stability and Generalization of Learning Algorithms that Converge to Global Optima
• Learning Not to Learn in the Presence of Noisy Labels
• Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process
• Coherent Point Drift Networks: Unsupervised Learning of Non-Rigid Point Set Registration
• NRGNN: Learning a Label Noise-Resistant Graph Neural Network on Sparsely and Noisily Labeled Graphs
• Complexity of Training ReLU Neural Network
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• Deep Complex Networks for Protocol-Agnostic Radio Frequency Device Fingerprinting in the Wild
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• MedAL: Deep Active Learning Sampling Method for Medical Image Analysis
• A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case
• A Corrective View of Neural Networks: Representation, Memorization and Learning
• Deep Reinforcement Learning with Robust and Smooth Policy
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• Dynamic Hard Pruning of Neural Networks at the Edge of the Internet
• Repulsive Deep Ensembles are Bayesian
• Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network
• Segmentation-Aware Image Denoising without Knowing True Segmentation
• How do infinite width bounded norm networks look in function space?
• Adversarially Robust Training through Structured Gradient Regularization
• Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness
• Towards Understanding Theoretical Advantages of Complex-Reaction Networks
• BraggNN: Fast X-ray Bragg Peak Analysis Using Deep Learning
• Implicit competitive regularization in GANs
• Ensemble Robustness and Generalization of Stochastic Deep Learning Algorithms
• Generalization Bounds for Convolutional Neural Networks
• Generalization bounds via distillation
• Information-theoretic generalization bounds for black-box learning algorithms
• Self-Damaging Contrastive Learning
• Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
• Kernel regression in high dimensions: Refined analysis beyond double descent
• Mean-Field Neural ODEs via Relaxed Optimal Control
• Saliency is a Possible Red Herring When Diagnosing Poor Generalization
• A Highly Parallel FPGA Implementation of Sparse Neural Network Training
• Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
• WeMix: How to Better Utilize Data Augmentation
• TDEFSI: Theory Guided Deep Learning Based Epidemic Forecasting with Synthetic Information
• Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
• RIGA: Covert and Robust White-Box Watermarking of Deep Neural Networks
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• Toward Better Generalization Bounds with Locally Elastic Stability
• A Second-Order Approach to Learning with Instance-Dependent Label Noise
• Frequency Principle in Deep Learning with General Loss Functions and Its Potential Application
• Variational Disentanglement for Domain Generalization
• Breast Mass Detection with Faster R-CNN: On the Feasibility of Learning from Noisy Annotations
• Neural Entropic Estimation: A faster path to mutual information estimation
• Stability and Generalization of Graph Convolutional Neural Networks
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression
• Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
• Understanding Adversarial Robustness Through Loss Landscape Geometries
• Deep pNML: Predictive Normalized Maximum Likelihood for Deep Neural Networks
• Positively Scale-Invariant Flatness of ReLU Neural Networks
• Contextual-Relation Consistent Domain Adaptation for Semantic Segmentation
• When Does Preconditioning Help or Hurt Generalization?
• The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
• Deep learning in magnetic resonance prostate segmentation: A review and a new perspective
• SGD Generalizes Better Than GD (And Regularization Doesn't Help)
• Neural tangent kernels, transportation mappings, and universal approximation
• Inferred vs traditional personality assessment: are we predicting the same thing?
• Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
• Disentangling neural mechanisms for perceptual grouping
• Uncertainty Quantification in CNN Through the Bootstrap of Convex Neural Networks
• Invariance of Weight Distributions in Rectified MLPs
• Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings
• Latent Causal Invariant Model
• Searching to Exploit Memorization Effect in Learning from Corrupted Labels
• Quasi-potential as an implicit regularizer for the loss function in the stochastic gradient descent
• Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
• Convex Geometry and Duality of Over-parameterized Neural Networks
• Déjà Vu: an empirical evaluation of the memorization properties of ConvNets
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• A Bayesian Perspective on Training Speed and Model Selection
• On Dropout and Nuclear Norm Regularization
• Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition
• Implicit Regularization of Stochastic Gradient Descent in Natural Language Processing: Observations and Implications
• Generalized Negative Correlation Learning for Deep Ensembling
• Generalization and Memorization: The Bias Potential Model
• Coresets for Robust Training of Neural Networks against Noisy Labels
• Transfer learning extensions for the probabilistic classification vector machine
• Empirical study of extreme overfitting points of neural networks
• Linear Regression with Distributed Learning: A Generalization Error Perspective
• Computationally Efficient Feature Significance and Importance for Machine Learning Models
• On the interplay between noise and curvature and its effect on optimization and generalization
• Implicit Regularization of Random Feature Models
• On Scalable and Efficient Computation of Large Scale Optimal Transport
• Data-Efficient Mutual Information Neural Estimator
• GraphFM: Graph Factorization Machines for Feature Interaction Modeling
• Linear Convergence of Adaptive Stochastic Gradient Descent
• Implicit Rugosity Regularization via Data Augmentation
• Data Separability for Neural Network Classifiers and the Development of a Separability Index
• Implicit Regularization of Accelerated Methods in Hilbert Spaces
• Term Revealing: Furthering Quantization at Run Time on Quantized DNNs
• Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
• Privacy-preserving Federated Bayesian Learning of a Generative Model for Imbalanced Classification of Clinical Data
• A Closer Look at Data Bias in Neural Extractive Summarization Models
• Mehler's Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
• Deep Learning in Information Security
• On the Role of Dataset Quality and Heterogeneity in Model Confidence
• Statistical Optimality of Interpolated Nearest Neighbor Algorithms
• Universal Supervised Learning for Individual Data
• Artificial Intelligence in Healthcare: Lost In Translation?
• Improving Generalization by Controlling Label-Noise Information in Neural Network Weights
• To Each Optimizer a Norm, To Each Norm its Generalization
• The Role of Information Complexity and Randomization in Representation Learning
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study
• GradSign: Model Performance Inference with Theoretical Insights
• Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers
• Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks
• Adaptive Sample Selection for Robust Learning under Label Noise
• Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
• Picasso: A Modular Framework for Visualizing the Learning Process of Neural Network Image Classifiers
• Non-attracting Regions of Local Minima in Deep and Wide Neural Networks
• Small Towers Make Big Differences
• Noisy Recurrent Neural Networks
• On the Effectiveness of Regularization Against Membership Inference Attacks
• Implicit Regularization in ReLU Networks with the Square Loss
• Scattering Transform Based Image Clustering using Projection onto Orthogonal Complement
• Anti-Distillation: Improving reproducibility of deep networks
• Benign Overfitting and Noisy Features
• Mini-batch Tempered MCMC
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets
• Gabor frames and deep scattering networks in audio processing
• What Context Features Can Transformer Language Models Use?
• Engineering problems in machine learning systems
• Deep Neural Networks for Choice Analysis: Extracting Complete Economic Information for Interpretation
• Genetic algorithms with DNN-based trainable crossover as an example of partial specialization of general search
• Multi-Class Classification from Noisy-Similarity-Labeled Data
• Out-of-Sample Extrapolation with Neuron Editing
• Robust and On-the-fly Dataset Denoising for Image Classification
• Channel Equilibrium Networks for Learning Deep Representation
• Towards Understanding the Generalization Bias of Two Layer Convolutional Linear Classifiers with Gradient Descent
• Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network
• Implicit Regularization and Convergence for Weight Normalization
• Data-Uncertainty Guided Multi-Phase Learning for Semi-Supervised Object Detection
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Label-Noise Robust Multi-Domain Image-to-Image Translation
• Mixture Proportion Estimation and PU Learning: A Modern Approach
• Neural Language Priors
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• On the Double Descent of Random Features Models Trained with SGD
• Privacy Inference Attacks and Defenses in Cloud-based Deep Neural Network: A Survey
• Stochastic gradient descent and fast relaxation to thermodynamic equilibrium: a stochastic control approach
• Statistically Significant Stopping of Neural Network Training
• On the Universality of the Double Descent Peak in Ridgeless Regression
• Teaching with Commentaries
• Deep Learning from Small Amount of Medical Data with Noisy Labels: A Meta-Learning Approach
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach
• A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained Model Lead to the Promised Land?
• A comprehensive study on the prediction reliability of graph neural networks for virtual screening
• On Calibration of Mixup Training for Deep Neural Networks
• DeepNNK: Explaining deep models and their generalization using polytope interpolation
• Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?
• Neural Networks Trained on Natural Scenes Exhibit Gestalt Closure
• A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels
• Model Extraction Attacks against Recurrent Neural Networks
• Deep learning methods based on cross-section images for predicting effective thermal conductivity of composites
• Distributed Optimization for Over-Parameterized Learning
• Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
• Intelligence plays dice: Stochasticity is essential for machine learning
• Co-matching: Combating Noisy Labels by Augmentation Anchoring
• Muddling Label Regularization: Deep Learning for Tabular Datasets
• AFD-Net: Adaptive Fully-Dual Network for Few-Shot Object Detection
• Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond
• Data Consistent CT Reconstruction from Insufficient Data with Learned Prior Images
• KNN-enhanced Deep Learning Against Noisy Labels
• Disrupting Model Training with Adversarial Shortcuts
• Image Captioning with Sparse Recurrent Neural Network
• [Extended version] Rethinking Deep Neural Network Ownership Verification: Embedding Passports to Defeat Ambiguity Attacks
• Rethinking the Artificial Neural Networks: A Mesh of Subnets with a Central Mechanism for Storing and Predicting the Data
• Noisy Annotation Refinement for Object Detection
• SASL: Saliency-Adaptive Sparsity Learning for Neural Network Acceleration
• Self-Regularity of Non-Negative Output Weights for Overparameterized Two-Layer Neural Networks
• Defending Against Image Corruptions Through Adversarial Augmentations
• Automatic Dataset Annotation to Learn CNN Pore Description for Fingerprint Recognition
• Global Convergence of Frank Wolfe on One Hidden Layer Networks
• Learning Neural Network Classifiers with Low Model Complexity
• A Simple yet Effective Baseline for Robust Deep Learning with Noisy Labels
• The Rate of Convergence of Variation-Constrained Deep Neural Networks
• A Survey of Techniques All Classifiers Can Learn from Deep Networks: Models, Optimizations, and Regularization
• What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data
• Gradient descent follows the regularization path for general losses
• Why flatness does and does not correlate with generalization for deep neural networks
• Probably Approximately Correct Constrained Learning
• From Data Quality to Model Quality: an Exploratory Study on Deep Learning
• Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks
• Biologically inspired architectures for sample-efficient deep reinforcement learning
• Generator Versus Segmentor: Pseudo-healthy Synthesis
• Deep Learning: Generalization Requires Deep Compositional Feature Space Design
• Network Cooperation with Progressive Disambiguation for Partial Label Learning
• Drill the Cork of Information Bottleneck by Inputting the Most Important Data
• Collegial Ensembles
• Semi-Supervised Class Discovery
• GM-Net: Learning Features with More Efficiency
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation
• SaaS: Speed as a Supervisor for Semi-supervised Learning
• A New Angle on L2 Regularization
• What training reveals about neural network complexity
• Functional Regularization for Representation Learning: A Unified Theoretical Perspective
• How Much Restricted Isometry is Needed In Nonconvex Matrix Recovery?
• The Geometry of Deep Networks: Power Diagram Subdivision
• Characterizing the Generalization Error of Gibbs Algorithm with Symmetrized KL information
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Weisfeiler-Lehman Embedding for Molecular Graph Neural Networks
• Improving Medical Annotation Quality to Decrease Labeling Burden Using Stratified Noisy Cross-Validation
• Neural Architecture Search with Random Labels
• A Conceptual Framework for Lifelong Learning
• Investigating the Compositional Structure Of Deep Neural Networks
• Towards a Measure of Individual Fairness for Deep Learning
• Training Sparse Neural Networks using Compressed Sensing
• Sequence learning in Associative Neuronal-Astrocytic Network
• Limited Gradient Descent: Learning With Noisy Labels
• Robust Representation Learning via Perceptual Similarity Metrics
• When and how epochwise double descent happens
• Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?
• Learning to Recognize Actionable Static Code Warnings (is Intrinsically Easy)
• Deep Neural Networks for Choice Analysis: Architectural Design with Alternative-Specific Utility Functions
• Faster Convergence & Generalization in DNNs
• CoDiM: Learning with Noisy Labels via Contrastive Semi-Supervised Learning
• HERO: Hessian-Enhanced Robust Optimization for Unifying and Improving Generalization and Quantization Performance
• Learning without gradient descent encoded by the dynamics of a neurobiological model
• Topological Data Analysis of Decision Boundaries with Application to Model Selection
• Unified Regularity Measures for Sample-wise Learning and Generalization
• Minimal Sufficient Views: A DNN model making predictions with more evidence has higher accuracy
• Using Wavelets to Analyze Similarities in Image-Classification Datasets
• Gentle Local Robustness implies Generalization
• Phases of learning dynamics in artificial neural networks: with or without mislabeled data
• Critical Percolation as a Framework to Analyze the Training of Deep Networks
• Sample Prior Guided Robust Model Learning to Suppress Noisy Labels
• The role of a layer in deep neural networks: a Gaussian Process perspective
• Improved visible to IR image transformation using synthetic data augmentation with cycle-consistent adversarial networks
• Training Classifiers that are Universally Robust to All Label Noise Levels
• An Essay on Optimization Mystery of Deep Learning
• Deep Within-Class Covariance Analysis for Robust Audio Representation Learning
• An Even More Optimal Stochastic Optimization Algorithm: Minibatching and Interpolation Learning
• A Tale Of Two Long Tails
• Learning Interclass Relations for Image Classification
• Towards the Memorization Effect of Neural Networks in Adversarial Training
• The Role of Regularization in Shaping Weight and Node Pruning Dependency and Dynamics
• Removing Undesirable Feature Contributions Using Out-of-Distribution Data
• Sign-RIP: A Robust Restricted Isometry Property for Low-rank Matrix Recovery
• High-Dimensional Optimization in Adaptive Random Subspaces
• Noise-robust classification with hypergraph neural network
• Do CNNs Encode Data Augmentations?
• Insights into Ordinal Embedding Algorithms: A Systematic Evaluation
• Learning and Exploiting Interclass Visual Correlations for Medical Image Classification
• The distance between the weights of the neural network is meaningful
• The role of invariance in spectral complexity-based generalization bounds
• A priori generalization error for two-layer ReLU neural network through minimum norm solution
• Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space
• Deep Networks as Logical Circuits: Generalization and Interpretation
• Deep Likelihood Network for Image Restoration with Multiple Degradation Levels
• InfoNEAT: Information Theory-based NeuroEvolution of Augmenting Topologies for Side-channel Analysis
• Who's Afraid of Adversarial Transferability?
• GRAFFL: Gradient-free Federated Learning of a Bayesian Generative Model
• An Extension of Fano's Inequality for Characterizing Model Susceptibility to Membership Inference Attacks
• Generalisation in fully-connected neural networks for time series forecasting
• PanDA: Panoptic Data Augmentation
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• BPGrad: Towards Global Optimality in Deep Learning via Branch and Pruning
• Predictive Model Degrees of Freedom in Linear Regression
• Nonlinear Collaborative Scheme for Deep Neural Networks
• Instance based Generalization in Reinforcement Learning
• Variational Bayesian Dropout with a Hierarchical Prior
• On the Existence of Logarithmic Terms in the Drag Coefficient and Nusselt Number of a Single Sphere at High Reynolds Numbers
• Memory and attention in deep learning
• On the Physical Interpretation of Proper Orthogonal Decomposition and Dynamic Mode Decomposition for Liquid Injection
• Multi-loss ensemble deep learning for chest X-ray classification
• Theoretical Guarantees for Model Auditing with Finite Adversaries
• Self-semi-supervised Learning to Learn from NoisyLabeled Data
• MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
• Arithmetic addition of two integers by deep image classification networks: experiments to quantify their autonomous reasoning ability
• Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets
• DP-CGAN: Differentially Private Synthetic Data and Label Generation
• Efficient decorrelation of features using Gramian in Reinforcement Learning
• CNN with large memory layers
• Unsupervised Difficulty Estimation with Action Scores
• Benign overfitting without concentration
• Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?
• Few-shot time series segmentation using prototype-defined infinite hidden Markov models
• Improved Complexities for Stochastic Conditional Gradient Methods under Interpolation-like Conditions
• Efficient Data-Dependent Learnability
• Doubly Robust Off-Policy Learning on Low-Dimensional Manifolds by Deep Neural Networks
• Computer activity learning from system call time series
• Large-Dimensional Random Matrix Theory and Its Applications in Deep Learning and Wireless Communications
• A Theoretical Analysis of Learning with Noisily Labeled Data
• HALO: Learning to Prune Neural Networks with Shrinkage
• Hard Class Rectification for Domain Adaptation
• Adversarial examples are useful too!
• Inherent Noise in Gradient Based Methods
• Adaptive Weight Decay for Deep Neural Networks
• Examining the causal structures of deep neural networks using information theory
• Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks
• Ultrasound Scatterer Density Classification Using Convolutional Neural Networks by Exploiting Patch Statistics
• Using Human Psychophysics to Evaluate Generalization in Scene Text Recognition Models
• Probabilistic Trust Intervals for Out of Distribution Detection
• Neuron Coverage-Guided Domain Generalization
• Efficient Modelling Across Time of Human Actions and Interactions
• Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective
• Procrustean Training for Imbalanced Deep Learning
• Leave-one-out Unfairness
• Adversarial Training with Stochastic Weight Average
• The Relativity of Induction
• Schematic Memory Persistence and Transience for Efficient and Robust Continual Learning
• Joint Text and Label Generation for Spoken Language Understanding
• Efficient Classification of Very Large Images with Tiny Objects
• Towards Understanding Generalization via Decomposing Excess Risk Dynamics
• Feature Selection on Thermal-stress Dataset
• Detecting Deep Neural Network Defects with Data Flow Analysis
• Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression
• Identifying and Exploiting Structures for Reliable Deep Learning
• Towards Quantifying Intrinsic Generalization of Deep ReLU Networks
• Reappraising Domain Generalization in Neural Networks
• An empirical study of the relation between network architecture and complexity
• A Generalization Theory based on Independent and Task-Identically Distributed Assumption
• DNNs as Layers of Cooperating Classifiers
• Expressive power of outer product manifolds on feed-forward neural networks
• Generating GPU Compiler Heuristics using Reinforcement Learning
• Mitigating Memorization in Sample Selection for Learning with Noisy Labels
• Think Global, Act Local: Relating DNN generalisation and node-level SNR
• Predicting the success of Gradient Descent for a particular Dataset-Architecture-Initialization (DAI)
• On the generalization of bayesian deep nets for multi-class classification
• On the potential for open-endedness in neural networks
• How BPE Affects Memorization in Transformers
• Effect of Various Regularizers on Model Complexities of Neural Networks in Presence of Input Noise
• Deep Optimization for Spectrum Repacking
• Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks
• Understanding the Behaviour of the Empirical Cross-Entropy Beyond the Training Distribution
• Go Small and Similar: A Simple Output Decay Brings Better Performance
• Notes on Deep Learning Theory
• Computational principles of intelligence: learning and reasoning with neural networks
• Generalization by design: Shortcuts to Generalization in Deep Learning
• The Variational InfoMax Learning Objective