Reconciling modern machine learning practice and the bias-variance trade-off

Abstract

Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine learning practice. The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners. In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This "double descent" curve subsumes the textbook U-shaped bias-variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.

References in corpus (10)

• Understanding deep learning requires rethinking generalization
• An Analysis of Deep Neural Network Models for Practical Applications
• Generalization Properties of Learning with Random Features
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• To understand deep learning we need to understand kernel learning
• Does data interpolation contradict statistical optimality?
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers
• Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations

Cited by in corpus (347)

• Reproducible scaling laws for contrastive language-image learning
• Big-Data Science in Porous Materials: Materials Genomics and Machine Learning
• Predicting materials properties without crystal structure: Deep representation learning from stoichiometry
• Prevalence of Neural Collapse during the terminal phase of deep learning training
• Shallow Neural Networks for Fluid Flow Reconstruction with Limited Sensors
• Traffic Prediction using Artificial Intelligence: Review of Recent Advances and Emerging Opportunities
• The data synergy effects of time-series deep learning models in hydrology
• Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• A Field Guide to Federated Optimization
• Scaling description of generalization with number of parameters in deep learning
• Generalization in Quantum Machine Learning: a Quantum Information Perspective
• The Random Feature Model for Input-Output Maps between Banach Spaces
• The Modern Mathematics of Deep Learning
• Two models of double descent for weak features
• Reliable extrapolation of deep neural operators informed by physics or sparse observations
• The transformer earthquake alerting model: A new versatile approach to earthquake early warning
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Going in circles is the way forward: the role of recurrence in visual inference
• Data augmentation instead of explicit regularization
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• Benign overfitting in ridge regression
• An Investigation of Why Overparameterization Exacerbates Spurious Correlations
• Generalisation error in learning with random features and the hidden manifold model
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
• On the Existence of Simpler Machine Learning Models
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• From Learning to Meta-Learning: Reduced Training Overhead and Complexity for Communication Systems
• A newcomer's guide to deep learning for inverse design in nano-photonics
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Overparameterized Neural Networks Implement Associative Memory
• Do We Need Zero Training Loss After Achieving Zero Training Error?
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• Generalization despite overfitting in quantum machine learning models
• Understanding and Improving Early Stopping for Learning with Noisy Labels
• Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• What shapes feature representations? Exploring datasets, architectures, and training
• Machine Learning from a Continuous Viewpoint
• Ten Quick Tips for Deep Learning in Biology
• On Interpretability of Artificial Neural Networks: A Survey
• Accelerating SGD with momentum for over-parameterized learning
• Overfitting in adversarially robust deep learning
• Classification vs regression in overparameterized regimes: Does the loss function matter?
• Combining physics-based and data-driven models: advancing the frontiers of research with Scientific Machine Learning
• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
• Deep learning of quantum entanglement from incomplete measurements
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Implicit Regularization and Momentum Algorithms in Nonlinearly Parameterized Adaptive Control and Prediction
• On the influence of over-parameterization in manifold based surrogates and deep neural operators
• Finite Versus Infinite Neural Networks: an Empirical Study
• Experimental noise in small-angle scattering can be assessed using the Bayesian indirect Fourier transformation
• Learning nonequilibrium control forces to characterize dynamical phase transitions
• Parsimonious Physics-Informed Random Projection Neural Networks for Initial-Value Problems of ODEs and index-1 DAEs
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• A framework for probabilistic weather forecast post-processing across models and lead times using machine learning
• Self-explaining AI as an alternative to interpretable AI
• Neuromorphic Overparameterisation and Few-Shot Learning in Multilayer Physical Neural Networks
• Is deep learning necessary for simple classification tasks?
• Manifold embedding data-driven mechanics
• The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Neural Network Training Using ${\ell }_1$-Regularization and Bi-fidelity Data
• Efficient Quantum State Sample Tomography with Basis-dependent Neural-networks
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Learning through atypical "phase transitions" in overparameterized neural networks
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• Neural Symbolic Regression that Scales
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Pixel super-resolved virtual staining of label-free tissue using diffusion models
• Deep Confident Steps to New Pockets: Strategies for Docking Generalization
• Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data
• Contrasting random and learned features in deep Bayesian linear regression
• Effects of noise on the overparametrization of quantum neural networks
• Quantum neural networks force fields generation
• The Quantum Path Kernel: a Generalized Quantum Neural Tangent Kernel for Deep Quantum Machine Learning
• Probing transfer learning with a model of synthetic correlated datasets
• Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime
• Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks
• A Precise Performance Analysis of Learning with Random Features
• Numerical Calabi-Yau metrics from holomorphic networks
• Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization
• Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
• Wide and Deep Neural Networks Achieve Optimality for Classification
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• Asymptotics of Ridge (less) Regression under General Source Condition
• Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification
• Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success
• Assessing Generalization of SGD via Disagreement
• Understanding Deep Learning via Decision Boundary
• Symbolic Regression via Neural Networks
• Data-Efficient Brain Connectome Analysis via Multi-Task Meta-Learning
• Operator Learning Using Random Features: A Tool for Scientific Computing
• A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent
• Activation function dependence of the storage capacity of treelike neural networks
• Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentration
• What causes the test error? Going beyond bias-variance via ANOVA
• Asymptotic errors for convex penalized linear regression beyond Gaussian matrices
• On Power Laws in Deep Ensembles
• Deep multi-task mining Calabi-Yau four-folds
• A case for new neural network smoothness constraints
• Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance
• Deep neural networks for choice analysis: Enhancing behavioral regularity with gradient regularization
• Failures of model-dependent generalization bounds for least-norm interpolation
• The Implicit Regularization of Ordinary Least Squares Ensembles
• Teacher-student learning for a binary perceptron with quantum fluctuations
• Random matrix analysis of deep neural network weight matrices
• Statistical and Algorithmic Insights for Semi-supervised Learning with Self-training
• Pre-Trained Models: Past, Present and Future
• On the Inherent Regularization Effects of Noise Injection During Training
• Asymptotics of Wide Convolutional Neural Networks
• Mean-Field Neural ODEs via Relaxed Optimal Control
• Exploring Memorization in Adversarial Training
• Deep neural networks have an inbuilt Occam's razor
• Kernel regression in high dimensions: Refined analysis beyond double descent
• Radio source-component association for the LOFAR Two-metre Sky Survey with region-based convolutional neural networks
• ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representations
• Accelerating superconductor discovery through tempered deep learning of the electron-phonon spectral function
• Emergence of Concepts in DNNs?
• Distributional Generalization: A New Kind of Generalization
• Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation
• Double descent in the condition number
• HARFE: Hard-Ridge Random Feature Expansion
• What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure
• Generalization bounds for deep learning
• Directional Pruning of Deep Neural Networks
• Convergence Guarantees for Gaussian Process Means With Misspecified Likelihoods and Smoothness
• Hold me tight! Influence of discriminative features on deep network boundaries
• Estimation under Model Misspecification with Fake Features
• Fitting very flexible models: Linear regression with large numbers of parameters
• The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
• Loss landscapes and optimization in over-parameterized non-linear systems and neural networks
• The Low-Rank Simplicity Bias in Deep Networks
• Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• Eight challenges in developing theory of intelligence
• Overparameterized Linear Regression under Adversarial Attacks
• Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness
• Random features and polynomial rules
• Asymptotic theory of in-context learning by linear attention
• Bias-Reduced Neural Networks for Parameter Estimation in Quantitative MRI
• Coding schemes in neural networks learning classification tasks
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• Towards an Understanding of Benign Overfitting in Neural Networks
• Provable Memorization via Deep Neural Networks using Sub-linear Parameters
• Robustness to Pruning Predicts Generalization in Deep Neural Networks
• Provable Super-Convergence with a Large Cyclical Learning Rate
• Dimensionality reduction, regularization, and generalization in overparameterized regressions
• End-to-end Kernel Learning via Generative Random Fourier Features
• Linear Regression with Distributed Learning: A Generalization Error Perspective
• Scaling Down Deep Learning with MNIST-1D
• Improved guarantees and a multiple-descent curve for Column Subset Selection and the Nyström method
• Optimization's Neglected Normative Commitments
• A Falsificationist Account of Artificial Neural Networks
• Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees
• Tractability from overparametrization: The example of the negative perceptron
• Taxonomizing local versus global structure in neural network loss landscapes
• Disentangling Adaptive Gradient Methods from Learning Rates
• How Powerful are Shallow Neural Networks with Bandlimited Random Weights?
• Bidirectional deep learning of polarization transfer in liquid crystals with application to quantum state preparation
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• Learning Algebraic Models of Quantum Entanglement
• For self-supervised learning, Rationality implies generalization, provably
• Regularizing Neural Networks via Adversarial Model Perturbation
• Powerpropagation: A sparsity inducing weight reparameterisation
• Learning Curve Theory
• Neural Networks and Polynomial Regression. Demystifying the Overparametrization Phenomena
• Early Stopping in Deep Networks: Double Descent and How to Eliminate it
• Interpolating Predictors in High-Dimensional Factor Regression
• Learning PAC-Bayes Priors for Probabilistic Neural Networks
• Large scale analysis of generalization error in learning using margin based classification methods
• Conditioning of Random Feature Matrices: Double Descent and Generalization Error
• The role of regularization in classification of high-dimensional noisy Gaussian mixture
• Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation
• The Recurrent Neural Tangent Kernel
• Deep learning inference with the Event Horizon Telescope II. The Zingularity framework for Bayesian artificial neural networks
• Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware
• How more data can hurt: Instability and regularization in next-generation reservoir computing
• The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
• Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning
• Benign Overfitting and Noisy Features
• Implicit Regularization via Neural Feature Alignment
• Compressing Deep Neural Networks via Layer Fusion
• Relative stability toward diffeomorphisms indicates performance in deep nets
• Small Towers Make Big Differences
• When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
• Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing
• Train simultaneously, generalize better: Stability of gradient-based minimax learners
• The Slow Deterioration of the Generalization Error of the Random Feature Model
• Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networks
• Convolutional Neural Networks Do Work with Pre-Defined Filters
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Statistical Mechanics of Support Vector Regression
• Simple Hamiltonian dynamics is a powerful quantum processing resource
• Deep Learning-Based Position Detection for Hydraulic Cylinders Using Scattering Parameters
• Noether: The More Things Change, the More Stay the Same
• Interpolation and Learning with Scale Dependent Kernels
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• On generalization bounds for deep networks based on loss surface implicit regularization
• Exploring Weight Importance and Hessian Bias in Model Pruning
• Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale
• Data Quality Matters For Adversarial Training: An Empirical Study
• For interpolating kernel machines, minimizing the norm of the ERM solution minimizes stability
• Data-driven effective model shows a liquid-like deep learning
• Improved Learning Rates for Stochastic Optimization
• Fast and Smooth Interpolation on Wasserstein Space
• What Constitutes a Less Discriminatory Algorithm?
• Diet deep generative audio models with structured lottery
• Implicit Gradient Regularization
• Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines
• Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models
• Recovery and Generalization in Over-Realized Dictionary Learning
• On the Universality of the Double Descent Peak in Ridgeless Regression
• When does gradient descent with logistic loss find interpolating two-layer networks?
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• On the Double Descent of Random Features Models Trained with SGD
• SCORE: Approximating Curvature Information under Self-Concordant Regularization
• Self-Regularity of Non-Negative Output Weights for Overparameterized Two-Layer Neural Networks
• Learning from learning machines: a new generation of AI technology to meet the needs of science
• Data augmentation and image understanding
• On the connection between least squares, regularization, and classical shadows
• On the Role of Optimization in Double Descent: A Least Squares Study
• Memory capacity of two layer neural networks with smooth activations
• Minimum ${\ell }_1$-norm interpolators: Precise asymptotics and multiple descent
• Avoiding The Double Descent Phenomenon of Random Feature Models Using Hybrid Regularization
• Kernel interpolation generalizes poorly
• On the Optimal Memorization Power of ReLU Neural Networks
• A Precise Performance Analysis of Support Vector Regression
• Global Convergence of Frank Wolfe on One Hidden Layer Networks
• Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection
• Learning Beyond Experience: Generalizing to Unseen State Space with Reservoir Computing
• Rician likelihood loss for quantitative MRI using self-supervised deep learning
• Can we avoid Double Descent in Deep Neural Networks?
• Optimization Variance: Exploring Generalization Properties of DNNs
• Simmering: Sufficient is better than optimal for training neural networks
• Under manipulations, are some AI models harder to audit?
• Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation
• Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data
• Model selection in atomistic simulation
• Implicit Sparse Regularization: The Impact of Depth and Early Stopping
• Deep Learning Classification of Lake Zooplankton
• Tight bounds for minimum l1-norm interpolation of noisy data
• When and how epochwise double descent happens
• On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications
• The loss landscape of deep linear neural networks: a second-order analysis
• Double Descent and Other Interpolation Phenomena in GANs
• Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions
• The Geometry of Over-parameterized Regression and Adversarial Perturbations
• Embracing the Disharmony in Medical Imaging: A Simple and Effective Framework for Domain Adaptation
• Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model
• The Performance Analysis of Generalized Margin Maximizer (GMM) on Separable Data
• Unified Gradient Reweighting for Model Biasing with Applications to Source Separation
• Are wider nets better given the same number of parameters?
• A Multi-resolution Theory for Approximating Infinite-$p$-Zero-$n$: Transitional Inference, Individualized Predictions, and a World Without Bias-Variance Trade-off
• Adversarial Robustness through Bias Variance Decomposition: A New Perspective for Federated Learning
• Applying Deutsch's concept of good explanations to artificial intelligence and neuroscience -- an initial exploration
• A finite sample analysis of the benign overfitting phenomenon for ridge function estimation
• Pre-interpolation loss behaviour in neural networks
• Subspace Fitting Meets Regression: The Effects of Supervision and Orthonormality Constraints on Double Descent of Generalization Errors
• Generalization Error of Generalized Linear Models in High Dimensions
• The Generalization Error of the Minimum-norm Solutions for Over-parameterized Neural Networks
• Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online
• Making Coherence Out of Nothing At All: Measuring the Evolution of Gradient Alignment
• On the Bias-Variance Tradeoff: Textbooks Need an Update
• Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens
• Is the Number of Trainable Parameters All That Actually Matters?
• Generalization error of minimum weighted norm and kernel interpolation
• Reasoning-Modulated Representations
• SVEMnet: An R package for Self-Validated Elastic-Net Ensembles and Multi-Response Optimization in Small-Sample Mixture-Process Experiments
• Prediction in latent factor regression: Adaptive PCR and beyond
• Foolish Crowds Support Benign Overfitting
• Revisiting minimum description length complexity in overparameterized models
• Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets
• Double Descent Risk and Volume Saturation Effects: A Geometric Perspective
• Phases of learning dynamics in artificial neural networks: with or without mislabeled data
• Implicit Bias of Linear RNNs
• A Brief Prehistory of Double Descent
• Gaussian Process Boosting
• To Boost or not to Boost: On the Limits of Boosted Neural Networks
• The Foes of Neural Network's Data Efficiency Among Unnecessary Input Dimensions
• Dimension Independent Generalization Error by Stochastic Gradient Descent
• Structured Directional Pruning via Perturbation Orthogonal Projection
• Lower bounds for the trade-off between bias and mean absolute deviation
• Fast Approximation and Estimation Bounds of Kernel Quadrature for Infinitely Wide Models
• Asymptotic Risk of Overparameterized Likelihood Models: Double Descent Theory for Deep Neural Networks
• Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows
• The Double Descent Behavior in Two Layer Neural Network for Binary Classification
• Predictive variational autoencoder for learning robust representations of time-series data
• Deep limits and cut-off phenomena for neural networks
• The High-Dimensional Asymptotics of Principal Component Regression
• Next-token prediction capacity: general upper bounds and a lower bound for transformers
• An Even More Optimal Stochastic Optimization Algorithm: Minibatching and Interpolation Learning
• Rethinking generalization of classifiers in separable classes scenarios and over-parameterized regimes
• The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
• Double descent in quantum kernel methods
• On the Relationship Between Double Descent of CNNs and Shape/Texture Bias Under Learning Process
• Instance based Generalization in Reinforcement Learning
• Programming with Neural Surrogates of Programs
• Effective Regularization Through Loss-Function Metalearning
• There is no Double-Descent in Random Forests
• Explaining generalization in deep learning: progress and fundamental limits
• Learning the hypotheses space from data through a U-curve algorithm
• Out-of-Distribution Generalization in Kernel Regression
• Asymptotics of Ridge Regression in Convolutional Models
• Mixed Semi-Supervised Generalized-Linear-Regression with Applications to Deep-Learning and Interpolators
• Tradeoff of generalization error in unsupervised learning
• Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective
• Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space
• Enhancing Generalization in Convolutional Neural Networks through Regularization with Edge and Line Features
• Minimum discrepancy principle strategy for choosing $k$ in $k$-NN regression
• Comments on Leo Breiman's paper 'Statistical Modeling: The Two Cultures' (Statistical Science, 2001, 16(3), 199-231)
• The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks
• A Neural Tangent Kernel Perspective of Infinite Tree Ensembles
• Dropout Drops Double Descent
• A deep neural network for multi-species fish detection using multiple acoustic cameras
• Role of overparametrization in quantum approximate optimization
• Behavioral Experiments for Understanding Catastrophic Forgetting
• Large-Dimensional Random Matrix Theory and Its Applications in Deep Learning and Wireless Communications
• Harmless interpolation in regression and classification with structured features
• On the Existence of Universal Lottery Tickets
• A unifying approach on bias and variance analysis for classification
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
• Plant 'n' Seek: Can You Find the Winning Ticket?
• Good Classifiers are Abundant in the Interpolating Regime
• Learning stochastic filtering
• Covariate Shift in High-Dimensional Random Feature Regression
• Data-driven Seasonal Climate Predictions via Variational Inference and Transformers
• Network Dynamics-Based Framework for Understanding Deep Neural Networks
• Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry
• Is interpolation benign for random forest regression?
• Relative Flatness and Generalization
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research
• Double Descent Optimization Pattern and Aliasing: Caveats of Noisy Labels
• An Overview of Low-Rank Structures in the Training and Adaptation of Large Models
• WANify: Gauging and Balancing Runtime WAN Bandwidth for Geo-distributed Data Analytics
• Can all variations within the unified mask-based beamformer framework achieve identical peak extraction performance?
• Bias-variance decomposition of overparameterized regression with random linear features
• Jitter: Random Jittering Loss Function
• Fundamental tradeoffs between memorization and robustness in random features and neural tangent regimes
• Distilling Double Descent
• Understanding Interpretability by generalized distillation in Supervised Classification
• Compression Implies Generalization
• A Mechanism for Producing Aligned Latent Spaces with Autoencoders
• Bayesian Deep Learning Hyperparameter Search for Robust Function Mapping to Polynomials with Noise