Out-of-Distribution Generalization in Kernel Regression

Abstract

In real word applications, data generating process for training a machine learning model often differs from what the model encounters in the test stage. Understanding how and whether machine learning models generalize under such distributional shifts have been a theoretical challenge. Here, we study generalization in kernel regression when the training and test distributions are different using methods from statistical physics. Using the replica method, we derive an analytical formula for the out-of-distribution generalization error applicable to any kernel and real datasets. We identify an overlap matrix that quantifies the mismatch between distributions for a given kernel as a key determinant of generalization performance under distribution shift. Using our analytical expressions we elucidate various generalization phenomena including possible improvement in generalization when there is a mismatch. We develop procedures for optimizing training and test distributions for a given data budget to find best and worst case generalizations under the shift. We present applications of our theory to real and synthetic datasets and for many kernels. We compare results of our theory applied to Neural Tangent Kernel with simulations of wide networks and show agreement. We analyze linear regression in further depth.

arXiv commentEq. (SI.1.59) corrected

References in corpus (21)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A review of domain adaptation without target labels
• Deep Neural Networks as Gaussian Processes
• Spin-Glass Theory for Pedestrians
• Do ImageNet Classifiers Generalize to ImageNet?
• Adversarial Examples Are Not Bugs, They Are Features
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Robustness May Be at Odds with Accuracy
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Exploring the Landscape of Spatial Robustness
• Statistical Mechanics of Support Vector Networks
• An Investigation of Why Overparameterization Exacerbates Spurious Correlations
• Scalable Kernel Methods via Doubly Stochastic Gradients
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• Out of Distribution Generalization in Machine Learning
• Optimal Regularization Can Mitigate Double Descent
• On the Inductive Bias of Neural Tangent Kernels
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• Multiple Descent: Design Your Own Generalization Curve
• Triple descent and the two kinds of overfitting: Where & why do they appear?