5 papers
Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMs
Tushaar Gangavarapu, Jiping Li, Christopher Vattheuer +2
Can modifying the training data distribution guide optimizers toward solutions with improved generalization when training large language models (LLMs)? In this work, we theoretical…
Risk Phase Transitions in Spiked Regression: Alignment Driven Benign and Catastrophic Overfitting
Jiping Li, Rishi Sonthalia
This paper analyzes the generalization error of minimum-norm interpolating solutions in linear regression using spiked covariance data models. The paper characterizes how varying s…
Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Models
Dang Nguyen, Jiping Li, Jinghao Zheng +1
Synthetically augmenting training datasets with diffusion models has become an effective strategy for improving the generalization of image classifiers. However, existing approache…
Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
Yihao Xue, Jiping Li, Baharan Mirzasoleiman
Weak-to-Strong Generalization (W2SG), where a weak model supervises a stronger one, serves as an important analogy for understanding how humans might guide superhuman intelligence…
Generalization for Least Squares Regression With Simple Spiked Covariances
Jiping Li, Rishi Sonthalia
Random matrix theory has proven to be a valuable tool in analyzing the generalization of linear models. However, the generalization properties of even two-layer neural networks tra…