Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMs

Tushaar Gangavarapu, Jiping Li, Christopher Vattheuer +2

Can modifying the training data distribution guide optimizers toward solutions with improved generalization when training large language models (LLMs)? In this work, we theoretical…