3 papers
cs.LG2025
Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures
Dang Nguyen, Wenhan Yang, Rathul Anand +2
Training with larger mini-batches improves the convergence rate and can yield superior performance. However, training with large mini-batches becomes prohibitive for Large Language…
cs.LG2025
Challenges and Opportunities in Improving Worst-Group Generalization in Presence of Spurious Features
Siddharth Joshi, Yu Yang, Yihao Xue +2
Deep neural networks often exploit *spurious* features that are present in the majority of examples within a class during training. This leads to *poor worst-group test accuracy*,…
cs.CL2024
SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
Yu Yang, Siddhartha Mishra, Jeffrey N Chiang +1
Despite the effectiveness of data selection for large language models (LLMs) during pretraining and instruction fine-tuning phases, improving data efficiency in supervised fine-tun…