On Representation Redundancy in Large-Scale Instruction Tuning Data Selection

Youwei Shu, Shaomian Zheng, Dingnan Jin +5

Data quality is a crucial factor in large language models training. While prior work has shown that models trained on smaller, high-quality datasets can outperform those trained on…