Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models

Mayee F. Chen, Nicholas Roberts, Kush Bhatia +4

The quality of training data impacts the performance of pre-trained large language models (LMs). Given a fixed budget of tokens, we study how to best select data that leads to good…