Measuring Sample Importance in Data Pruning for Language Models based on Information Entropy

Minsang Kim, Seungjun Baek

Compute-efficient training of language models has become an important issue. We consider data pruning for data-efficient training of LLMs. In this work, we consider a data pruning…