Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining

Yuto Nishida, Hirokazu Kiyomaru, Yusuke Oda +6

Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a m…