From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts

Daniel Christoph, Max Ploner, Patrick Haller +1

Sample efficiency is a crucial property of language models with practical implications for training efficiency. In real-world text, information follows a long-tailed distribution.…