Entropy Law: The Story Behind Data Compression and LLM Performance

Mingjia Yin, Chuhan Wu, Yufei Wang +7

Data is the cornerstone of large language models (LLMs), but not all data is useful for model learning. Carefully selected data can better elicit the capabilities of LLMs with much…