An Information-Theoretic Criterion for Efficient Data Synthesis

Hanyu Li, Zhengqi Sun, Xiaotie Deng

Synthetic data becomes crucial for large language model training, but its effectiveness is highly inconsistent. We provide an information-theoretic account of this inconsistency: s…