Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning

Xiaonan Luo, Yue Huang, Kehan Guo +4

Model collapse is a central challenge in learning from synthetic data: as later-generation large language models (LLMs) are trained on an increasing proportion of model-generated d…