Break Step: Recursive Training Resonates with Replayed Sampling Noise

Yangze Liu, Zhongyi Han

How fast does a language model degrade when trained on its own outputs? Theory traces it to gradually accumulating errors, while experiments report repeated phrases within ten gene…