Forking: Sudden Overfitting Under Replay

Shanbin Yu, Shaoyang Guo, Haoran Zhao +2

This paper studies forking, a generalization failure discovered in NanoGPT autoresearch. Under data replay, models with an over-encoding n-gram memory branch show a sharp separatio…