What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use

Yixiao Chen, Ke Cheng, Jiangtao Guan +5

What should data teach a language model at a particular point in training? A circuit view reveals three distinct bottlenecks: forming a computation, making its required content ava…