On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability

Chenyu Zheng, Wei Huang, Rongzhen Wang +3

Autoregressively trained transformers have brought a profound revolution to the world, especially with their in-context learning (ICL) ability to address downstream tasks. Recently…