Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens

Ruifeng Ren, Yong Liu

Pre-trained large language models based on Transformers have demonstrated remarkable in-context learning (ICL) abilities. With just a few demonstration examples, the models can imp…