Theory on Attention Dynamics for Out-of-Distribution In-Context Learning

Junze Deng, Daouda Sow, Sen Lin +1

Transformers have demonstrated remarkable in-context learning (ICL) capabilities, enabling them to perform new tasks without additional fine-tuning. However, their performance ofte…