How do Transformers perform In-Context Autoregressive Learning?

Michael E. Sander, Raja Giryes, Taiji Suzuki +2

Transformers have achieved state-of-the-art performance in language modeling tasks. However, the reasons behind their tremendous success are still unclear. In this paper, towards a…