Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability

Ivan Lee, Nan Jiang, Taylor Berg-Kirkpatrick

What is the relationship between model architecture and the ability to perform in-context learning? In this empirical study, we take the first steps toward answering this question.…