Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Heads

Shaswat Patel, Vishvesh Trivedi, Yue Han +2

Retrieval heads, a subset of attention heads in Transformers, were studied in English, showing its crucial role in retrieving information from the context. We expand the study of r…