Identifying and Evaluating Inactive Heads in Pretrained LLMs

Pedro Sandoval-Segura, Xijun Wang, Ashwinee Panda +4

Attention is foundational to large language models (LLMs), enabling different heads to have diverse focus on relevant input tokens. However, learned behaviors like attention sinks,…