Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity

Pamela D. Rivière, Sean Trott

Despite an in-principle understanding of self-attention matrix operations in Transformer language models (LMs), it remains unclear precisely how these operations map onto interpret…