Emergent Structure in the Marginal Attention Space of Language Models

Valentino Maiorca, Walter Nelson, Francesco Locatello

While representation similarity across independently trained language models is well-documented, how internal mechanics such as attention behave across models remains far less char…