Informational Antilocality and the Locality Bias in LLMs

Andrew McInnerney, Shane Storks, Steven Abney +1

We consider the ability of transformer-based language models (LLMs) to learn what we call k-antilocal languages, i.e., languages that have no mutual information across any span of…