Chain and Causal Attention for Efficient Entity Tracking

Erwan Fagnou, Paul Caillon, Blaise Delattre +1

This paper investigates the limitations of transformers for entity-tracking tasks in large language models. We identify a theoretical constraint, showing that transformers require…