The Role of Logic and Automata in Understanding Transformers

Anthony W. Lin, Pablo Barcelo

The advent of transformers has in recent years led to powerful and revolutionary Large Language Models (LLMs). Despite this, our understanding on the capability of transformers is…