On Limitations of the Transformer Architecture

Binghui Peng, Srini Narayanan, Christos Papadimitriou

What are the root causes of hallucinations in large language models (LLMs)? We use Communication Complexity to prove that the Transformer layer is incapable of composing functions…