Theoretical limitations of multi-layer Transformer

Lijie Chen, Binghui Peng, Hongxun Wu

Transformers, especially the decoder-only variants, are the backbone of most modern large language models; yet we do not have much understanding of their expressive power except fo…