On the Computational Hardness of Transformers

Barna Saha, Yinzhan Xu, Christopher Ye +1

The transformer has revolutionized modern AI across language, vision, and beyond. It consists of $L$ layers, each running $H$ attention heads in parallel and feeding the combined o…