The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought

Moritz Brösamle, Moritz Brösamle, Stephan Eckstein

Existing expressivity results for transformers typically rely on hardmax attention, high precision, and other architectural modifications that disconnect them from the models used…