Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry

Ayyub Shaffy, Manoj Dhakal, Safal Shrestha +1

Counting is a simple task on which transformers can nevertheless fail at long sequence lengths. We study the cost of counting in a single attention layer, up to a maximum length $L…