From Small to Large: Generalization Bounds for Transformers on Variable-Size Inputs

Anastasiia Alokhina, Pan Li

Transformers exhibit a notable property of \emph{size generalization}, demonstrating an ability to extrapolate from smaller token sets to significantly longer ones. This behavior h…