CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry

Huzama Ahmad, Cao Viet Hai Nam, Se-Young Yun

Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value. We present two efficiency methods that exploit this asymmetry.…