Variable-Width Transformers

Zhaofeng Wu, Oliver Sieberling, Shawn Tan +3

Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across…