DepthBench: Measuring How Residual Connections Enable More Computational Depth

Keyu Wang, Yangyi Huang, Jiale Kang +3

Depth is a natural way to increase the computational capacity in Transformers, yet the contribution of deeper layers can diminish as depth grows larger. Recent approaches enhance n…