Geometric and Dynamic Scaling in Deep Transformers

Haoran Su, Chenyu You

Despite their empirical success, pushing Transformer architectures to extreme depth often leads to a paradoxical failure: representations become increasingly redundant, lose rank,…