Attention is Just Another Name for Coupling? A Fast-Slow ODE Perspective on Hierarchical Pretraining

Zhengyuan Gao

We re-interpret Transformer pretraining as a fast-slow, singularly perturbed flow along depth, with untied weights as its non-autonomous feature. The linearised dynamics is a depth…