Transformer as Linear Expansion of Learngene

Shiyu Xia, Miaosen Zhang, Xu Yang +3

We propose expanding the shared Transformer module to produce and initialize Transformers of varying depths, enabling adaptation to diverse resource constraints. Drawing an analogy…