Landscape-Aware Growing: The Power of a Little LAG

Stefani Karp, Nikunj Saunshi, Sobhan Miryoosefi +2

Recently, there has been increasing interest in efficient pretraining paradigms for training Transformer-based models. Several recent approaches use smaller models to initialize la…