48 citations · 100 across the 9 of their papers we have counts for
3 papers · 1 filter
On the Inductive Bias of Stacking Towards Improving Reasoning
Nikunj Saunshi, Stefani Karp, Shankar Krishnan +3
Given the increasing scale of model sizes, novel training strategies like gradual stacking [Gong et al., 2019, Reddi et al., 2023] have garnered interest. Stacking enables efficien…
Landscape-Aware Growing: The Power of a Little LAG
Stefani Karp, Nikunj Saunshi, Sobhan Miryoosefi +2
Recently, there has been increasing interest in efficient pretraining paradigms for training Transformer-based models. Several recent approaches use smaller models to initialize la…
Efficient Stagewise Pretraining via Progressive Subnetworks
Abhishek Panigrahi, Nikunj Saunshi, Kaifeng Lyu +4
Recent developments in large language models have sparked interest in efficient pretraining methods. Stagewise training approaches to improve efficiency, like gradual stacking and…