When is Warmstarting Effective for Scaling Language Models?

Neeratyoy Mallik, Maciej Janowski, Johannes Hog +4

Model growth from a given checkpoint aims to accelerate training of a larger model, offering potential resource savings. Despite recent interest, warmstarting has seen limited prac…