34 citations · 49 across the 12 of their papers we have counts for
Showing 2024Show all
2 papers · 1 filter
cs.LG2024
Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training
Atli Kosson, Bettina Messmer, Martin Jaggi
Learning Rate Warmup is a popular heuristic for training neural networks, especially at larger batch sizes, despite limited understanding of its benefits. Warmup decreases the upda…
cs.LG2024★ 2 cited
Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
Alexander Hägele, Elie Bakouch, Atli Kosson +3
Scale has become a main ingredient in obtaining strong machine learning models. As a result, understanding a model's scaling properties is key to effectively designing both the rig…