5 citations · 6 across the 6 of their papers we have counts for
Showing 2025Show all
2 papers · 1 filter
cs.LG2025
Fantastic Pretraining Optimizers and Where to Find Them
Kaiyue Wen, David Hall, Tengyu Ma +1
AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We posit that two methodologi…
cs.LG2025
Non-Asymptotic Length Generalization
Thomas Chen, Tengyu Ma, Zhiyuan Li
Length generalization is the ability of a learning algorithm to learn a hypothesis which generalizes to longer inputs than the inputs in the training set. In this paper, we provide…