From the 1 of 25 linked papers with an AI index.
256 citations · 457 across the 19 of their papers we have counts for
25 papers
Non-Asymptotic Length Generalization
Thomas Chen, Tengyu Ma, Zhiyuan Li
Length generalization is the ability of a learning algorithm to learn a hypothesis which generalizes to longer inputs than the inputs in the training set. In this paper, we provide…
PENCIL: Long Thoughts with Short Memory
Chenxiao Yang, Nathan Srebro, David McAllester +1
While state-of-the-art LLMs have demonstrated great promise of using long Chains-of-Thought (CoT) to boost reasoning, scaling it up to more challenging problems at test-time is fun…
Structured Preconditioners in Adaptive Optimization: A Unified Analysis
Shuo Xie, Tianhao Wang, Sashank Reddi +2
We present a novel unified analysis for a broad class of adaptive optimization algorithms with structured (e.g., layerwise, diagonal, and kronecker-factored) preconditioners for bo…
Weak-to-Strong Generalization Even in Random Feature Networks, Provably
Marko Medvedev, Kaifeng Lyu, Dingli Yu +3
Weak-to-Strong Generalization (Burns et al., 2024) is the phenomenon whereby a strong student, say GPT-4, learns a task from a weak teacher, say GPT-2, and ends up significantly ou…
Adam Exploits -geometry of Loss Landscape via Coordinate-wise Adaptivity
Shuo Xie, Mohamad Amin Mohamadi, Zhiyuan Li
Adam outperforms SGD when training language models. Yet this advantage is not well-understood theoretically -- previous convergence analysis for Adam and SGD mainly focuses on the…
Implicit Bias of AdamW: Norm Constrained Optimization
Shuo Xie, Zhiyuan Li
Adam with decoupled weight decay, also known as AdamW, is widely acclaimed for its superior performance in language modeling tasks, surpassing Adam with regularization in…