7 papers
From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes
Yichen Wang, Fanghui Liu, Yudong Chen
Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the observed loss. We study this…
Muon Sublates the Edge of Stability in LLM Pretraining
Yanzhe Chen, Qifang Zhao, Xiaoxiao Xu +1
Muon is increasingly used for language-model pretraining, yet its large-step dynamics are not captured by the classical edge-of-stability (EoS) picture of gradient descent (GD). In…
LeanMarathon: Toward Reliable AI Co-Mathematicians through Long-Horizon Lean Autoformalization
Yuanhe Zhang, Yuekai Sun, Taiji Suzuki +2
Long-horizon autoformalization of research mathematics fails not only at hard lemmas, but at scale: statements drift, dependencies tangle, context decays, and local repairs corrupt…
AI4SLT: Empirical Processes in Lean 4 for Formal Statistical Learning Theory
Yuanhe Zhang, Jason D. Lee, Fanghui Liu
We present the first comprehensive Lean 4 formalization of statistical learning theory (SLT) grounded in empirical process theory. Our en-to-end formal infrastructure implement the…
DAG-Math: Graph-of-Thought Guided Mathematical Reasoning in LLMs
Yuanhe Zhang, Ilja Kuzborskij, Jason D. Lee +2
Large Language Models (LLMs) demonstrate strong performance on mathematical problems when prompted with Chain-of-Thought (CoT), yet it remains unclear whether this success stems fr…
The Curve: The Shape of Generalization through the Lens of Norm-based Capacity Control
Yichen Wang, Yudong Chen, Lorenzo Rosasco +1
Understanding how the test risk scales with model complexity is a central question in machine learning. Classical theory is challenged by the learning curves observed for large ove…
LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently
Yuanhe Zhang, Fanghui Liu, Yudong Chen
This paper explores how theory can guide and enhance practical algorithms, using Low-Rank Adaptation (LoRA, Hu et al. 2022) in large language models as a case study. We rigorously…