4 citations · 4 across the 7 of their papers we have counts for
9 papers
Data Schedules and Learning Rate Schedules Should Be Designed Together
Haodong Wen, Kairong Luo, Tingkai Yan +2
Mid-training (MT) is a training phase in LLM training pipelines that continues pretraining (PT) on smaller datasets of higher value. Taking the PT and MT phases as a whole, trainin…
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek-AI, :, Anyi Xu +585
The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation…
Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Haodong Wen, Yiran Zhang, Yingfa Chen +1
While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an…
Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold
Xinghan Li, Haodong Wen, Kaifeng Lyu
Despite the popularity of the Adam optimizer in practice, most theoretical analyses study Stochastic Gradient Descent (SGD) as a proxy for Adam, and little is known about how the s…
How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
Kairong Luo, Zhenbo Sun, Haodong Wen +5
Due to the scarcity of high-quality data, large language models (LLMs) are often trained on mixtures of data with varying quality levels, even after sophisticated data curation. A…
Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression
Tingkai Yan, Haodong Wen, Binghui Li +3
While data scaling laws of large language models (LLMs) have been widely examined in the one-pass regime with massive corpora, their form under limited data and repeated epochs rem…
A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules
Kairong Luo, Haodong Wen, Shengding Hu +5
Training large models is both resource-intensive and time-consuming, making it crucial to understand the quantitative relationship between model performance and hyperparameters. In…
Exploring the Robustness of In-Context Learning with Noisy Labels
Chen Cheng, Xinzhi Yu, Haodong Wen +4
Recently, the mysterious In-Context Learning (ICL) ability exhibited by Transformer architectures, especially in large language models (LLMs), has sparked significant research inte…
Rethinking the Graph Polynomial Filter via Positive and Negative Coupling Analysis
Haodong Wen, Bodong Du, Ruixun Liu +2
Recently, the optimization of polynomial filters within Spectral Graph Neural Networks (GNNs) has emerged as a prominent research focus. Existing spectral GNNs mainly emphasize pol…