424 citations · 474 across the 24 of their papers we have counts for
16 papers · 1 filter
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham +14
Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation. To overco…
GQA-μP: The maximal parameterization update for grouped query attention
Kyle R. Chickering, Huijuan Wang, Mengxi Wu +7
Hyperparameter transfer across model architectures dramatically reduces the amount of compute necessary for tuning large language models (LLMs). The maximal update parameterization…
K2-V2: A 360-Open, Reasoning-Enhanced LLM
K2 Team, Zhengzhong Liu, Liping Tang +36
We introduce K2-V2, a 360-open LLM built from scratch as a superior base for reasoning adaptation, in addition to functions such as conversation and knowledge retrieval from genera…
Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
Shane Bergsma, Nolan Dey, Joel Hestness
Data curriculums have become central to successful LLM training, yet principles governing optimal data placement remain unclear. We introduce the *training re-evaluation curve (TRE…
Scaling with Collapse: Efficient and Predictable Training of LLM Families
Shane Bergsma, Bin Claire Zhang, Nolan Dey +3
Effective LLM training depends on predictable scaling of key quantities -- such as final loss and optimal hyperparameters -- with model and dataset size. Qiu et al. (2025) recently…
Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
Shane Bergsma, Nolan Dey, Gurpreet Gosal +3
Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate and weight decay . We study scaling laws for HPs: formulas for how to scale HPs…