Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
On the Nonlinearity of Learning Rate Scaling for LLM Training
Zaiwen Yang, Huaqing Zhang, Jing Xu +1
Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs. Exist…
cs.LG2025
Task Generalization With AutoRegressive Compositional Structure: Can Learning From Tasks Generalize to Tasks?
Amirhesam Abedsoltan, Huaqing Zhang, Kaiyue Wen +3
Large language models (LLMs) exhibit remarkable task generalization, solving tasks they were never explicitly trained on with only a few demonstrations. This raises a fundamental q…
cs.LG2024
From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency
Kaiyue Wen, Huaqing Zhang, Hongzhou Lin +1
Chain-of-thought (CoT) significantly enhances the reasoning performance of large language models (LLM). While current theoretical studies often attribute this improvement to increa…