4 papers
SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining
Qiulin Shang, Binyu Wang, Yongqi Qiao +3
Learning-rate (LR) scheduling plays a central role in large language model (LLM) pretraining, yet current practice still relies heavily on hand-crafted heuristics such as Warmup-Co…
QuantForge: Discovering Residual Decompositions for MXFP4 Post-Training Quantization
Qiulin Shang, Zhoutong Wu, Jie Hu +1
Four-bit post-training quantization can reduce the memory demands of large language models, but preserving accuracy under strict MXFP4 W4A4 requires coordinating several design cho…
Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking
Athanasios Glentis, Jiaxiang Li, Qiulin Shang +4
Fueled by their remarkable ability to tackle diverse tasks across multiple domains, large language models (LLMs) have grown at an unprecedented rate, with some recent models contai…
A Mathematics-Inspired Learning-to-Optimize Framework for Decentralized Optimization
Yutong He, Qiulin Shang, Xinmeng Huang +2
Most decentralized optimization algorithms are handcrafted. While endowed with strong theoretical guarantees, these algorithms generally target a broad class of problems, thereby n…