1 citations · 1 across the 11 of their papers we have counts for
6 papers · 1 filter
How Bregman Divergences Shape Shampoo
Bing Liu, Wenjie Zhou, Chengcheng Zhao +4
Understanding the principles behind Shampoo has recently guided the development of more effective neural network optimizers. These methods learn a preconditioner by optimizing the…
GNMR: Runtime Stability Control for Low-Precision Large Language Model Training
Boao Kong, Weichen Jia, Engao Zhang +6
Training stability is a key bottleneck in low-precision language model training: efficient low-cost paths can still produce short-lived numerical risks at a small set of operators.…
BROS: Bias-Corrected Randomized Subspaces for Memory-Efficient Single-Loop Bilevel Optimization
Hengrui Zhang, Boao Kong, Engao Zhang +1
Stochastic bilevel optimization (SBO) has become a standard framework for hyperparameter learning, data reweighting, representation learning, and data-mixture optimization in deep…
Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specialization
Rizhen Hu, Yuan Cao, Boao Kong +2
Sparse Mixture-of-Experts (MoE) models scale Transformers efficiently but suffer from expert overlap -- redundant representations across experts and routing ambiguity, resulting in…
Row-Stochastic Matrices Can Provably Outperform Doubly Stochastic Matrices in Decentralized Learning
Bing Liu, Boao Kong, Limin Lu +2
Decentralized learning often involves a weighted global loss with heterogeneous node weights . We revisit two natural strategies for incorporating these weights: (i) embedding t…
CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure
Boao Kong, Junzhu Liang, Yuxi Liu +2
Low-rank architectures have become increasingly important for efficient large language model (LLM) pre-training, providing substantial reductions in both parameter complexity and m…