48 citations · 141 across the 26 of their papers we have counts for
3 papers · 1 filter
Rethinking Routing Exploration in Modern Mixture-of-Experts
Bingcong Li, Hao Ma, Michael Muehlebach +2
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained exper…
Low-Rank Adaptation Redux for Large Models
Bingcong Li, Yilang Zhang, Georgios B. Giannakis
Low-rank adaptation (LoRA) has emerged as the de facto standard for parameter-efficient fine-tuning (PEFT) of foundation models, enabling the adaptation of billion-parameter networ…
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
Yilang Zhang, Bingcong Li, Niao He +1
Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep layers are often underutilized. This pap…