Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
Sparse Weight Decomposition for Efficient Circuit Extraction
Chuanhao Yan, Xuhan Huang, Yawen Duan +4
Dense pretrained transformers do not naturally expose interpretable units for circuit extraction. Existing approaches obtain such units by learning auxiliary sparse representations…
cs.LG2024
GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory
Haoze Wu, Zihan Qiu, Zili Wang +2
Mixture-of-Experts (MoE) has been demonstrated as an efficient method to scale up models. By dynamically and sparsely selecting activated experts, MoE can effectively reduce comput…