1.5k citations · 1.9k across the 37 of their papers we have counts for
Showing cs.LGShow all
3 papers · 1 filter
cs.LG2024
Turn Waste into Worth: Rectifying Top- Router of MoE
Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei +7
Sparse Mixture of Experts (MoE) models are popular for training large language models due to their computational efficiency. However, the commonly used top- routing mechanism su…
cs.LG2024★ 1 cited
Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT
Zhengfu He, Xuyang Ge, Qiong Tang +3
Sparse dictionary learning has been a rapidly growing technique in mechanistic interpretability to attack superposition and extract more human-understandable features from model ac…
cs.LG2021★ 6 cited
Learning to Teach with Student Feedback
Yitao Liu, Tianxiang Sun, Xipeng Qiu +1
Knowledge distillation (KD) has gained much attention due to its effectiveness in compressing large-scale pre-trained models. In typical KD methods, the small student model is trai…