94 citations · 245 across the 21 of their papers we have counts for
Showing 2022Show all
2 papers · 1 filter
cs.LG2022★ 1 cited
A Study on Transformer Configuration and Training Objective
Fuzhao Xue, Jianghai Chen, Aixin Sun +6
Transformer-based models have delivered impressive results on many tasks, particularly vision and language tasks. In many model training situations, conventional configurations are…
cs.LG2022★ 5 cited
One Student Knows All Experts Know: From Sparse to Dense
Fuzhao Xue, Xiaoxin He, Xiaozhe Ren +2
Human education system trains one student by multiple experts. Mixture-of-experts (MoE) is a powerful sparse architecture including multiple experts. However, sparse MoE model is e…