4 citations · 42 across the 48 of their papers we have counts for
Showing 2023 · cs.CLShow all
3 papers · 2 filters
cs.CL2023
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts
Shwai He, Run-Ze Fan, Liang Ding +3
Scaling the size of language models usually leads to remarkable advancements in NLP tasks. But it often comes with a price of growing computational cost. Although a sparse Mixture…
cs.CL2023
Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models
Miaoxi Zhu, Qihuang Zhong, Li Shen +4
Quantization is a promising approach for reducing memory overhead and accelerating inference, especially in large pre-trained language model (PLM) scenarios. While having no access…
cs.CL2023★ 3 cited
Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
Boan Liu, Liang Ding, Li Shen +4
The Mixture of Experts (MoE) has emerged as a highly successful technique in deep learning, based on the principle of divide-and-conquer to maximize model capacity without signific…