activity
20232026
most citedDense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models

2 citations · 5 across the 8 of their papers we have counts for

collaborators
Showing cs.LGShow all

2 papers · 1 filter

cs.LG2024

: towards data-free Transferable Parameter Efficient Finetuning

Runqian Wang, Soumya Ghosh, David Cox +4

Low-rank adapters (LoRA) and their variants are popular parameter-efficient fine-tuning (PEFT) techniques that closely match full model fine-tune performance while requiring only a…

cs.LG2024★ 2 cited

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models

Bowen Pan, Yikang Shen, Haokun Liu +5

Mixture-of-Experts (MoE) language models can reduce computational costs by 2-4 compared to dense models without sacrificing performance, making them more efficient in compu…