8 citations · 12 across the 39 of their papers we have counts for
6 papers · 1 filter
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
Fan Mo, Yuxuan Han, Geng Zhang +2
Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models. However, sparse ac…
K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling
Zhiwei Tang, Yuanyu He, Yizheng Han +4
Autoregressive (AR) language modeling is the dominant paradigm for text generation, yet its sequential token-by-token decoding makes inference memory-bound and inefficient. Existin…
MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
Geng Zhang, Yuxuan Han, Yuxuan Lou +3
Mixture-of-Experts (MoE) enables efficient scaling of large language models by activating only a subset of experts per input token. However, deploying MoE-based models incurs signi…
Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights
Zhiyuan Liang, Dongwen Tang, Yuhao Zhou +11
Modern Parameter-Efficient Fine-Tuning (PEFT) methods such as low-rank adaptation (LoRA) reduce the cost of customizing large language models (LLMs), yet still require a separate o…
Recurrent Diffusion for Large-Scale Parameter Generation
Kai Wang, Dongwen Tang, Wangbo Zhao +3
Parameter generation has long struggled to match the scale of today large vision and language models, curbing its broader utility. In this paper, we introduce Recurrent Diffusion f…
Prioritize Alignment in Dataset Distillation
Zekai Li, Ziyao Guo, Wangbo Zhao +8
Dataset Distillation aims to compress a large dataset into a significantly more compact, synthetic one without compromising the performance of the trained models. To achieve this,…