4 papers
Shape Mutating Expert Compression:LorExperts and BTExperts
Inesh Chakrabarti, Sourjya Roy, Bowen Bao +3
Mixture-of-Experts (MoE) language models deliver high capacity at low per-token compute, but deploying them cheaply requires compressing their many expert weight matrices. Expert p…
SDFP: Speculative Decoding with FIT-Pruned Models for Training-Free and Plug-and-Play LLM Acceleration
Hanyu Wei, Zunhai Su, Peng Lu +4
Large language models (LLMs) underpin interactive multimedia applications such as captioning, retrieval, recommendation, and creative content generation, yet their autoregressive d…
Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
Yixing Xu, Chao Li, Xuanwu Yin +4
Low-rank adaptation (LoRA) is one of the most popular methods among parameter-efficient fine-tuning (PEFT) methods to adapt pre-trained large language models (LLMs) to specific dow…
DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization
Haowei Zhu, Dehua Tang, Ji Liu +12
Diffusion models have achieved remarkable progress in the field of image generation due to their outstanding capabilities. However, these models require substantial computing resou…