28 citations · 42 across the 14 of their papers we have counts for
4 papers · 1 filter
PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Models
Lancheng Zou, Shuo Yin, Zehua Pei +3
Channel permutation is a powerful technique for enhancing the accuracy of N:M sparse models by reordering the channels of weight matrices to prioritize the retention of important w…
PreMoE: Proactive Inference for Efficient Mixture-of-Experts
Zehua Pei, Ying Zhang, Hui-Ling Zhen +6
Mixture-of-Experts (MoE) models offer dynamic computation, but are typically deployed as static full-capacity models, missing opportunities for deployment-specific specialization.…
Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
Zehua Pei, Hui-Ling Zhen, Lancheng Zou +5
Scaling large language models (LLMs) improves performance but significantly increases inference costs, with feed-forward networks (FFNs) consuming the majority of computational res…
From Pruning to Grafting: Dynamic Knowledge Redistribution via Learnable Layer Fusion
Zehua Pei, Hui-Ling Zhen, Xianzhi Yu +3
Structured pruning of Generative Pre-trained Transformers (GPTs) offers a promising path to efficiency but often suffers from irreversible performance degradation due to the discar…