3 papers
cs.AI2026
Function Over Form: Distributional Orthogonalization in Mixture-of-Experts with Replica Expert Mechanism
Jinfan He, Yunzhuo Liu, Kai Zhang +3
The scaling of LLMs increasingly relies on MoE architectures to decouple active computation from total parameter count. However, the efficacy of MoE is often constrained by expert…
cs.CL2026
CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residuals
Hanzuo Liu, Xuan Qi, Chunyu Liu +6
Repeated queries over shared documents repeatedly execute the same lower transformer layers. We introduce CoMem, which makes split depth j an explicit reusable-context axis: write…
cs.LG2026
SparseForge: Efficient Semi-Structured LLM Sparsification via Annealing of Hessian-Guided Soft-Mask
Liu Hanzuo, Chaofan Lin, Weixuan Sun +4
Semi-structured sparsity provides a practical path to accelerate large language models (LLMs) with native hardware support, but post-training semi-structured pruning often suffers…