8 papers
Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models
Hongyu Zhang, Cheng Yan, Xiang Xia +1
Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool. Training-free expe…
REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models
Xiang Xia, Cheng Yan, Yiming Zhang +3
Mixture-of-experts (MoE) models increase parameter capacity by activating only a small subset of experts for each token. This conditional-computation paradigm has enabled autoregre…
SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models
Jian Gu, Aldeida Aleti, Chunyang Chen +1
Residual-stream analysis asks how language-model computation evolves across depth, but intermediate decoding requires comparable readout coordinates across layers. If embedding anc…
Rethinking Weight Tying: Pseudo-Inverse Tying for LM Stable Training and Updates
Jian Gu, Aldeida Aleti, Chunyang Chen +1
Weight tying is widely used in compact language models to reduce parameters by sharing the token table between the input embedding and the output projection. However, parameter sha…
Beyond Neural Incompatibility: Cross-Scale Knowledge Transfer in Language Models through Latent Semantic Alignment
Jian Gu, Aldeida Aleti, Chunyang Chen +1
Language Models (LMs) encode substantial knowledge in their parameters, yet it remains unclear how to transfer such knowledge in a fine-grained manner, namely parametric knowledge…
SeMe: Training-Free Language Model Merging via Semantic Alignment
Jian Gu, Aldeida Aleti, Chunyang Chen +1
Despite the remarkable capabilities of Language Models (LMs) across diverse tasks, no single model consistently outperforms others, necessitating efficient methods to combine their…