6 papers · 1 filter
Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
Kening Zheng, Wei-Chieh Huang, Jiahao Huo +9
Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we…
Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs
Zhipeng Yang, Junzhuo Li, Siyu Xia +1
We show that large language models (LLMs) exhibit an : they sequentially decompose and execute composite tasks layer-by-layer. Two claims ground…
Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis
Junzhuo Li, Bo Wang, Xiuze Zhou +3
The interpretability of Mixture-of-Experts (MoE) models, especially those with heterogeneous designs, remains underexplored. Existing attribution methods for dense models fail to c…
Unveiling Instruction-Specific Neurons & Experts: An Analytical Framework for LLM's Instruction-Following Capabilities
Junyan Zhang, Yubo Gao, Yibo Yan +8
The finetuning of Large Language Models (LLMs) has significantly advanced their instruction-following capabilities, yet the underlying computational mechanisms driving these improv…
Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models
Yanggan Gu, Junzhuo Li, Sirui Huang +3
Aligning small language models (SLMs) with human values typically involves distilling preference knowledge from large language models (LLMs). However, existing distillation methods…
The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis
Chen Yang, Junzhuo Li, Xinyao Niu +11
Uncovering early-stage metrics that reflect final model performance is one core principle for large-scale pretraining. The existing scaling law demonstrates the power-law correlati…