4 papers · 1 filter
Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought
Ching-Chieh Tsao, Zhuoyi Lin, Wenya Wang
Mixture-of-Experts architectures have revolutionized scaling, yet the underlying logic of their routing remains a black box. In this paper, we uncover a fundamental governing princ…
Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training
Hang Chen, Jiaying Zhu, Hongyang Chen +3
The "Locate-then-Update" paradigm has become a predominant approach in the post-training of large language models (LLMs), identifying critical components via mechanistic interpreta…
Skill Path: Unveiling Language Skills from Circuit Graphs
Hang Chen, Jiaying Zhu, Xinyu Yang +1
Circuit graph discovery has emerged as a fundamental approach to elucidating the skill mechanistic of language models. Despite the output faithfulness of circuit graphs, they suffe…
Quantifying Semantic Emergence in Language Models
Hang Chen, Xinyu Yang, Jiaying Zhu +1
Large language models (LLMs) are widely recognized for their exceptional capacity to capture semantics meaning. Yet, there remains no established metric to quantify this capability…