4 papers · 1 filter
Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
Peng Xu, Sijia Chen, Junzhuo Li +1
Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ti…
Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
Kening Zheng, Wei-Chieh Huang, Jiahao Huo +9
Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we…
Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
Junzhuo Li, Peijie Jiang, Changxin Tian +3
This paper presents a novel extension of neural scaling laws to Mixture-of-Experts (MoE) models, focusing on the optimal allocation of compute between expert and attention sub-laye…
Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models
Bo Wang, Junzhuo Li, Hong Chen +3
Mixture-of-Experts (MoE) architectures decouple model capacity from per-token computation, enabling scaling beyond the computational limits imposed by dense scaling laws. Yet how M…