Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs

Charles Ye, Bo Yuan, Lee Sharkey

An LLM's residual stream is both state and instruction: it encodes the current context and determines the next transformation. We introduce a parameter-free decomposition for Mixtu…