Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models

Zihan Wang, Rui Pan, Jiarui Yao +7

We propose Chain-of-Experts (CoE), a new Mixture-of-Experts (MoE) architecture that introduces sequential expert communication within each layer. Unlike traditional MoE models, whe…