Attention-Aware Routing: Coupling Routing and Attention in MoEs

Despoina Kosmopoulou, Anastasios Tsetsilas, Efthymios Georgiou +3

In Mixture-of-Experts language models, the router typically selects and weights experts based on the token's hidden state, utilizing limited contextual information. We propose Atte…