HarMoEny: Efficient Multi-GPU Inference of MoE Models

Zachary Doucet, Rishi Sharma, Martijn de Vos +3

Mixture-of-Experts (MoE) models offer computational efficiency during inference by activating only a subset of specialized experts for a given input. This enables efficient model s…