SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts

Alexandre Muzio, Alex Sun, Churan He

The advancement of deep learning has led to the emergence of Mixture-of-Experts (MoEs) models, known for their dynamic allocation of computational resources based on input. Despite…