Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7

Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-traini…