COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Search

Shibal Ibrahim, Wenyu Chen, Hussein Hazimeh +3

The sparse Mixture-of-Experts (Sparse-MoE) framework efficiently scales up model capacity in various domains, such as natural language processing and vision. Sparse-MoEs select a s…