BRANCH-MoE: Balance-Aware Tree Routing for Large Embedding Models

Gang Fu, Adel Javanmard, MohammadHossein Bateni +1

Mixture-of-experts (MoE) layers increase model capacity without a proportional increase in per-example computation. However, conventional flat routers can yield imbalanced expert u…