Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning

Sanskar Pandey, Ruhaan Chopra, Saad Murtaza Bhat +1

Mixture-of-Experts (MoE) models enable conditional computation by routing inputs to specialized experts, but these experts rely on identical inductive biases, thus limiting represe…