Dense Backpropagation Improves Training for Sparse Mixture-of-Experts

Ashwinee Panda, Vatsal Baherwani, Zain Sarwar +4

Mixture of Experts (MoE) pretraining is more scalable than dense Transformer pretraining, because MoEs learn to route inputs to a sparse set of their feedforward parameters. Howeve…