On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts

Fanqi Yan, Huy Nguyen, Dung Le +3

The softmax-contaminated mixture of experts (MoE) model is deployed when a large-scale pre-trained model, which plays the role of a fixed expert, is fine-tuned for learning downstr…