Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models

Yuanbo Tang, Yan Tang, Naifan Zhang +2

Mixture-of-Experts based large language models (MoE LLMs) have shown significant promise in multitask adaptability by dynamically routing inputs to specialized experts. Despite the…