Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models

Sheng Shen, Le Hou, Yanqi Zhou +17

Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.…