Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

Jongseok Park, Sunga Kim, Zhenyu Gu +2

Mixture of Experts (MoE) architecture has become the standard for state-of-the-art large language models, owing to its computational efficiency through sparse expert activation. Ho…