Cheng Yang

collaborators

Showing cs.LGShow all

2 papers · 1 filter

cs.LG2025

EcoSpa: Efficient Transformer Training with Coupled Sparsity

Jinqi Xiao, Cheng Luo, Lingyi Huang +8

Transformers have become the backbone of modern AI, yet their high computational demands pose critical system challenges. While sparse training offers efficiency gains, existing me…

cs.LG2024

MoE-I $^{2}$ : Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition

Cheng Yang, Yang Sui, Jinqi Xiao +7

The emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. Compared to traditional LLMs, MoE LLMs outperform traditional LLMs by…

EcoSpa: Efficient Transformer Training with Coupled Sparsity

MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition

MoE-I $^{2}$ : Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition