Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems

Guowei Liu, Hongming Li, Yaning Guo +5

Deploying large-scale MoE models presents challenges in memory capacity and bandwidth for expert activation. While Attention-FFN Disaggregation (AFD) has emerged as a potential arc…