4 papers
Hardware Mechanisms to Dynamically Throttle AI Performance
Haiyue Ma, Lauren Malek, Joseph Forzani +1
As more capable AI models are increasingly integrated into critical computer systems, the lack of control over AI intent motivates safety mechanisms. Existing software safeguards i…
Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
Yanpeng Yu, Haiyue Ma, Krish Agarwal +10
Expert Parallelism (EP) permits Mixture of Experts (MoE) models to scale beyond a single GPU. To address load imbalance across GPUs in EP, existing approaches aim to balance the nu…
Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMM
Haiyue Ma, Jian Liu, Ronny Krashinsky
Dropout, a network operator, when enabled is likely to dramatically impact the performance of Flash-Attention, which in turn increases the end-to-end training time of Large-Languag…
MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
Haiyue Ma, Zhixu Du, Yiran Chen
In multi-GPU Mixture-of-Experts (MoE) network, experts are distributed across different GPUs, which creates load imbalance as each expert processes different number of tokens. Rece…