BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

Ali Abbasi, Justin Shi, Soheil Kolouri

Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferring model weights from accelera…