Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference

Yinghan Li, Yifei Li, Jiejing Zhang +13

It has long been a problem to arrange and execute irregular workloads on massively parallel devices. We propose a general framework for statically batching irregular workloads into…