1 paper · 1 filter
Jingyuan Li, Maolin Wang, Hayden So
In expert-parallel Mixture-of-Experts serving, a layer finishes only when the most loaded GPU of its group does. Token routing shifts at every decode step and concentrates on a few…