What Sets the Serving Speed of Looped Language Models? Measurements and Optimizations Graded by Exactness

Dezhen Lu, Runguo Li, Zixiang Ni +5

Looped language models apply one block of layers many times per token, so a serving engine pays every per-layer overhead once per loop; we measure serving for Ouro and Huginn-0125…