1 paper · 1 filter
Gaurav Agarwal, Ashish Garg, Isha Singhal
Concurrent autoregressive inference creates a fundamental interference problem: prefilling a newly arrived long prompt can delay tokens for requests that are already decoding. Fixe…