Efficient Iterative Retrieval with Heterogeneous Batching

Dohyun Park, Hubertus Franke, Daniel G. Waddington +2

Modern information retrieval increasingly employs both embedding and generative models to handle complex queries. However, current serving systems suffer from low throughput and po…