ColBERT-serve: Efficient Multi-Stage Memory-Mapped Scoring

Kaili Huang, Thejas Venkatesh, Uma Dingankar +9

We study serving retrieval models, specifically late interaction models like ColBERT, to many concurrent users at once and under a small budget, in which the index may not fit in m…