4 papers
FastLane: Efficient Routed Systems for Late-Interaction Retrieval
Ramnath Kumar, Prateek Jain, Cho-Jui Hsieh
Late-interaction retrieval models like ColBERT achieve superior accuracy by enabling token-level interactions, but their computational cost hinders scalability and integration with…
Compressing Many-Shots in In-Context Learning
Devvrit Khatri, Pranamya Kulkarni, Nilesh Gupta +9
Large Language Models (LLMs) have been shown to be able to learn different tasks without explicit finetuning when given many input-output examples / demonstrations through In-Conte…
Matryoshka Model Learning for Improved Elastic Student Models
Chetan Verma, Aditya Srinivas Timmaraju, Cho-Jui Hsieh +7
Industry-grade ML models are carefully designed to meet rapidly evolving serving constraints, which requires significant resources for model development. In this paper, we propose…
Matryoshka Quantization
Pranav Nair, Puranjay Datta, Jeff Dean +2
Quantizing model weights is critical for reducing the communication and inference costs of large models. However, quantizing models -- especially to low precisions like int4 or int…