961 citations · 1.3k across the 23 of their papers we have counts for
37 papers
Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking
Keshav Santhanam, Jon Saad-Falcon, Martin Franz +7
Neural information retrieval (IR) systems have progressed rapidly in recent years, in large part due to the release of publicly available benchmarking tasks. Unfortunately, some di…
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
Trevor Gale, Deepak Narayanan, Cliff Young +1
We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs. Our system is motivated by the limitations of current frameworks, which restrict the dynami…
HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions
Lingjiao Chen, Zhihua Jin, Sabri Eyuboglu +3
Commercial ML APIs offered by providers such as Google, Amazon and Microsoft have dramatically simplified ML adoption in many applications. Numerous companies and academics pay to…
Estimating and Explaining Model Performance When Both Covariates and Labels Shift
Lingjiao Chen, Matei Zaharia, James Zou
Deployed machine learning (ML) models often encounter new user data that differs from their training data. Therefore, estimating how well a given model might perform on the new dat…
PLAID: An Efficient Engine for Late Interaction Retrieval
Keshav Santhanam, Omar Khattab, Christopher Potts +1
Pre-trained language models are increasingly important components across multiple information retrieval (IR) paradigms. Late interaction, introduced with the ColBERT model and rece…
Finding Label and Model Errors in Perception Data With Learned Observation Assertions
Daniel Kang, Nikos Arechiga, Sudeep Pillai +2
ML is being deployed in complex, real-world scenarios where errors have impactful consequences. In these systems, thorough testing of the ML pipelines is critical. A key component…