565 citations · 711 across the 35 of their papers we have counts for
38 papers
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation
Kaustubh D. Dhole, Charles L. A. Clarke, Eugene Y. Agichtein
Rubrics aim to make language-model evaluation transparent by decomposing response quality into interpretable criteria. However, natural-language rubrics are often ambiguous, requir…
How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation
Chase M. Fensore, Kaustubh Dhole, Jason Fan +2
Evaluating retrieval-augmented generation (RAG) systems requires benchmarks that capture diverse question characteristics, yet practitioners lack empirical guidance on which dimens…
Agents' Last Exam
Yiyou Sun, Xinyang Han, Weichen Zhang +355
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional d…
Scaling Laws for Cross-Encoder Reranking
Rahul Seetharaman, Aman Bansal, Hamed Zamani +1
Scaling laws are well studied for language models and first-stage retrieval, but not for reranking. We present the first systematic study of scaling laws for cross-encoder reranker…
RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation
Kaustubh D. Dhole, Eugene Agichtein
Large language models (LLMs) are increasingly evaluated and sometimes trained using automated graders such as LLM-as-judges that output scalar scores or preferences. While convenie…
BabyReasoningBench: Generating Developmentally-Inspired Reasoning Tasks for Evaluating Baby Language Models
Kaustubh D. Dhole
Traditional evaluations of reasoning capabilities of language models are dominated by adult-centric benchmarks that presuppose broad world knowledge, complex instruction following,…
Adversarial Lens: Exploiting Attention Layers to Generate Adversarial Examples for Evaluation
Kaustubh Dhole
Recent advances in mechanistic interpretability suggest that intermediate attention layers encode token-level hypotheses that are iteratively refined toward the final output. In th…
Toward Honest Language Models for Deductive Reasoning
Jiarui Liu, Kaustubh Dhole, Yingheng Wang +7
Deductive reasoning is the process of deriving conclusions strictly from the given premises, without relying on external knowledge. We define honesty in this setting as a model's a…
Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge
Chase Fensore, Kaustubh Dhole, Joyce C Ho +1
We present our submission to the LiveRAG Challenge 2025, which evaluates retrieval-augmented generation (RAG) systems on dynamic test sets using the FineWeb-10BT corpus. Our final…
InsertRank: LLMs can reason over BM25 scores to Improve Listwise Reranking
Rahul Seetharaman, Kaustubh D. Dhole, Aman Bansal
Large Language Models (LLMs) have demonstrated significant strides across various information retrieval tasks, particularly as rerankers, owing to their strong generalization and k…
Generative Product Recommendations for Implicit Superlative Queries
Kaustubh D. Dhole, Nikhita Vedula, Saar Kuzi +3
In Recommender Systems, users often seek the best products through indirect, vague, or under-specified queries, such as "best shoes for trail running". Such queries, also referred…
A Multi-Encoder Frozen-Decoder Approach for Fine-Tuning Large Language Models
Kaustubh D. Dhole
Among parameter-efficient fine-tuning methods, freezing has emerged as a popular strategy for speeding up training, reducing catastrophic forgetting, and improving downstream perfo…
To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation
Kaustubh D. Dhole
Retrieval-Augmented Generation equips large language models with the capability to retrieve external knowledge, thereby mitigating hallucinations by incorporating information beyon…
PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models
Kaustubh D. Dhole
Using LLMs as rerankers requires experimenting with various hyperparameters, such as prompt formats, model choice, and reformulation strategies. We introduce PyTerrier-GenRank, a P…
ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges
Kaustubh D. Dhole, Kai Shu, Eugene Agichtein
Computational argumentation, which involves generating answers or summaries for controversial topics like abortion bans and vaccination, has become increasingly important in today'…
Chatting with Logs: An exploratory study on Finetuning LLMs for LogQL
Vishwanath Seshagiri, Siddharth Balyan, Vaastav Anand +5
Logging is a critical function in modern distributed applications, but the lack of standardization in log query languages and formats creates significant challenges. Developers cur…
Generative Query Reformulation Using Ensemble Prompting, Document Fusion, and Relevance Feedback
Kaustubh D. Dhole, Ramraj Chandradevan, Eugene Agichtein
Query Reformulation (QR) is a set of techniques used to transform a user's original search query to a text that better aligns with the user's intent and improves their search exper…
GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation
Kaustubh Dhole, Eugene Agichtein
Query Reformulation(QR) is a set of techniques used to transform a user's original search query to a text that better aligns with the user's intent and improves their search experi…
DUQGen: Effective Unsupervised Domain Adaptation of Neural Rankers by Diversifying Synthetic Query Generation
Ramraj Chandradevan, Kaustubh D. Dhole, Eugene Agichtein
State-of-the-art neural rankers pre-trained on large task-specific training data such as MS-MARCO, have been shown to exhibit strong performance on various ranking tasks without do…
QueryExplorer: An Interactive Query Generation Assistant for Search and Exploration
Kaustubh D. Dhole, Shivam Bajaj, Ramraj Chandradevan +1
Formulating effective search queries remains a challenging task, particularly when users lack expertise in a specific domain or are not proficient in the language of the content. P…
KAUCUS: Knowledge Augmented User Simulators for Training Language Model Assistants
Kaustubh D. Dhole
An effective multi-turn instruction-following assistant can be developed by creating a simulator that can generate useful interaction data. Apart from relying on its intrinsic weig…
An Interactive Query Generation Assistant using LLM-based Prompt Modification and User Feedback
Kaustubh D. Dhole, Ramraj Chandradevan, Eugene Agichtein
While search is the predominant method of accessing information, formulating effective queries remains a challenging task, especially for situations where the users are not familia…
Contextual Response Interpretation for Automated Structured Interviews: A Case Study in Market Research
Harshita Sahijwani, Kaustubh Dhole, Ankur Purwar +2
Structured interviews are used in many settings, importantly in market research on topics such as brand perception, customer habits, or preferences, which are critical to product d…