10 papers
ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation
Kaustubh D. Dhole, Charles L. A. Clarke, Eugene Y. Agichtein
Rubrics aim to make language-model evaluation transparent by decomposing response quality into interpretable criteria. However, natural-language rubrics are often ambiguous, requir…
How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation
Chase M. Fensore, Kaustubh Dhole, Jason Fan +2
Evaluating retrieval-augmented generation (RAG) systems requires benchmarks that capture diverse question characteristics, yet practitioners lack empirical guidance on which dimens…
RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation
Kaustubh D. Dhole, Eugene Agichtein
Large language models (LLMs) are increasingly evaluated and sometimes trained using automated graders such as LLM-as-judges that output scalar scores or preferences. While convenie…
Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge
Chase Fensore, Kaustubh Dhole, Joyce C Ho +1
We present our submission to the LiveRAG Challenge 2025, which evaluates retrieval-augmented generation (RAG) systems on dynamic test sets using the FineWeb-10BT corpus. Our final…
Generative Product Recommendations for Implicit Superlative Queries
Kaustubh D. Dhole, Nikhita Vedula, Saar Kuzi +3
In Recommender Systems, users often seek the best products through indirect, vague, or under-specified queries, such as "best shoes for trail running". Such queries, also referred…
ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges
Kaustubh D. Dhole, Kai Shu, Eugene Agichtein
Computational argumentation, which involves generating answers or summaries for controversial topics like abortion bans and vaccination, has become increasingly important in today'…
Generative Query Reformulation Using Ensemble Prompting, Document Fusion, and Relevance Feedback
Kaustubh D. Dhole, Ramraj Chandradevan, Eugene Agichtein
Query Reformulation (QR) is a set of techniques used to transform a user's original search query to a text that better aligns with the user's intent and improves their search exper…
Leveraging Interesting Facts to Enhance User Engagement with Conversational Interfaces
Nikhita Vedula, Giuseppe Castellucci, Eugene Agichtein +2
Conversational Task Assistants (CTAs) guide users in performing a multitude of activities, such as making recipes. However, ensuring that interactions remain engaging, interesting,…
An Interactive Query Generation Assistant using LLM-based Prompt Modification and User Feedback
Kaustubh D. Dhole, Ramraj Chandradevan, Eugene Agichtein
While search is the predominant method of accessing information, formulating effective queries remains a challenging task, especially for situations where the users are not familia…
A Deep Reinforcement Learning Approach for Interactive Search with Sentence-level Feedback
Jianghong Zhou, Joyce C. Ho, Chen Lin +1
Interactive search can provide a better experience by incorporating interaction feedback from the users. This can significantly improve search accuracy as it helps avoid irrelevant…