9 citations · 10 across the 8 of their papers we have counts for
8 papers
Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar +4
As user simulators are increasingly used for interactive training and evaluation of AI assistants, it is essential that they represent the diverse behaviors of real users. While ex…
Measuring Combinatorial Creativity in Large Language Models and Humans
Samuel Schapiro, Sumuk Shashidhar, Akshay K. Jagadish +11
Combinatorial creativity, the process of combining familiar ideas in unfamiliar ways, is one of the foremost engines of scientific, technological, and artistic progress. This work…
Question Generation for Assessing Early Literacy Reading Comprehension
Xiaocheng Yang, Sumuk Shashidhar, Dilek Hakkani-Tur
Assessment of reading comprehension through content-based interactions plays an important role in the reading acquisition process. In this paper, we propose a novel approach for ge…
Spark: A System for Scientifically Creative Idea Generation
Aishik Sanyal, Samuel Schapiro, Sumuk Shashidhar +3
Recently, large language models (LLMs) have shown promising abilities to generate novel research ideas in science, a direction which coincides with many foundational principles in…
AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents
Takyoung Kim, Janvijay Singh, Shuhaib Mehri +6
The growing capabilities of large language models (LLMs) in instruction-following and context-understanding lead to the era of agents with numerous applications. Among these, task…
YourBench: Easy Custom Evaluation Sets for Everyone
Sumuk Shashidhar, Clémentine Fourrier, Alina Lozovskia +3
Evaluating large language models (LLMs) effectively remains a critical bottleneck, as traditional static benchmarks suffer from saturation and contamination, while human evaluation…
Unsupervised Human Preference Learning
Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai +1
Large language models demonstrate impressive reasoning abilities but struggle to provide personalized content due to their lack of individual user preference information. Existing…
Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai +2
The dominance of proprietary LLMs has led to restricted access and raised information privacy concerns. High-performing open-source alternatives are crucial for information-sensiti…