343 citations · 521 across the 20 of their papers we have counts for
4 papers · 1 filter
Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries
Chaitanya Malaviya, Joseph Chee Chang, Dan Roth +3
Language model users often issue queries that lack specification, where the context under which a query was issued -- such as the user's identity, the query's intent, and the crite…
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya +3
Language agents, built on top of language models (LMs), are systems that can interact with complex environments, such as the open web. In this work, we examine whether such agents…
DOLOMITES: Domain-Specific Long-Form Methodical Tasks
Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev +7
Experts in various fields routinely perform methodical writing tasks to plan, organize, and report their work. From a clinician writing a differential diagnosis for a patient, to a…
Calibrating Large Language Models with Sample Consistency
Qing Lyu, Kumar Shridhar, Chaitanya Malaviya +6
Accurately gauging the confidence level of Large Language Models' (LLMs) predictions is pivotal for their reliable application. However, LLMs are often uncalibrated inherently and…