11 citations · 32 across the 41 of their papers we have counts for
27 papers · 1 filter
Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
Delip Rao, Chris Callison-Burch
LLM judges now score outputs against rubrics, in benchmarks and as training rewards. Decision models return probabilities over permitted answers with a confidence score but no free…
EviStreams: Human-in-the-Loop AI Data Extraction for Systematic Reviews in Medicine
Sai Karthik Kosuri, Ankita Shashikant Bhosale, Michael Glick +2
Systematic reviews underpin clinical guidelines, yet their data-extraction step is a major expert-labor bottleneck bound by a protocolized workflow: two reviewers extract each stud…
Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning
Ajay Patel, Kartik Hosanagar, Ramayya Krishnan +2
Large language models (LLMs) are improving rapidly as reflected in benchmark scores, yet these AI benchmarks largely test capabilities such as factual recall, narrow question answe…
Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
Delip Rao, Chris Callison-Burch
Whether a rubric-based LLM judge can replace human annotation is decided by its measured agreement with human labels. Yet the same verdicts can support wildly varying agreement num…
When Verification Fails: How Compositionally Infeasible Claims Escape Rejection
Muxin Liu, Delip Rao, Grace Kim +1
Scientific claim verification, the task of determining whether claims are entailed by scientific evidence, is fundamental to establishing discoveries in evidence while preventing m…
Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents
Delip Rao, Eric Wong, Chris Callison-Burch
Large language models and deep research agents supply citation URLs to support their claims, yet the reliability of these citations has not been systematically measured. We address…
What Do Claim Verification Datasets Actually Test? A Reasoning Trace Analysis
Delip Rao, Chris Callison-Burch
Despite rapid progress in claim verification, we lack a systematic understanding of what reasoning these benchmarks actually exercise. We generate structured reasoning traces for 2…
Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks
Delip Rao, Chris Callison-Burch
Rubric-based LLM judges have become indispensable for evaluating and optimizing systems on non-verifiable tasks, where success cannot be reduced to exact programmatic checks. Yet t…
FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
Ajay Patel, Colin Raffel, Chris Callison-Burch
Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstruct…
Machine Text Detectors are Membership Inference Attacks
Ryuto Koike, Liam Dugan, Masahiro Kaneko +2
Although membership inference attacks (MIAs) and machine-generated text detection target different goals, their methods often exploit similar signals based on a language model's pr…
The Media Bias Detector: A Framework for Annotating and Analyzing the News at Scale
Samar Haider, Amir Tohidi, Jenny S. Wang +4
Mainstream news organizations shape public perception not only directly through the articles they publish but also through the choices they make about which topics to cover (or ign…
Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap
Andrew Zhu, Chris Callison-Burch
Imagine AI assistants that enhance conversations without interrupting them: quietly providing relevant information during a medical consultation, seamlessly preparing materials as…
Toward Beginner-Friendly LLMs for Language Learning: Controlling Difficulty in Conversation
Meiqing Jin, Liam Dugan, Chris Callison-Burch
Practicing conversations with large language models (LLMs) presents a promising alternative to traditional in-person language learning. However, most LLMs generate text at a near-n…
Domain Gating Ensemble Networks for AI-Generated Text Detection
Arihant Tripathi, Liam Dugan, Charis Gao +6
As state-of-the-art language models continue to improve, the need for robust detection of machine-generated text becomes increasingly critical. However, current state-of-the-art ma…
First Steps Towards Overhearing LLM Agents: A Case Study With Dungeons & Dragons Gameplay
Andrew Zhu, Evan Osgood, Chris Callison-Burch
Much work has been done on conversational LLM agents which directly assist human users with tasks. We present an alternative paradigm for interacting with LLM agents, which we call…
NSF-SciFy: Mining the NSF Awards Database for Scientific Claims
Delip Rao, Weiqiu You, Eric Wong +1
We introduce NSF-SciFy, a comprehensive dataset of scientific claims and investigation proposals extracted from National Science Foundation award abstracts. While previous scientif…
mStyleDistance: Multilingual Style Embeddings and their Evaluation
Justin Qiu, Jiacheng Zhu, Ajay Patel +2
Style embeddings are useful for stylistic analysis and style transfer; however, only English style embeddings have been made available. We introduce Multilingual StyleDistance (mSt…
GenAI Content Detection Task 3: Cross-Domain Machine-Generated Text Detection Challenge
Liam Dugan, Andrew Zhu, Firoj Alam +3
Recently there have been many shared tasks targeting the detection of generated text from Large Language Models (LLMs). However, these shared tasks tend to focus either on cases wh…
WithdrarXiv: A Large-Scale Dataset for Retraction Study
Delip Rao, Jonathan Young, Thomas Dietterich +1
Retractions play a vital role in maintaining scientific integrity, yet systematic studies of retractions in computer science and other STEM fields remain scarce. We present Withdra…
StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples
Ajay Patel, Jiacheng Zhu, Justin Qiu +4
Style representations aim to embed texts with similar writing styles closely and texts with different styles far apart, regardless of content. However, the contrastive triplets oft…
Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness
Bryan Li, Fiona Luo, Samar Haider +7
The paradigm of retrieval-augmented generated (RAG) helps mitigate hallucinations of large language models (LLMs). However, RAG also introduces biases contained within the retrieve…
Uncovering Differences in Persuasive Language in Russian versus English Wikipedia
Bryan Li, Aleksey Panasyuk, Chris Callison-Burch
We study how differences in persuasive language across Wikipedia articles, written in either English and Russian, can uncover each culture's distinct perspective on different subje…
You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling
Jaewoo Song, Andrew Zhu, Chris Callison-Burch
Developing a consistent and reliable AI game master for text-based games is a challenging task due to the limitations of large language models (LLMs) and the complexity of the game…
ReDel: A Toolkit for LLM-Powered Recursive Multi-Agent Systems
Andrew Zhu, Liam Dugan, Chris Callison-Burch
Recently, there has been increasing interest in using Large Language Models (LLMs) to construct complex multi-agent systems to perform tasks such as compiling literature reviews, d…