128 citations · 173 across the 25 of their papers we have counts for
8 papers · 1 filter
DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots
Jared Moore, Andrea Mock, Yifan Mai +9
Mental health professionals have raised concerns about risks of psychological harm from interaction with large language models (LLMs), including "delusional spirals" in which conce…
Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries
Jean Feng, Vishal Patel, Patrick Heagerty +5
Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in…
Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
Jan Batzner, Sree Harsha Nelaturu, Damian Stachura +45
AI evaluations are widely used for testing and understanding progress. However, the diverse evaluators bring with them inconsistencies that challenge analysis and comparison. First…
Adaptive auditing of AI systems with anytime-valid guarantees
Siyu Zhou, Patrick Vossler, Venkatesh Sivaraman +2
A major bottleneck in characterizing the failure modes of generative AI systems is the cost and time of annotation and evaluation. Consequently, adaptive testing paradigms have gai…
Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading
Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2
Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice…
Characterizing Delusional Spirals through Human-LLM Chat Logs
Jared Moore, Ashish Mehta, William Agnew +11
As large language models (LLMs) have proliferated, disturbing anecdotal reports of negative psychological effects, such as delusions, self-harm, and ``AI psychosis,'' have emerged…
Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis
Penny Chong, Harshavardhan Abichandani, Jiyuan Shen +4
Agent applications are increasingly adopted to automate workflows across diverse tasks. However, due to the heterogeneous domains they operate in, it is challenging to create a sca…
OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets
Jiyuan Shen, Peiyue Yuan, Atin Ghosh +2
Multimodal Large Language Models (MLLMs) enhance the potential of natural language processing. However, their actual impact on document information extraction remains unclear. In p…