9 papers · 1 filter
Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
Xiao Ye, Jacob Dineen, Evan Zhu +3
Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two cha…
Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
Jacob Dineen, Aswin RRV, Zhikun Xu +1
Co-evolutionary self-play, where one language model generates problems and another solves them, promises curriculum learning without human supervision. The promise breaks down earl…
Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications
Xiao Ye, Jacob Dineen, Zhaonan Li +11
Medical Large language models achieve strong scores on standard benchmarks; however, the transfer of those results to safe and reliable performance in clinical workflows remains a…
ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluation
Qin Liu, Jacob Dineen, Yuxi Huang +4
Benchmarks are central to measuring the capabilities of large language models and guiding model development, yet widespread data leakage from pretraining corpora undermines their v…
RECAP: Transparent Inference-Time Emotion Alignment for Medical Dialogue Systems
Adarsh Srinivasan, Jacob Dineen, Muhammad Umar Afzal +3
Large language models in healthcare often produce emotionally flat or opaque responses, failing to provide the transparent reasoning required for clinical trust. We present RECAP (…
CC-LEARN: Cohort-based Consistency Learning
Xiao Ye, Shaswat Shrivastava, Zhaonan Li +6
Large language models excel at many tasks but still struggle with consistent, robust reasoning. We introduce Cohort-based Consistency Learning (CC-Learn), a reinforcement learning…