6 papers
Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary
S. Ashwin Hebbar, Peiyao Sheng, Sewoong Oh +1
Superhuman game engines in domains like chess have made expert-level evaluations easily accessible, yet they communicate what is true without the natural-language explanations that…
Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models
Kyuyoung Kim, Kevin Wang, Yunfei Xie +7
Training language models to produce both correct answers and sound reasoning remains an open challenge. Reinforcement learning with verifiable rewards typically optimizes only fina…
Are Robust LLM Fingerprints Adversarially Robust?
Anshul Nasery, Edoardo Contente, Alkin Kaz +2
Model fingerprinting has emerged as a promising paradigm for claiming model ownership. However, robustness evaluations of these schemes have mostly focused on benign perturbations…
CHANCERY: Evaluating Corporate Governance Reasoning Capabilities in Language Models
Lucas Irwin, Arda Kaz, Peiyao Sheng +2
Law has long been a domain that has been popular in natural language processing (NLP) applications. Reasoning (ratiocination and the ability to make connections to precedent) is a…
Open Deep Search: Democratizing Search with Open-source Reasoning Agents
Salaheddin Alzubi, Creston Brooks, Purva Chiniya +9
We introduce Open Deep Search (ODS) to close the increasing gap between the proprietary search AI solutions, such as Perplexity's Sonar Reasoning Pro and OpenAI's GPT-4o Search Pre…
Training AI to be Loyal
Sewoong Oh, Himanshu Tyagi, Pramod Viswanath
Loyal AI is loyal to the community that builds it. An AI is loyal to a community if the community has ownership, alignment, and control. Community owned models can only be used wit…