12 citations · 21 across the 18 of their papers we have counts for
18 papers
Muse Spark Safety & Preparedness Report
Cristina Menghini, Peter Ney, Hamza Kwisaba +117
Muse Spark is the latest large language model developed by Meta. In this report, we first present evaluations for catastrophic risk domains under Meta's Advanced AI Scaling Framewo…
On Safety Risks in Experience-Driven Self-Evolving Agents
Weixiang Zhao, Yichen Zhang, Yingshuo Wang +8
Experience-driven self-evolution has emerged as a promising paradigm for improving the autonomy of large language model agents, yet its reliance on self-curated experience introduc…
Progress over Points: Reframing LM Benchmarks Around Scientific Objectives
Alwin Jin, Sean M. Hendryx, Vaskar Nath
Current benchmarks that test LLMs on static, already-solved problems (e.g., math word problems) effectively demonstrated basic capability acquisition. The natural progression has b…
ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi +13
Deep Research (DR) is an emerging agent application that leverages large language models (LLMs) to address open-ended queries. It requires the integration of several capabilities,…
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Xiang Deng, Jeff Da, Edwin Pan +19
We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, compl…
Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
Anisha Gunjal, Anthony Wang, Elaine Lau +4
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for complex reasoning tasks with clear correctness signals such as math and coding. However, extending it…
Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models
Vaskar Nath, Elaine Lau, Anisha Gunjal +3
We study the process through which reasoning models trained with reinforcement learning on verifiable rewards (RLVR) can learn to solve new problems. We find that RLVR drives perfo…
Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards
Jeff Da, Clinton Wang, Xiang Deng +3
Reinforcement Learning from Verifiable Rewards (RLVR) has been widely adopted as the de facto method for enhancing the reasoning capabilities of large language models and has demon…
EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges
Clinton J. Wang, Dean Lee, Cristina Menghini +7
As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging mu…
ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark
Vaskar Nath, Pranav Raja, Claire Yoon +1
Despite recent advances in AI, the development of systems capable of executing complex, multi-step reasoning tasks involving multiple tools remains a significant challenge. Current…
Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents
Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar +9
For safety reasons, large language models (LLMs) are trained to refuse harmful user instructions, such as assisting dangerous activities. We study an open question in this work: do…
Planning In Natural Language Improves LLM Search For Code Generation
Evan Wang, Federico Cassano, Catherine Wu +7
While scaling training compute has led to remarkable improvements in large language models (LLMs), scaling inference compute has not yet yielded analogous gains. We hypothesize tha…
Revisiting the Superficial Alignment Hypothesis
Mohit Raghavendra, Vaskar Nath, Sean Hendryx
The Superficial Alignment Hypothesis posits that almost all of a language model's abilities and knowledge are learned during pre-training, while post-training is about giving a mod…
Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data
Spencer Whitehead, Jacob Phillips, Sean Hendryx
Multimodal language models can exhibit hallucinations in their outputs, which limits their reliability. The ability to automatically detect these errors is important for mitigating…
Learning Goal-Conditioned Representations for Language Reward Models
Vaskar Nath, Dylan Slack, Jeff Da +4
Techniques that learn improved representations via offline data or self-supervised objectives have shown impressive results in traditional reinforcement learning (RL). Nevertheless…
A Careful Examination of Large Language Model Performance on Grade School Arithmetic
Hugh Zhang, Jeff Da, Dean Lee +12
Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actuall…
Out-of-Distribution Detection & Applications With Ablated Learned Temperature Energy
Will LeVine, Benjamin Pikus, Jacob Phillips +3
As deep neural networks become adopted in high-stakes domains, it is crucial to identify when inference inputs are Out-of-Distribution (OOD) so that users can be alerted of likely…
A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift
Will LeVine, Benjamin Pikus, Anthony Chen +1
Foundation models, specifically Large Language Models (LLMs), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves trai…