23 citations · 45 across the 11 of their papers we have counts for
12 papers
LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning
Sumeet Ramesh Motwani, Daniel Nichols, Charles London +17
As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this…
AutoOR: Scalably Post-training LLMs to Autoformulate Operations Research Problems
Sumeet Ramesh Motwani, Chuan Du, Aleksander Petrov +4
Optimization problems are central to decision-making in manufacturing, logistics, scheduling, and other industrial settings. Translating complicated descriptions of these problems…
Measuring Progress in Reasoning Toward Mathematical Discovery with Automatic Verification
Erik Y. Wang, Sumeet R. Motwani, James V. Roggeveen +9
Can AI make progress on important, unsolved mathematical problems? Large language models are now capable of sophisticated mathematical and scientific reasoning, but whether they ca…
h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning
Sumeet Ramesh Motwani, Alesia Ivanova, Ziyang Cai +5
Large language models excel at short-horizon reasoning tasks, but performance drops as reasoning horizon lengths increase. Existing approaches to combat this rely on inference-time…
REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites
Divyansh Garg, Shaun VanWeelden, Diego Caples +15
We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites. REAL comprises high-fidelity, deterministic repli…
Multi-Agent Risks from Advanced AI
Lewis Hammond, Alan Chan, Jesse Clifton +41
The rapid development of advanced AI agents and the imminent deployment of many instances of these agents will give rise to multi-agent systems of unprecedented complexity. These s…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1168
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…
MALT: Improving Reasoning with Multi-Agent LLM Training
Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das +6
Large Language Models (LLMs) often produce answers with a single chain-of-thought, which restricts their ability to explore reasoning paths or self-correct flawed outputs in comple…
Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
Pranav Putta, Edmund Mills, Naman Garg +4
Large Language Models (LLMs) have shown remarkable capabilities in natural language tasks requiring complex reasoning, yet their application in agentic, multi-step reasoning within…
Unelicitable Backdoors in Language Models via Cryptographic Transformer Circuits
Andis Draguns, Andrew Gritsevskiy, Sumeet Ramesh Motwani +3
The rapid proliferation of open-source language models significantly increases the risks of downstream backdoor attacks. These backdoors can introduce dangerous behaviours during m…
Secret Collusion among AI Agents: Multi-Agent Deception via Steganography
Sumeet Ramesh Motwani, Mikhail Baranchuk, Martin Strohmeier +4
Recent capability increases in large language models (LLMs) open up applications in which groups of communicating generative AI agents solve joint tasks. This poses privacy and sec…
STARC: A General Framework For Quantifying Differences Between Reward Functions
Joar Skalse, Lucy Farnik, Sumeet Ramesh Motwani +3
In order to solve a task using reinforcement learning, it is necessary to first formalise the goal of that task as a reward function. However, for many real-world tasks, it is very…