4 papers
Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
Subbarao Kambhampati, Karthik Valmeekam, Siddhant Bhambri +6
Intermediate token generation (ITG), where a model produces output before the solution, has become a standard method to improve the performance of language models on reasoning task…
Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
Karthik Valmeekam, Vardhan Palod, Kaya Stechly +2
Recent impressive results from large reasoning models have been interpreted as a triumph of Chain of Thought (CoT), and especially of the process of training on CoTs sampled from b…
PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?
Atharva Gundawar, Som Sagar, Ransalu Senanayake
Vision-Language Models (VLMs) are increasingly pivotal for generalist robot manipulation, enabling tasks such as physical reasoning, policy generation, and failure detection. Howev…
REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites
Divyansh Garg, Shaun VanWeelden, Diego Caples +15
We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites. REAL comprises high-fidelity, deterministic repli…