6 papers · 1 filter
Evaluating Language Models' Evaluations of Games
Katherine M. Collins, Cedegao E. Zhang, Graham Todd +9
Reasoning is not just about solving problems -- it is also about evaluating which problems are worth solving at all. Evaluations of artificial intelligence (AI) systems primarily f…
Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like People
Gabriel Grand, Valerio Pepe, Jacob Andreas +1
Many emerging applications of AI--from scientific discovery to medical diagnosis--require agents to seek information strategically: forming hypotheses, asking targeted questions, a…
Code-enabled language models can outperform reasoning models on diverse tasks
Cedegao E. Zhang, Cédric Colas, Gabriel Poesia +2
Reasoning models (RMs), language models (LMs) trained with reinforcement learning to produce long-form natural language reasoning, have been remarkably successful, but they still r…
On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts
Linlu Qiu, Cedegao E. Zhang, Joshua B. Tenenbaum +2
Language use is shaped by pragmatics -- i.e., reasoning about communicative goals and norms in context. As language models (LMs) are increasingly used as conversational agents, it…
Self-Steering Language Models
Gabriel Grand, Joshua B. Tenenbaum, Vikash K. Mansinghka +2
While test-time reasoning enables language models (LMs) to tackle complex tasks, searching or planning in natural language can be slow, costly, and error-prone. But even when LMs s…
Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models
Lionel Wong, Katherine M. Collins, Lance Ying +8
When faced with novel situations, people are able to marshal relevant considerations from a wide range of background knowledge and put these to use in inferences and predictions. W…