5 papers
CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
Lance Ying, Jinzhou Wu, Yingshan Susan Wang +53
Understanding and modeling human intelligence are parallel goals shared by artificial intelligence (AI) and cognitive science. As AI systems grow increasingly capable, in what ways…
Shared circuits predict whether LLMs generalize across formats in arithmetic reasoning
Andrea Gregor de Varda, Sana Pandey, Pengrui Han +2
In many forms of reasoning, including arithmetic reasoning, generalizing across superficial changes in input format is effortless for humans: anyone who can solve 2+5 can also solv…
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
Investigating Knowledge Transfer Across Interactive Dialogue Games
Filippo Momentè, Mir Nafis Sharear Shopnil, Andrea de Varda +5
Dialogue games represent a challenging setting where complex cognitive skills are required to accomplish tasks while coordinating with other players. Considering that language repr…
Modular Cognitive Architecture Emerges in Large Language Models
Pengrui Han, Jacob Andreas, Evelina Fedorenko +1
The human brain exhibits a striking degree of functional specialization, with distinct networks supporting language, formal reasoning, reasoning about other minds, and reasoning ab…