35 citations · 71 across the 23 of their papers we have counts for
24 papers
PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement
Yian Wang, Han Yang, Minghao Guo +5
Automatically generating interactive 3D environments is crucial for scaling up robotic data collection in simulation. While prior work has primarily focused on 3D asset placement,…
Scaling up the think-aloud method
Daniel Wurgaft, Ben Prystawski, Kanishk Gandhi +3
The think-aloud method, where participants voice their thoughts as they solve a task, is a valuable source of rich data about human reasoning processes. Yet, it has declined in pop…
PickScan: Object discovery and reconstruction from handheld interactions
Vincent van der Brugge, Marc Pollefeys, Joshua B. Tenenbaum +2
Reconstructing compositional 3D representations of scenes, where each object is represented with its own 3D model, is a highly desirable capability in robotics and augmented realit…
Learning Linear Attention in Polynomial Time
Morris Yau, Ekin Akyürek, Jiayuan Mao +3
Previous research has explored the computational expressivity of Transformer models in simulating Boolean circuits or Turing machines. However, the learnability of these simulators…
Can Large Language Models Understand Symbolic Graphics Programs?
Zeju Qiu, Weiyang Liu, Haiwen Feng +7
Against the backdrop of enthusiasm for large language models (LLMs), there is a growing need to scientifically assess their capabilities and shortcomings. This is nontrivial in par…
Compositional Physical Reasoning of Objects and Events from Videos
Zhenfang Chen, Shilong Dong, Kexin Yi +5
Understanding and reasoning about objects' physical properties in the natural world is a fundamental challenge in artificial intelligence. While some properties like colors and sha…
People use fast, goal-directed simulation to reason about novel games
Cedegao E. Zhang, Katherine M. Collins, Lionel Wong +3
People can evaluate features of problems and their potential solutions well before we can effectively solve them. When considering a game we have never played, for instance, we mig…
Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads
Anoop Cherian, Kuan-Chuan Peng, Suhas Lohit +3
Recent years have seen a significant progress in the general-purpose problem solving abilities of large vision and language models (LVLMs), such as ChatGPT, Gemini, etc.; some of t…
Representational Alignment Supports Effective Machine Teaching
Ilia Sucholutsky, Katherine M. Collins, Maya Malaviya +11
A good teacher should not only be knowledgeable, but should also be able to communicate in a way that the student understands -- to share the student's representation of the world.…
SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge
Andong Wang, Bo Wu, Sunli Chen +5
Learning commonsense reasoning from visual contexts and scenes in real-world is a crucial step toward advanced artificial intelligence. However, existing video reasoning benchmarks…
LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery
Pingchuan Ma, Tsun-Hsuan Wang, Minghao Guo +5
Large Language Models have recently gained significant attention in scientific discovery for their extensive knowledge and advanced reasoning capabilities. However, they encounter…
Finding structure in logographic writing with library learning
Guangyuan Jiang, Matthias Hofer, Jiayuan Mao +3
One hallmark of human language is its combinatoriality -- reusing a relatively small inventory of building blocks to create a far larger inventory of increasingly complex structure…
Physically Compatible 3D Object Modeling from a Single Image
Minghao Guo, Bohan Wang, Pingchuan Ma +6
We present a computational framework that transforms single images into 3D physical objects. The visual geometry of a physical object in an image is determined by three orthogonal…
ContPhy: Continuum Physical Concept Learning and Reasoning from Videos
Zhicheng Zheng, Xin Yan, Zhenfang Chen +4
We introduce the Continuum Physical Dataset (ContPhy), a novel benchmark for assessing machine physical commonsense. ContPhy complements existing physical reasoning benchmarks by e…
Bayes3D: fast learning and inference in structured generative models of 3D objects and scenes
Nishad Gothoskar, Matin Ghavami, Eric Li +8
Robots cannot yet match humans' ability to rapidly learn the shapes of novel 3D objects and recognize them robustly despite clutter and occlusion. We present Bayes3D, an uncertaint…
Learning adaptive planning representations with natural language guidance
Lionel Wong, Jiayuan Mao, Pratyusha Sharma +5
Effective planning in the real world requires not only world knowledge, but the ability to leverage that knowledge to build the right representation of the task at hand. Decades of…
How to guess a gradient
Utkarsh Singhal, Brian Cheung, Kartik Chandra +4
How much can you say about the gradient of a neural network without computing a loss or knowing the label? This may sound like a strange question: surely the answer is "very little…
What Planning Problems Can A Relational Neural Network Solve?
Jiayuan Mao, Tomás Lozano-Pérez, Joshua B. Tenenbaum +1
Goal-conditioned policies are generally understood to be "feed-forward" circuits, in the form of neural networks that map from the current state and the goal specification to the n…
Learning Reusable Manipulation Strategies
Jiayuan Mao, Joshua B. Tenenbaum, Tomás Lozano-Pérez +1
Humans demonstrate an impressive ability to acquire and generalize manipulation "tricks." Even from a single demonstration, such as using soup ladles to reach for distant objects,…
What's Left? Concept Grounding with Logic-Enhanced Foundation Models
Joy Hsu, Jiayuan Mao, Joshua B. Tenenbaum +1
Recent works such as VisProg and ViperGPT have smartly composed foundation models for visual reasoning-using large language models (LLMs) to produce programs that can be executed b…
AI for Mathematics: A Cognitive Science Perspective
Cedegao E. Zhang, Katherine M. Collins, Adrian Weller +1
Mathematics is one of the most powerful conceptual systems developed and used by the human species. Dreams of automated mathematicians have a storied history in artificial intellig…
LILO: Learning Interpretable Libraries by Compressing and Documenting Code
Gabriel Grand, Lionel Wong, Maddy Bowers +4
While large language models (LLMs) now excel at code generation, a key aspect of software development is the art of refactoring: consolidating code into libraries of reusable and r…
Getting aligned on representational alignment
Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller +30
Biological and artificial information processing systems form representations of the world that they can use to categorize, reason, plan, navigate, and make decisions. How can we m…
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
Qiao Gu, Alihusein Kuwajerwala, Sacha Morin +13
For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and pla…