1.6k citations · 9.7k across the 222 of their papers we have counts for
232 papers
Inferring Relational Potentials in Interacting Systems
Armand Comas-Massagué, Yilun Du, Christian Fernandez +4
Systems consisting of interacting agents are prevalent in the world, ranging from dynamical systems in physics to complex biological networks. To build systems which can interact r…
Compositional Diffusion-Based Continuous Constraint Solvers
Zhutian Yang, Jiayuan Mao, Yilun Du +4
This paper introduces an approach for learning to solve continuous constraint satisfaction problems (CCSP) in robotic reasoning and planning. Previous methods primarily rely on han…
Tactile Estimation of Extrinsic Contact Patch for Stable Placement
Kei Ota, Devesh K. Jha, Krishna Murthy Jatavallabhula +2
Precise perception of contact interactions is essential for fine-grained manipulation skills for robots. In this paper, we present the design of feedback skills for robots that mus…
The Future of Fundamental Science Led by Generative Closed-Loop Artificial Intelligence
Hector Zenil, Jesper Tegnér, Felipe S. Abrahão +17
Recent advances in machine learning and AI, including Generative AI and LLMs, are disrupting technological innovation, product development, and society as a whole. AI's contributio…
Neural Amortized Inference for Nested Multi-agent Reasoning
Kunal Jha, Tuan Anh Le, Chuanyang Jin +3
Multi-agent interactions, such as communication, teaching, and bluffing, often rely on higher-order social inference, i.e., understanding how others infer oneself. Such intricate r…
Building Cooperative Embodied Agents Modularly with Large Language Models
Hongxin Zhang, Weihua Du, Jiaming Shan +5
In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instanti…
Inferring the Goals of Communicating Agents from Actions and Instructions
Lance Ying, Tan Zhi-Xuan, Vikash Mansinghka +1
When humans cooperate, they frequently coordinate their activity through both verbal communication and non-verbal actions, using this information to infer a shared goal and plan. H…
The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputs
Lance Ying, Katherine M. Collins, Megan Wei +5
Human beings are social creatures. We routinely reason about other agents, and a crucial component of this social reasoning is inferring people's goals as we learn about their acti…
From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought
Lionel Wong, Gabriel Grand, Alexander K. Lew +4
How does language inform our downstream thinking? In particular, how do humans make meaning from language--and how can we leverage a theory of linguistic meaning to build machines…
Probabilistic Adaptation of Text-to-Video Models
Mengjiao Yang, Yilun Du, Bo Dai +3
Large text-to-video models trained on internet-scale data have demonstrated exceptional capabilities in generating high-fidelity videos from arbitrary textual descriptions. However…
Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties
Hsiao-Yu Tung, Mingyu Ding, Zhenfang Chen +6
General physical scene understanding requires more than simply localizing and recognizing objects -- it requires knowledge that objects can have different latent properties (e.g.,…
Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision
Ayush Tewari, Tianwei Yin, George Cazenavette +5
Denoising diffusion models are a powerful type of generative models used to capture complex distributions of real-world signals. However, their applicability is limited to scenario…
Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models
Nan Liu, Yilun Du, Shuang Li +2
Text-to-image generative models have enabled high-resolution image synthesis across different domains, but require users to specify the content they wish to generate. In this paper…
Evaluating Language Models for Mathematics through Interactions
Katherine M. Collins, Albert Q. Jiang, Simon Frieder +11
There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of e…
Acting as Inverse Inverse Planning
Kartik Chandra, Tzu-Mao Li, Josh Tenenbaum +1
Great storytellers know how to take us on a journey. They direct characters to act -- not necessarily in the most rational way -- but rather in a way that leads to interesting situ…
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Yilun Du, Shuang Li, Antonio Torralba +2
Large language models (LLMs) have demonstrated remarkable capabilities in language generation, understanding, and few-shot learning in recent years. An extensive body of work has e…
Generalized Planning in PDDL Domains with Pretrained Large Language Models
Tom Silver, Soham Dan, Kavitha Srinivas +3
Recent work has considered whether large language models (LLMs) can function as planners: given a task, generate a plan. We investigate whether LLMs can serve as generalized planne…
Evaluating statistical language models as pragmatic reasoners
Benjamin Lipkin, Lionel Wong, Gabriel Grand +1
The relationship between communicated language and intended meaning is often probabilistic and sensitive to context. Numerous strategies attempt to estimate such a mapping, often l…
3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes
Haotian Xue, Antonio Torralba, Joshua B. Tenenbaum +3
Given a visual scene, humans have strong intuitions about how a scene can evolve over time under given actions. The intuition, often termed visual intuitive physics, is a critical…
Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following
Mingyu Ding, Yan Xu, Zhenfang Chen +4
Humans, even at a very early age, can learn visual concepts and understand geometry and layout through active interaction with the environment, and generalize their compositions to…
Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention
Mingyu Ding, Yikang Shen, Lijie Fan +5
Humans possess a versatile mechanism for extracting structured representations of our visual world. When looking at an image, we can decompose the scene into entities and their par…
Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics
Pingchuan Ma, Peter Yichen Chen, Bolei Deng +4
We propose a hybrid neural network (NN) and PDE approach for learning generalizable PDE dynamics from motion observations. Many NN approaches learn an end-to-end model that implici…
Planning with Sequence Models through Iterative Energy Minimization
Hongyi Chen, Yilun Du, Yiye Chen +2
Recent works have shown that sequence modeling can be effectively used to train reinforcement learning (RL) policies. However, the success of applying existing sequence models to p…
DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics
Sizhe Li, Zhiao Huang, Tao Chen +4
In this work, we aim to learn dexterous manipulation of deformable objects using multi-fingered hands. Reinforcement learning approaches for dexterous rigid object manipulation wou…