143 citations · 440 across the 42 of their papers we have counts for
43 papers
Robot Learning with Visual Predicted Force
Haonan Chen, Feiyang Wu, Yuxiang Ma +11
Force-aware manipulation typically relies on specialized force or tactile sensors. We show that force-aware manipulation can instead be achieved through visual force prediction fro…
Retriever: Composing the Perception-Reasoning-Action Loop for Long-Horizon Manipulation
Linfeng Zhao, Haojie Huang, Jiayuan Mao +3
Building long-horizon robot agents requires composing closed-loop pipelines -- perception, belief update, planning, and control -- whose components run at different clocks and with…
Learning Compositional Behaviors from Demonstration and Language
Weiyu Liu, Neil Nie, Ruohan Zhang +2
We introduce Behavior from Language and Demonstration (BLADE), a framework for long-horizon robotic manipulation by integrating imitation learning and model-based planning. BLADE l…
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Manling Li, Shiyu Zhao, Qineng Wang +12
We aim to evaluate Large Language Models (LLMs) for embodied decision making. While a significant body of work has been leveraging LLMs for decision making in embodied environments…
What Makes a Maze Look Like a Maze?
Joy Hsu, Jiayuan Mao, Joshua B. Tenenbaum +2
A unique aspect of human visual understanding is the ability to flexibly interpret abstract concepts: acquiring lifted rules explaining what they symbolize, grounding them across f…
Composable Part-Based Manipulation
Weiyu Liu, Jiayuan Mao, Joy Hsu +3
In this paper, we propose composable part-based manipulation (CPM), a novel approach that leverages object-part decomposition and part-part correspondences to improve learning and…
Learning Planning Abstractions from Language
Weiyu Liu, Geng Chen, Joy Hsu +2
This paper presents a framework for learning state and action abstractions in sequential decision-making domains. Our framework, planning abstraction from language (PARL), utilizes…
Learning adaptive planning representations with natural language guidance
Lionel Wong, Jiayuan Mao, Pratyusha Sharma +5
Effective planning in the real world requires not only world knowledge, but the ability to leverage that knowledge to build the right representation of the task at hand. Decades of…
What Planning Problems Can A Relational Neural Network Solve?
Jiayuan Mao, Tomás Lozano-Pérez, Joshua B. Tenenbaum +1
Goal-conditioned policies are generally understood to be "feed-forward" circuits, in the form of neural networks that map from the current state and the goal specification to the n…
Learning Reusable Manipulation Strategies
Jiayuan Mao, Joshua B. Tenenbaum, Tomás Lozano-Pérez +1
Humans demonstrate an impressive ability to acquire and generalize manipulation "tricks." Even from a single demonstration, such as using soup ladles to reach for distant objects,…
What's Left? Concept Grounding with Logic-Enhanced Foundation Models
Joy Hsu, Jiayuan Mao, Joshua B. Tenenbaum +1
Recent works such as VisProg and ViperGPT have smartly composed foundation models for visual reasoning-using large language models (LLMs) to produce programs that can be executed b…
Learning to Act from Actionless Videos through Dense Correspondences
Po-Chen Ko, Jiayuan Mao, Yilun Du +2
In this work, we present an approach to construct a video-based robot policy capable of reliably executing diverse tasks across different robots and environments from few video dem…
HandMeThat: Human-Robot Communication in Physical and Social Environments
Yanming Wan, Jiayuan Mao, Joshua B. Tenenbaum
We introduce HandMeThat, a benchmark for a holistic evaluation of instruction understanding and following in physical and social environments. While previous datasets primarily foc…
CLEVRER-Humans: Describing Physical and Causal Events the Human Way
Jiayuan Mao, Xuelin Yang, Xikun Zhang +2
Building machines that can reason about physical events and their causal relationships is crucial for flexible interaction with the physical world. However, most existing physical…
Compositional Diffusion-Based Continuous Constraint Solvers
Zhutian Yang, Jiayuan Mao, Yilun Du +4
This paper introduces an approach for learning to solve continuous constraint satisfaction problems (CCSP) in robotic reasoning and planning. Previous methods primarily rely on han…
Programmatically Grounded, Compositionally Generalizable Robotic Manipulation
Renhao Wang, Jiayuan Mao, Joy Hsu +3
Robots operating in the real world require both rich manipulation skills as well as the ability to semantically reason about when to apply those skills. Towards this goal, recent w…
NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations
Joy Hsu, Jiayuan Mao, Jiajun Wu
Grounding object properties and relations in 3D scenes is a prerequisite for a wide range of artificial intelligence tasks, such as visually grounded dialogues and embodied manipul…
Sparse and Local Networks for Hypergraph Reasoning
Guangxuan Xiao, Leslie Pack Kaelbling, Jiajun Wu +1
Reasoning about the relationships between entities from input facts (e.g., whether Ari is a grandparent of Charlie) generally requires explicit consideration of other entities that…
On the Expressiveness and Generalization of Hypergraph Neural Networks
Zhezheng Luo, Jiayuan Mao, Joshua B. Tenenbaum +1
This extended abstract describes a framework for analyzing the expressiveness, learning, and (structural) generalization of hypergraph neural networks (HyperGNNs). Specifically, we…
Learning Rational Subgoals from Demonstrations and Instructions
Zhezheng Luo, Jiayuan Mao, Jiajun Wu +3
We present a framework for learning useful subgoals that support efficient long-term planning to achieve novel goals. At the core of our framework is a collection of rational subgo…
PDSketch: Integrated Planning Domain Programming and Learning
Jiayuan Mao, Tomás Lozano-Pérez, Joshua B. Tenenbaum +1
This paper studies a model learning and online planning approach towards building flexible and general robots. Specifically, we investigate how to exploit the locality and sparsity…
Translating a Visual LEGO Manual to a Machine-Executable Plan
Ruocheng Wang, Yunzhi Zhang, Jiayuan Mao +2
We study the problem of translating an image-based, step-by-step assembly manual created by human designers into machine-interpretable instructions. We formulate this problem as a…
Programmatic Concept Learning for Human Motion Description and Synthesis
Sumith Kulal, Jiayuan Mao, Alex Aiken +1
We introduce Programmatic Motion Concepts, a hierarchical motion representation for human actions that captures both low-level motion and high-level description as motion concepts.…
FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
Lingjie Mei, Jiayuan Mao, Ziqi Wang +2
We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultaneously…