63 citations · 437 across the 52 of their papers we have counts for
18 papers · 1 filter
Revisiting the Roles of "Text" in Text Games
Yi Gu, Shunyu Yao, Chuang Gan +2
Text games present opportunities for natural language understanding (NLU) methods to tackle reinforcement learning (RL) challenges. However, recent work has questioned the necessit…
Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation
Peihao Chen, Dongyu Ji, Kunyang Lin +4
We address a practical yet challenging problem of training robot agents to navigate in an environment following a path described by some language instructions. The instructions oft…
Learning Active Camera for Multi-Object Navigation
Peihao Chen, Dongyu Ji, Kunyang Lin +5
Getting robots to navigate to multiple objects autonomously is essential yet difficult in robot applications. One of the key challenges is how to explore environments efficiently w…
Learning Physical Dynamics with Subequivariant Graph Neural Networks
Jiaqi Han, Wenbing Huang, Hengbo Ma +3
Graph Neural Networks (GNNs) have become a prevailing tool for learning physical dynamics. However, they still encounter several challenges: 1) Physical laws abide by symmetry, whi…
JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions
Mo Yu, Yi Gu, Xiaoxiao Guo +5
Commonsense reasoning simulates the human ability to make presumptions about our physical world, and it is an essential cornerstone in building general AI systems. We propose a new…
Retrospectives on the Embodied AI Workshop
Matt Deitke, Dhruv Batra, Yonatan Bisk +36
We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped i…
MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning
Kefan Su, Siyuan Zhou, Jiechuan Jiang +3
Decentralized learning has shown great promise for cooperative multi-agent reinforcement learning (MARL). However, non-stationarity remains a significant challenge in fully decentr…
3D Concept Grounding on Neural Fields
Yining Hong, Yilun Du, Chunru Lin +2
In this paper, we address the challenging problem of 3D concept grounding (i.e. segmenting and learning visual concepts) by looking at RGBD images and reasoning about paired questi…
Finding Fallen Objects Via Asynchronous Audio-Visual Integration
Chuang Gan, Yi Gu, Siyuan Zhou +7
The way an object looks and sounds provide complementary reflections of its physical properties. In many settings cues from vision and audition arrive asynchronously but must be in…
Prompting Decision Transformer for Few-Shot Policy Generalization
Mengdi Xu, Yikang Shen, Shun Zhang +4
Humans can leverage prior experience and learn novel tasks from a handful of demonstrations. In contrast to offline meta-reinforcement learning, which aims to achieve quick adaptat…
RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation
Pingchuan Ma, Tao Du, Joshua B. Tenenbaum +2
This work considers identifying parameters characterizing a physical system's dynamic motion directly from a video whose rendering configurations are inaccessible. Existing solutio…
Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics
Sizhe Li, Zhiao Huang, Tao Du +3
Differentiable physics has recently been shown as a powerful tool for solving soft-body manipulation tasks. However, the differentiable physics solver often gets stuck when the ini…
Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
Yining Hong, Kaichun Mo, Li Yi +4
This paper studies the problem of fixing malfunctional 3D objects. While previous works focus on building passive perception models to learn the functionality from static 3D object…
ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
Zhenfang Chen, Kexin Yi, Yunzhu Li +4
Objects' motions in nature are governed by complex interactions and their properties. While some properties, such as shape and material, can be identified via the object's visual a…
Learning Neural Acoustic Fields
Andrew Luo, Yilun Du, Michael J. Tarr +3
Our environment is filled with rich and dynamic acoustic information. When we walk into a cathedral, the reverberations as much as appearance inform us of the sanctuary's wide open…
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools
Xingyu Lin, Zhiao Huang, Yunzhu Li +3
We consider the problem of sequential robotic manipulation of deformable objects using tools. Previous works have shown that differentiable physics simulators provide gradients to…
FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
Lingjie Mei, Jiayuan Mao, Ziqi Wang +2
We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultaneously…
Linking Emergent and Natural Languages via Corpus Transfer
Shunyu Yao, Mo Yu, Yang Zhang +3
The study of language emergence aims to understand how human languages are shaped by perceptual grounding and communicative intent. Computational approaches to emergent communicati…