312 citations · 2.2k across the 74 of their papers we have counts for
10 papers · 1 filter
Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware Flow
Zhenyu Jiang, Hanwen Jiang, Yuke Zhu
Dense visual correspondence plays a vital role in robotic perception. This work focuses on establishing the dense correspondence between a pair of images that captures dynamic scen…
MUTEX: Learning Unified Policies from Multimodal Task Specifications
Rutav Shah, Roberto Martín-Martín, Yuke Zhu
Humans use different modalities, such as speech, text, images, videos, etc., to communicate their intent and goals with teammates. For robots to become better assistants, we aim to…
Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation
Mingyo Seo, Steve Han, Kyutae Sim +4
We tackle the problem of developing humanoid loco-manipulation skills with deep imitation learning. The difficulty of collecting task demonstrations and training policies for human…
Symbolic State Space Optimization for Long Horizon Mobile Manipulation Planning
Xiaohan Zhang, Yifeng Zhu, Yan Ding +4
In existing task and motion planning (TAMP) research, it is a common assumption that experts manually specify the state space for task-level planning. A well-developed state space…
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
Bo Liu, Yifeng Zhu, Chongkai Gao +4
Lifelong learning offers a promising paradigm of building a generalist agent that learns and adapts over its lifespan. Unlike traditional lifelong learning problems in image and te…
Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids
Wei Dong, Chris Choy, Charles Loop +3
Indoor scene reconstruction from monocular images has long been sought after by augmented reality and robotics developers. Recent advances in neural field representations and monoc…
Voyager: An Open-Ended Embodied Agent with Large Language Models
Guanzhi Wang, Yuqi Xie, Yunfan Jiang +5
We introduce Voyager, the first LLM-powered embodied lifelong learning agent in Minecraft that continuously explores the world, acquires diverse skills, and makes novel discoveries…
Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception
Cheng-Chun Hsu, Zhenyu Jiang, Yuke Zhu
Virtualizing the physical world into virtual models has been a critical technique for robot navigation and planning in the real world. To foster manipulation with articulated objec…
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
Chen Wang, Linxi Fan, Jiankai Sun +5
Imitation learning from human demonstrations is a promising paradigm for teaching robots manipulation skills in the real world. However, learning complex long-horizon tasks often r…
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Zhuolin Yang, Wei Ping, Zihan Liu +13
Augmenting pretrained language models (LMs) with a vision encoder (e.g., Flamingo) has obtained the state-of-the-art results in image-to-text generation. However, these models stor…