53 citations · 139 across the 14 of their papers we have counts for
14 papers
SUN: Agentic Robot Policy Learning with Persistent Task Programs
Weiqi Wang, Zhi Li, Yudong Lei +7
Model-based control can directly execute specified objectives, while learning can amortize such behaviors into reactive policies, making their combination a natural solution to mul…
GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
Yichi Zhang, Ziqiao Ma, Xiaofeng Gao +3
Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences…
Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning
Jiachen Li, Qiaozi Gao, Michael Johnston +6
Prompt-based learning has been demonstrated as a compelling paradigm contributing to large language models' tremendous success (LLMs). Inspired by their success in language tasks,…
LEMMA: Learning Language-Conditioned Multi-Robot Manipulation
Ran Gong, Xiaofeng Gao, Qiaozi Gao +3
Complex manipulation tasks often require robots with complementary capabilities to collaborate. We introduce a benchmark for LanguagE-Conditioned Multi-robot MAnipulation (LEMMA) f…
Alexa, play with robot: Introducing the First Alexa Prize SimBot Challenge on Embodied AI
Hangjie Shi, Leslie Ball, Govind Thattai +39
The Alexa Prize program has empowered numerous university students to explore, experiment, and showcase their talents in building conversational agents through challenges like the…
ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes
Ran Gong, Jiangyong Huang, Yizhou Zhao +9
Understanding the continuous states of objects is essential for task learning and planning in the real world. However, most existing task learning benchmarks assume discrete (e.g.,…
Alexa Arena: A User-Centric Interactive Platform for Embodied AI
Qiaozi Gao, Govind Thattai, Suhaila Shakiah +24
We introduce Alexa Arena, a user-centric simulation platform for Embodied AI (EAI) research. Alexa Arena provides a variety of multi-room layouts and interactable objects, for the…
VRKitchen2.0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse
Yizhou Zhao, Steven Gong, Xiaofeng Gao +2
With the recent progress of simulations by 3D modeling software and game engines, many researchers have focused on Embodied AI tasks in the virtual environment. However, the resear…
Effects of Augmented-Reality-Based Assisting Interfaces on Drivers' Object-wise Situational Awareness in Highly Autonomous Vehicles
Xiaofeng Gao, Xingwei Wu, Samson Ho +2
Although partially autonomous driving (AD) systems are already available in production vehicles, drivers are still required to maintain a sufficient level of situational awareness…
DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following
Xiaofeng Gao, Qiaozi Gao, Ran Gong +3
Language-guided Embodied AI benchmarks requiring an agent to navigate an environment and manipulate objects typically allow one-way communication: the human user gives a natural la…
Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration
Xiaofeng Gao, Luyao Yuan, Tianmin Shu +2
Aligning humans' assessment of what a robot can do with its true capability is crucial for establishing a common ground between human and robot partners when they collaborate on a…
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Xiaofeng Gao, Ran Gong, Yizhou Zhao +3
Human collaborators can effectively communicate with their partners to finish a common task by inferring each other's mental states (e.g., goals, beliefs, and desires). Such mind-a…
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
Xiaofeng Gao, Ran Gong, Tianmin Shu +3
One of the main challenges of advancing task-oriented learning such as visual task planning and reinforcement learning is the lack of realistic and standardized environments for tr…
Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions
Tianmin Shu, Xiaofeng Gao, Michael S. Ryoo +1
In this paper, we present a general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions, and transfer…