4 papers
TiPToP: A Modular Open-Vocabulary Robot Manipulation System That Plans
William Shen, Nishanth Kumar, Sahit Chintalapudi +8
We present TiPToP, a modular manipulation system that integrates pretrained foundation models with a GPU-accelerated Task and Motion Planner to solve tasks directly from RGB images…
Follow the Signs: Using Textual Cues and LLMs to Guide Efficient Robot Navigation
Jing Cao, Nishanth Kumar, Aidan Curtis
Autonomous navigation in unfamiliar environments often relies on geometric mapping and planning strategies that overlook rich semantic cues such as signs, room numbers, and textual…
Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfaction
Aidan Curtis, Nishanth Kumar, Jing Cao +2
Recent developments in pretrained large language models (LLMs) applied to robotics have demonstrated their capacity for sequencing a set of discrete skills to achieve open-ended go…
MMToM-QA: Multimodal Theory of Mind Question Answering
Chuanyang Jin, Yutong Wu, Jing Cao +7
Theory of Mind (ToM), the ability to understand people's mental states, is an essential ingredient for developing machines with human-level social intelligence. Recent machine lear…