TidyBot: Personalized Robot Assistance with Large Language Models

Abstract

For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people's preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of large language models (LLMs) to infer generalized user preferences that are broadly applicable to future interactions. This approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios.

Accepted to Autonomous Robots (AuRo) - Special Issue: Large Language Models in Robotics, 2023 and IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023. Project page: https://tidybot.cs.princeton.edu

References in corpus (6)

• Learning Transferable Visual Models From Natural Language Supervision
• Text2Motion: From Natural Language Instructions to Feasible Plans
• TidyBot: Personalized Robot Assistance with Large Language Models
• BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
• Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning
• My House, My Rules: Learning Tidying Preferences with Graph Neural Networks

Cited by in corpus (17)

• Text2Motion: From Natural Language Instructions to Feasible Plans
• TidyBot: Personalized Robot Assistance with Large Language Models
• A Survey on Integration of Large Language Models with Intelligent Robots
• Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration
• Real-World Robot Applications of Foundation Models: A Review
• VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robots
• Deploying and Evaluating LLMs to Program Service Mobile Robots
• Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models
• Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review
• Distributed Threat Intelligence at the Edge Devices: A Large Language Model-Driven Approach
• Goal-oriented Semantic Communication for Robot Arm Reconstruction in Digital Twin: Feature and Temporal Selections
• Bootstrapping Cognitive Agents with a Large Language Model
• Visual Environment-Interactive Planning for Embodied Complex-Question Answering
• Task-Driven Co-Design of Mobile Manipulators
• LLM-GROP: Visually Grounded Robot Task and Motion Planning with Large Language Models
• Hierarchical Language Models for Semantic Navigation and Manipulation in an Aerial-Ground Robotic System
• In-situ Value-aligned Human-Robot Interactions with Physical Constraints