Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors

Abstract

In recent years, a number of models that learn the relations between vision and language from large datasets have been released. These models perform a variety of tasks, such as answering questions about images, retrieving sentences that best correspond to images, and finding regions in images that correspond to phrases. Although there are some examples, the connection between these pre-trained vision-language models and robotics is still weak. If they are directly connected to robot motions, they lose their versatility due to the embodiment of the robot and the difficulty of data collection, and become inapplicable to a wide range of bodies and situations. Therefore, in this study, we categorize and summarize the methods to utilize the pre-trained vision-language models flexibly and easily in a way that the robot can understand, without directly connecting them to robot motions. We discuss how to use these models for robot motion selection and motion planning without re-training the models. We consider five types of methods to extract information understandable for robots, and show the results of state recognition, object recognition, affordance recognition, relation recognition, and anomaly detection based on the combination of these five methods. We expect that this study will add flexibility and ease-of-use, as well as new applications, to the recognition behavior of existing robots.

arXiv commentAccepted at Humanoids2023

References in corpus (4)

• OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models
• Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
• VQA-based Robotic State Recognition Optimized with Genetic Algorithm

Cited by in corpus (4)

• Real-World Robot Applications of Foundation Models: A Review
• Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization
• CLIP-Clique: Graph-based Correspondence Matching Augmented by Vision Language Models for Object-based Global Localization
• Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization