VQA-based Robotic State Recognition Optimized with Genetic Algorithm

Abstract

State recognition of objects and environment in robots has been conducted in various ways. In most cases, this is executed by processing point clouds, learning images with annotations, and using specialized sensors. In contrast, in this study, we propose a state recognition method that applies Visual Question Answering (VQA) in a Pre-Trained Vision-Language Model (PTVLM) trained from a large-scale dataset. By using VQA, it is possible to intuitively describe robotic state recognition in the spoken language. On the other hand, there are various possible ways to ask about the same event, and the performance of state recognition differs depending on the question. Therefore, in order to improve the performance of state recognition using VQA, we search for an appropriate combination of questions using a genetic algorithm. We show that our system can recognize not only the open/closed of a refrigerator door and the on/off of a display, but also the open/closed of a transparent door and the state of water, which have been difficult to recognize.

arXiv commentAccepted at ICRA2023

References in corpus (2)

• OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
• Vision-Language Intelligence: Tasks, Representation Learning, and Large Models

Cited by in corpus (5)

• Real-World Robot Applications of Foundation Models: A Review
• Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors
• Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization
• Daily Assistive View Control Learning of Low-Cost Low-Rigidity Robot via Large-Scale Vision-Language Model
• Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization