MQA: Answering the Question via Robotic Manipulation

Abstract

In this paper, we propose a novel task, Manipulation Question Answering (MQA), where the robot performs manipulation actions to change the environment in order to answer a given question. To solve this problem, a framework consisting of a QA module and a manipulation module is proposed. For the QA module, we adopt the method for the Visual Question Answering (VQA) task. For the manipulation module, a Deep Q Network (DQN) model is designed to generate manipulation actions for the robot to interact with the environment. We consider the situation where the robot continuously manipulating objects inside a bin until the answer to the question is found. Besides, a novel dataset that contains a variety of object models, scenarios and corresponding question-answer pairs is established in a simulation environment. Extensive experiments have been conducted to validate the effectiveness of the proposed framework.

arXiv commenthas been accepted by Robotics: Science and Systems 2021

References in corpus (6)

• Interactive Perception: Leveraging Action in Perception and Perception in Action
• Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter
• Rearrangement: A Challenge for Embodied AI
• Deep Reinforcement Learning for Robotic Pushing and Picking in Cluttered Environment
• Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching
• Towards Embodied Scene Description

Cited by in corpus (3)

• Survey on reinforcement learning for language processing
• Deep Learning for Embodied Vision Navigation: A Survey
• LEMMA: Learning Language-Conditioned Multi-Robot Manipulation