How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning

Abstract

Selection of appropriate tools and use of them when performing daily tasks is a critical function for introducing robots for domestic applications. In previous studies, however, adaptability to target objects was limited, making it difficult to accordingly change tools and adjust actions. To manipulate various objects with tools, robots must both understand tool functions and recognize object characteristics to discern a tool-object-action relation. We focus on active perception using multimodal sensorimotor data while a robot interacts with objects, and allow the robot to recognize their extrinsic and intrinsic characteristics. We construct a deep neural networks (DNN) model that learns to recognize object characteristics, acquires tool-object-action relations, and generates motions for tool selection and handling. As an example tool-use situation, the robot performs an ingredients transfer task, using a turner or ladle to transfer an ingredient from a pot to a bowl. The results confirm that the robot recognizes object characteristics and servings even when the target ingredients are unknown. We also examine the contributions of images, force, and tactile data and show that learning a variety of multimodal information results in rich perception for tool use.

arXiv commentBest Paper Award of Cognitive Robotics in ICRA2021 IEEE Robotics and Automation Letters 2021, Proceedings of the 2021 International Conference on Robotics and Automation (ICRA 2021), 2021

Cited by in corpus (8)

• A Survey of Embodied Learning for Object-Centric Robotic Manipulation
• Self-Supervised Learning of Visual Servoing for Low-Rigidity Robots Considering Temporal Body Changes
• BSL: Navigation Method Considering Blind Spots Based on ROS Navigation Stack and Blind Spots Layer for Mobile Robot
• Adaptive Robotic Tool-Tip Control Learning Considering Online Changes in Grasping State
• DeepGraviLens: a Multi-Modal Architecture for Classifying Gravitational Lensing Data
• Few-shot transfer of tool-use skills using human demonstrations with proximity and tactile sensing
• Manipulate as Human: Learning Task-oriented Manipulation Skills by Adversarial Motion Priors
• When the Inference Meets the Explicitness or Why Multimodality Can Make Us Forget About the Perfect Predictor