TACT: Humanoid Whole-body Contact Manipulation through Deep Imitation Learning with Tactile Modality

Abstract

Manipulation with whole-body contact by humanoid robots offers distinct advantages, including enhanced stability and reduced load. On the other hand, we need to address challenges such as the increased computational cost of motion generation and the difficulty of measuring broad-area contact. We therefore have developed a humanoid control system that allows a humanoid robot equipped with tactile sensors on its upper body to learn a policy for whole-body manipulation through imitation learning based on human teleoperation data. This policy, named tactile-modality extended ACT (TACT), has a feature to take multiple sensor modalities as input, including joint position, vision, and tactile measurements. Furthermore, by integrating this policy with retargeting and locomotion control based on a biped model, we demonstrate that the life-size humanoid robot RHP7 Kaleido is capable of achieving whole-body contact manipulation while maintaining balance and walking. Through detailed experimental verification, we show that inputting both vision and tactile modalities into the policy contributes to improving the robustness of manipulation involving broad and delicate contact.

References in corpus (5)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Multi-Fingered In-Hand Manipulation with Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors
• Humanoid Loco-Manipulations Pattern Generation and Stabilization Control
• Whole-body Multi-contact Motion Control for Humanoid Robots Based on Distributed Tactile Sensors
• Optimization-based Posture Generation for Whole-body Contact Motion by Contact Point Search on the Body Surface