Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks

Abstract

Linking human whole-body motion and natural language is of great interest for the generation of semantic representations of observed human behaviors as well as for the generation of robot behaviors based on natural language input. While there has been a large body of research in this area, most approaches that exist today require a symbolic representation of motions (e.g. in the form of motion primitives), which have to be defined a-priori or require complex segmentation algorithms. In contrast, recent advances in the field of neural networks and especially deep learning have demonstrated that sub-symbolic representations that can be learned end-to-end usually outperform more traditional approaches, for applications such as machine translation. In this paper we propose a generative model that learns a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks (RNNs) and sequence-to-sequence learning. Our approach does not require any segmentation or manual feature engineering and learns a distributed representation, which is shared for all motions and descriptions. We evaluate our approach on 2,846 human whole-body motions and 6,187 natural language descriptions thereof from the KIT Motion-Language Dataset. Our results clearly demonstrate the effectiveness of the proposed model: We show that our model generates a wide variety of realistic motions only from descriptions thereof in form of a single sentence. Conversely, our model is also capable of generating correct and detailed natural language descriptions from human motions.

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• The KIT Motion-Language Dataset
• Deep representation learning for human motion prediction and classification
• Visualizing Natural Language Descriptions: A Survey

Cited by in corpus (14)

• Action2Motion: Conditioned Generation of 3D Human Motions
• PADL: Language-Directed Physics-Based Character Control
• V2CNet: A Deep Learning Framework to Translate Videos to Commands for Robotic Manipulation
• Progressive Transformers for End-to-End Sign Language Production
• 3D Human Motion Estimation via Motion Compression and Refinement
• Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data
• PoseScript: Linking 3D Human Poses and Natural Language
• Motion2Language, unsupervised learning of synchronized semantic motion segmentation
• Verbal Focus-of-Attention System for Learning-from-Observation
• Understanding Action Sequences based on Video Captioning for Learning-from-Observation
• BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics
• Caption Generation of Robot Behaviors based on Unsupervised Learning of Action Segments
• Action2video: Generating Videos of Human 3D Actions
• Guided Attention for Interpretable Motion Captioning