The KIT Motion-Language Dataset

Abstract

Linking human motion and natural language is of great interest for the generation of semantic representations of human activities as well as for the generation of robot activities based on natural language input. However, while there have been years of research in this area, no standardized and openly available dataset exists to support the development and evaluation of such systems. We therefore propose the KIT Motion-Language Dataset, which is large, open, and extensible. We aggregate data from multiple motion capture databases and include them in our dataset using a unified representation that is independent of the capture system or marker set, making it easy to work with the data regardless of its origin. To obtain motion annotations in natural language, we apply a crowd-sourcing approach and a web-based tool that was specifically build for this purpose, the Motion Annotation Tool. We thoroughly document the annotation process itself and discuss gamification methods that we used to keep annotators motivated. We further propose a novel method, perplexity-based selection, which systematically selects motions for further annotation that are either under-represented in our dataset or that have erroneous annotations. We show that our method mitigates the two aforementioned problems and ensures a systematic annotation process. We provide an in-depth analysis of the structure and contents of our resulting dataset, which, as of October 10, 2016, contains 3911 motions with a total duration of 11.23 hours and 6278 annotations in natural language that contain 52,903 words. We believe this makes our dataset an excellent choice that enables more transparent and comparable research in this important area.

5 figures, 4 tables, submitted to Big Data journal, Special Issue on Robotics

Cited by in corpus (21)

• Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks
• Crossmodal Language Grounding in an Embodied Neurocognitive Model
• HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances
• SATO: Stable Text-to-Motion Framework
• 4D Facial Expression Diffusion Model
• PDP: Physics-Based Character Animation via Diffusion Policy
• Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language
• Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data
• PoseScript: Linking 3D Human Poses and Natural Language
• ExpGest: Expressive Speaker Generation Using Diffusion Model and Hybrid Audio-Text Guidance
• A Survey on Human Interaction Motion Generation
• Motion2Language, unsupervised learning of synchronized semantic motion segmentation
• Progressive Human Motion Generation Based on Text and Few Motion Frames
• A Framework for Evaluating Motion Segmentation Algorithms
• Temporal Consistency-Aware Text-to-Motion Generation
• Semantics-Aware Human Motion Generation from Audio Instructions
• Towards Robust and Controllable Text-to-Motion via Masked Autoregressive Diffusion
• BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics
• Motion Diffusion Autoencoders: Enabling Attribute Manipulation in Human Motion Demonstrated on Karate Techniques
• Motion-2-To-3: Leveraging 2D Motion Data for 3D Motion Generations
• MAAIG: Motion Analysis And Instruction Generation