A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020

Abstract

Co-speech gestures, gestures that accompany speech, play an important role in human communication. Automatic co-speech gesture generation is thus a key enabling technology for embodied conversational agents (ECAs), since humans expect ECAs to be capable of multi-modal communication. Research into gesture generation is rapidly gravitating towards data-driven methods. Unfortunately, individual research efforts in the field are difficult to compare: there are no established benchmarks, and each study tends to use its own dataset, motion visualisation, and evaluation methodology. To address this situation, we launched the GENEA Challenge, a gesture-generation challenge wherein participating teams built automatic gesture-generation systems on a common dataset, and the resulting systems were evaluated in parallel in a large, crowdsourced user study using the same motion-rendering pipeline. Since differences in evaluation outcomes between systems now are solely attributable to differences between the motion-generation methods, this enables benchmarking recent approaches against one another in order to get a better impression of the state of the art in the field. This paper reports on the purpose, design, results, and implications of our challenge.

Accepted for publication at the 26th International Conference on Intelligent User Interfaces (IUI'21). 11 pages, 5 figures

References in corpus (6)

• Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
• Gesticulator: A framework for semantically-aware speech-driven gesture generation
• Analyzing Input and Output Representations for Speech-Driven Gesture Generation
• Deep Learning Based Assessment of Synthetic Speech Naturalness
• Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings
• Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

Cited by in corpus (17)

• Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
• Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
• Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning
• The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation
• A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation
• Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022
• Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speech
• HEMVIP: Human Evaluation of Multiple Videos in Parallel
• SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents
• Evaluating Data-Driven Co-Speech Gestures of Embodied Conversational Agents through Real-Time Interaction
• BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer
• UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons
• Learning to Generate Pointing Gestures in Situated Embodied Conversational Agents
• The ReprGesture entry to the GENEA Challenge 2022
• Integrated Speech and Gesture Synthesis