Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022

Abstract

This paper reports on the second GENEA Challenge to benchmark data-driven automatic co-speech gesture generation. Participating teams used the same speech and motion dataset to build gesture-generation systems. Motion generated by all these systems was rendered to video using a standardised visualisation pipeline and evaluated in several large, crowdsourced user studies. Unlike when comparing different research papers, differences in results are here only due to differences between methods, enabling direct comparison between systems. The dataset was based on 18 hours of full-body motion capture, including fingers, of different persons engaging in a dyadic conversation. Ten teams participated in the challenge across two tiers: full-body and upper-body gesticulation. For each tier, we evaluated both the human-likeness of the gesture motion and its appropriateness for the specific speech signal. Our evaluations decouple human-likeness from gesture appropriateness, which has been a difficult problem in the field. The evaluation results show some synthetic gesture conditions being rated as significantly more human-like than 3D human motion capture. To the best of our knowledge, this has not been demonstrated before. On the other hand, all synthetic motion is found to be vastly less appropriate for the speech than the original motion-capture recordings. We also find that conventional objective metrics do not correlate well with subjective human-likeness ratings in this large evaluation. The one exception is the Fréchet gesture distance (FGD), which achieves a Kendall's tau rank correlation of around $-0.5$. Based on the challenge results we formulate numerous recommendations for system building and evaluation.

The first three authors made equal contributions and share joint first authorship. Accepted for publication in the ACM Transactions on Graphics (TOG).Please see https://youngwoo-yoon.github.io/GENEAchallenge2022/ for all challenge materials. arXiv admin note: text overlap with arXiv:2208.10441

References in corpus (15)

• Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
• Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
• Analyzing Input and Output Representations for Speech-Driven Gesture Generation
• Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
• A Comprehensive Review of Data-Driven Co-Speech Gesture Generation
• The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation
• A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020
• Deep Learning Based Assessment of Synthetic Speech Naturalness
• Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speech
• Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speech
• SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents
• Evaluating Data-Driven Co-Speech Gestures of Embodied Conversational Agents through Real-Time Interaction
• The ReprGesture entry to the GENEA Challenge 2022
• Integrated Speech and Gesture Synthesis
• Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis

Cited by in corpus (2)

• A Comprehensive Review of Data-Driven Co-Speech Gesture Generation
• TranSTYLer: Multimodal Behavioral Style Transfer for Facial and Body Gestures Generation