Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

Abstract

This paper presents a novel framework for speech-driven gesture production, applicable to virtual agents to enhance human-computer interaction. Specifically, we extend recent deep-learning-based, data-driven methods for speech-driven gesture generation by incorporating representation learning. Our model takes speech as input and produces gestures as output, in the form of a sequence of 3D coordinates. We provide an analysis of different representations for the input (speech) and the output (motion) of the network by both objective and subjective evaluations. We also analyse the importance of smoothing of the produced motion. Our results indicated that the proposed method improved on our baseline in terms of objective measures. For example, it better captured the motion dynamics and better matched the motion-speed distribution. Moreover, we performed user studies on two different datasets. The studies confirmed that our proposed method is perceived as more natural than the baseline, although the difference in the studies was eliminated by appropriate post-processing: hip-centering and smoothing. We conclude that it is important to take both motion representation and post-processing into account when designing an automatic gesture-production method.

arXiv commentExtension of our IVA'19 paper. Accepted at the International Journal of Human-Computer Interaction. See more at https://svito-zar.github.io/audio2gestures/. arXiv admin note: substantial text overlap with arXiv:1903.03369

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
• QuaterNet: A Quaternion-based Recurrent Model for Human Motion
• Analyzing Input and Output Representations for Speech-Driven Gesture Generation
• A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020
• Speech-driven Animation with Meaningful Behaviors
• Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings
• Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis

Cited by in corpus (12)

• Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
• A Comprehensive Review of Data-Driven Co-Speech Gesture Generation
• A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020
• Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis
• Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022
• Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speech
• HEMVIP: Human Evaluation of Multiple Videos in Parallel
• The ReprGesture entry to the GENEA Challenge 2022
• Learning to Generate Pointing Gestures in Situated Embodied Conversational Agents
• Integrated Speech and Gesture Synthesis
• A Framework for Integrating Gesture Generation Models into Interactive Conversational Agents
• Multimodal Quantitative Measures for Multiparty Behaviour Evaluation