Unpaired Motion Style Transfer from Video to Animation

Abstract

Transferring the motion style from one animation clip to another, while preserving the motion content of the latter, has been a long-standing problem in character animation. Most existing data-driven approaches are supervised and rely on paired data, where motions with the same content are performed in different styles. In addition, these approaches are limited to transfer of styles that were seen during training. In this paper, we present a novel data-driven framework for motion style transfer, which learns from an unpaired collection of motions with style labels, and enables transferring motion styles not observed during training. Furthermore, our framework is able to extract motion styles directly from videos, bypassing 3D reconstruction, and apply them to the 3D input motion. Our style transfer network encodes motions into two latent codes, for content and for style, each of which plays a different role in the decoding (synthesis) process. While the content code is decoded into the output motion by several temporal convolutional layers, the style code modifies deep features via temporally invariant adaptive instance normalization (AdaIN). Moreover, while the content code is encoded from 3D joint rotations, we learn a common embedding for style from either 3D or 2D joint positions, enabling style extraction from videos. Our results are comparable to the state-of-the-art, despite not requiring paired training data, and outperform other methods when transferring previously unseen styles. To our knowledge, we are the first to demonstrate style transfer directly from videos to 3D animations - an ability which enables one to extend the set of style examples far beyond motions captured by MoCap systems.

arXiv commentSIGGRAPH 2020. Project page: https://deepmotionediting.github.io/style_transfer , Video: https://www.youtube.com/watch?v=m04zuBSdGrc , Code: https://github.com/DeepMotionEditing/deep-motion-editing

References in corpus (6)

• Instance Normalization: The Missing Ingredient for Fast Stylization
• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Learning Character-Agnostic Motion for Motion Retargeting in 2D
• Few-Shot Unsupervised Image-to-Image Translation
• Neural Rendering and Reenactment of Human Actor Videos

Cited by in corpus (23)

• ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters
• Motion Puzzle: Arbitrary Motion Style Transfer by Body Part
• GANimator: Neural Motion Synthesis from a Single Sequence
• A Survey on Deep Learning for Skeleton-Based Human Animation
• Conditional Motion In-betweening
• Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model
• Iterative Motion Editing with Natural Language
• AdaptNet: Policy Adaptation for Physics-Based Character Control
• RSMT: Real-time Stylized Motion Transition for Characters
• Generating Emotive Gaits for Virtual Agents Using Affect-Based Autoregression
• Pose Representations for Deep Skeletal Animation
• MOCHA: Real-Time Motion Characterization via Context Matching
• WalkTheDog: Cross-Morphology Motion Alignment via Phase Manifolds
• Decoupling Contact for Fine-Grained Motion Style Transfer
• An Implicit Physical Face Model Driven by Expression and Style
• Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes
• Development of a Realistic Crowd Simulation Environment for Fine-grained Validation of People Tracking Methods
• AStF: Motion Style Transfer via Adaptive Statistics Fusor
• ExpertEdit: Learning Skill-Aware Motion Editing from Expert Videos
• D-LORD for Motion Stylization
• STyMo: Fast and Controllable Few-Shot Motion Style Transfer
• Deep Generative Modelling of Human Reach-and-Place Action
• Motion Style Slider: Endpoint-Supervised Continuous Style Control for Human Motion Diffusion