MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics

Abstract

Long-term human motion can be represented as a series of motion modes---motion sequences that capture short-term temporal dynamics---with transitions between them. We leverage this structure and present a novel Motion Transformation Variational Auto-Encoders (MT-VAE) for learning motion sequence generation. Our model jointly learns a feature embedding for motion modes (that the motion sequence can be reconstructed from) and a feature transformation that represents the transition of one motion mode to the next motion mode. Our model is able to generate multiple diverse and plausible motion sequences in the future from the same input. We apply our approach to both facial and full body motion, and demonstrate applications like analogy-based motion transfer and video synthesis.

arXiv commentPublished at ECCV 2018

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Unsupervised Learning of Video Representations using LSTMs
• Toward Multimodal Image-to-Image Translation
• DRAW: A Recurrent Neural Network For Image Generation
• Generating Videos with Scene Dynamics
• Deep Convolutional Inverse Graphics Network
• FlowNet: Learning Optical Flow with Convolutional Networks
• A Neural Representation of Sketch Drawings
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Stochastic Video Generation with a Learned Prior
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• Unsupervised Learning for Physical Interaction through Video Prediction
• Unsupervised Learning of Disentangled Representations from Video
• Learning to Generate Long-term Future via Hierarchical Prediction
• Toward Controlled Generation of Text
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Hierarchical Long-term Video Prediction without Supervision
• Recurrent Network Models for Human Dynamics
• Time-Contrastive Networks: Self-Supervised Learning from Video
• Forecasting Human Dynamics from Static Images