Video Reenactment as Inductive Bias for Content-Motion Disentanglement
arXiv:2102.00324 · doi:10.1109/TIP.2022.3153140
Abstract
Independent components within low-dimensional representations are essential inputs in several downstream tasks, and provide explanations over the observed data. Video-based disentangled factors of variation provide low-dimensional representations that can be identified and used to feed task-specific models. We introduce MTC-VAE, a self-supervised motion-transfer VAE model to disentangle motion and content from videos. Unlike previous work on video content-motion disentanglement, we adopt a chunk-wise modeling approach and take advantage of the motion information contained in spatiotemporal neighborhoods. Our model yields independent per-chunk representations that preserve temporal consistency. Hence, we reconstruct whole videos in a single forward-pass. We extend the ELBO's log-likelihood term and include a Blind Reenactment Loss as an inductive bias to leverage motion disentanglement, under the assumption that swapping motion features yields reenactment between two videos. We evaluate our model with recently-proposed disentanglement metrics and show that it outperforms a variety of methods for video motion-content disentanglement. Experiments on video reenactment show the effectiveness of our disentanglement in the input space where our model outperforms the baselines in reconstruction quality and motion alignment.
arXiv commentPre-print to appear in IEEE Trans. on Image Processing. Project page, high resolution images, and source code at https://mipl.gitlab.io/mtc-vae/
References in corpus (20)
- Auto-Encoding Variational Bayes
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Unsupervised Learning of Video Representations using LSTMs
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Disentangling by Factorising
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Stochastic Video Generation with a Learned Prior
- FD-GAN: Pose-guided Feature Distilling GAN for Robust Person Re-identification
- Towards a Definition of Disentangled Representations
- Disentangling factors of variation in deep representations using adversarial training
- Stochastic Adversarial Video Prediction
- First Order Motion Model for Image Animation
- Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations
- Learning Deep Disentangled Embeddings with the F-Statistic Loss
- Weakly-Supervised Disentanglement Without Compromises
- Generating Visually Aligned Sound from Videos
- Weakly Supervised Disentanglement with Guarantees
- On Linear Identifiability of Learned Representations
- TransMoMo: Invariance-Driven Unsupervised Video Motion Retargeting
- Toward a Controllable Disentanglement Network