Self-Supervised Equivariant Scene Synthesis from Video
arXiv:2102.00863
Abstract
We propose a self-supervised framework to learn scene representations from video that are automatically delineated into background, characters, and their animations. Our method capitalizes on moving characters being equivariant with respect to their transformation across frames and the background being constant with respect to that same transformation. After training, we can manipulate image encodings in real time to create unseen combinations of the delineated components. As far as we know, we are the first method to perform unsupervised extraction and synthesis of interpretable background, character, and animation. We demonstrate results on three datasets: Moving MNIST with backgrounds, 2D video game sprites, and Fashion Modeling.
arXiv commentarXiv admin note: text overlap with arXiv:2011.05787
References in corpus (8)
- Scikit-learn: Machine Learning in Python
- Generating Videos with Scene Dynamics
- Decomposing Motion and Content for Natural Video Sequence Prediction
- CycleGAN, a Master of Steganography
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Equivariant Neural Rendering
- Coordinate-based Texture Inpainting for Pose-Guided Image Generation
- Transformable Bottleneck Networks