Learned Equivariant Rendering without Transformation Supervision

Abstract

We propose a self-supervised framework to learn scene representations from video that are automatically delineated into objects and background. Our method relies on moving objects being equivariant with respect to their transformation across frames and the background being constant. After training, we can manipulate and render the scenes in real time to create unseen combinations of objects, transformations, and backgrounds. We show results on moving MNIST with backgrounds.

Workshop on Differentiable Vision, Graphics, and Physics in Machine Learning at NeurIPS 2020