Time-Aware and View-Aware Video Rendering for Unsupervised Representation Learning

Abstract

The recent success in deep learning has lead to various effective representation learning methods for videos. However, the current approaches for video representation require large amount of human labeled datasets for effective learning. We present an unsupervised representation learning framework to encode scene dynamics in videos captured from multiple viewpoints. The proposed framework has two main components: Representation Learning Network (RL-NET), which learns a representation with the help of Blending Network (BL-NET), and Video Rendering Network (VR-NET), which is used for video synthesis. The framework takes as input video clips from different viewpoints and time, learns an internal representation and uses this representation to render a video clip from an arbitrary given viewpoint and time. The ability of the proposed network to render video frames from arbitrary viewpoints and time enable it to learn a meaningful and robust representation of the scene dynamics. We demonstrate the effectiveness of the proposed method in rendering view-aware as well as time-aware video clips on two different real-world datasets including UCF-101 and NTU-RGB+D. To further validate the effectiveness of the learned representation, we use it for the task of view-invariant activity classification where we observe a significant improvement (~26%) in the performance on NTU-RGB+D dataset compared to the existing state-of-the art methods.

References in corpus (11)

• Conditional Generative Adversarial Nets
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• DRAW: A Recurrent Neural Network For Image Generation
• Generating Videos with Scene Dynamics
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Unsupervised Learning of View-invariant Action Representations
• Stochastic Variational Video Prediction
• Dynamics Transfer GAN: Generating Video by Transferring Arbitrary Temporal Dynamics from a Source Video to a Single Target Image
• Improving Video Generation for Multi-functional Applications
• Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks
• Adversarial Framework for Unsupervised Learning of Motion Dynamics in Videos