Unsupervised Learning of Object Structure and Dynamics from Videos

Abstract

Extracting and predicting object structure and dynamics from videos without supervision is a major challenge in machine learning. To address this challenge, we adopt a keypoint-based image representation and learn a stochastic dynamics model of the keypoints. Future frames are reconstructed from the keypoints and a reference frame. By modeling dynamics in the keypoint coordinate space, we achieve stable learning and avoid compounding of errors in pixel space. Our method improves upon unstructured representations both for pixel-level video prediction and for downstream tasks requiring object-level understanding of motion dynamics. We evaluate our model on diverse datasets: a multi-agent sports dataset, the Human3.6M dataset, and datasets based on continuous control tasks from the DeepMind Control Suite. The spatially structured representation outperforms unstructured representations on a range of motion-related tasks such as object tracking, action recognition and reward prediction.

arXiv comment33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Unsupervised Learning of Video Representations using LSTMs
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• A Recurrent Latent Variable Model for Sequential Data
• DeepMind Control Suite
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Learning Latent Dynamics for Planning from Pixels
• Stochastic Video Generation with a Learned Prior
• Video (language) modeling: a baseline for generative models of natural videos
• Stochastic Adversarial Video Prediction
• Towards Accurate Generative Models of Video: A New Metric & Challenges
• Learning to Generate Long-term Future via Hierarchical Prediction
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Unsupervised Learning of Object Keypoints for Perception and Control
• Hierarchical Long-term Video Prediction without Supervision
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Stochastic Prediction of Multi-Agent Interactions from Partial Observations
• Accurate and Diverse Sampling of Sequences based on a "Best of Many" Sample Objective
• Unsupervised Discovery of Parts, Structure, and Dynamics

Cited by in corpus (5)

• FitVid: Overfitting in Pixel-Level Video Prediction
• Vision-based system identification and 3D keypoint discovery using dynamics constraints
• AutoTrajectory: Label-free Trajectory Extraction and Prediction from Videos using Dynamic Points
• Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation
• CGAP2: Context and gap aware predictive pose framework for early detection of gestures