Learning Temporal Dynamics from Cycles in Narrated Video
arXiv:2101.02337
Abstract
Learning to model how the world changes as time elapses has proven a challenging problem for the computer vision community. We propose a self-supervised solution to this problem using temporal cycle consistency jointly in vision and language, training on narrated video. Our model learns modality-agnostic functions to predict forward and backward in time, which must undo each other when composed. This constraint leads to the discovery of high-level transitions between moments in time, since such transitions are easily inverted and shared across modalities. We justify the design of our model with an ablation study on different configurations of the cycle consistency problem. We then show qualitatively and quantitatively that our approach yields a meaningful, high-level model of the future and past. We apply the learned dynamics model without further training to various tasks, such as predicting future action and temporally ordering sets of images. Project page: https://dave.ml/mmcc
arXiv commentICCV 2021
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Unsupervised Learning of Video Representations using LSTMs
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- Deep multi-scale video prediction beyond mean square error
- Video (language) modeling: a baseline for generative models of natural videos
- Unsupervised Learning for Physical Interaction through Video Prediction
- Learning Video Representations using Contrastive Bidirectional Transformer
- Towards Automatic Learning of Procedures from Web Instructional Videos
- Learning to Generate Long-term Future via Hierarchical Prediction
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Self-Supervised Visual Planning with Temporal Skip Connections
- Memory-augmented Dense Predictive Coding for Video Representation Learning
- Stochastic Variational Video Prediction
- End-to-End Neural Sentence Ordering Using Pointer Network
- Stochastic Prediction of Multi-Agent Interactions from Partial Observations