Time-Agnostic Prediction: Predicting Predictable Video Frames

Abstract

Prediction is arguably one of the most basic functions of an intelligent system. In general, the problem of predicting events in the future or between two waypoints is exceedingly difficult. However, most phenomena naturally pass through relatively predictable bottlenecks---while we cannot predict the precise trajectory of a robot arm between being at rest and holding an object up, we can be certain that it must have picked the object up. To exploit this, we decouple visual prediction from a rigid notion of time. While conventional approaches predict frames at regularly spaced temporal intervals, our time-agnostic predictors (TAP) are not tied to specific times so that they may instead discover predictable "bottleneck" frames no matter when they occur. We evaluate our approach for future and intermediate frame prediction across three robotic manipulation tasks. Our predictions are not only of higher visual quality, but also correspond to coherent semantic subgoals in temporally extended tasks.

arXiv comment8 pages, plus appendices

References in corpus (13)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Generating Videos with Scene Dynamics
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Deep multi-scale video prediction beyond mean square error
• Stochastic Video Generation with a Learned Prior
• Video (language) modeling: a baseline for generative models of natural videos
• Unsupervised Learning for Physical Interaction through Video Prediction
• Stochastic Adversarial Video Prediction
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Self-Supervised Visual Planning with Temporal Skip Connections

Cited by in corpus (15)

• Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction
• Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
• Temporal Difference Variational Auto-Encoder
• Procedure Planning in Instructional Videos
• Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models
• Vid2Param: Modelling of Dynamics Parameters from Video
• Identifying Critical States by the Action-Based Variance of Expected Return
• Temporal Dynamic Model for Resting State fMRI Data: A Neural Ordinary Differential Equation approach
• Causal Future Prediction in a Minkowski Space-Time
• Beta DVBF: Learning State-Space Models for Control from High Dimensional Observations
• On the potential for open-endedness in neural networks
• Point-to-Point Video Generation
• Stochastic Dynamics for Video Infilling
• Variational Predictive Routing with Nested Subjective Timescales
• Generating the support with extreme value losses