Guiding Video Prediction with Explicit Procedural Knowledge

Abstract

We propose a general way to integrate procedural knowledge of a domain into deep learning models. We apply it to the case of video prediction, building on top of object-centric deep models and show that this leads to a better performance than using data-driven models alone. We develop an architecture that facilitates latent space disentanglement in order to use the integrated procedural knowledge, and establish a setup that allows the model to learn the procedural interface in the latent space using the downstream task of video prediction. We contrast the performance to a state-of-the-art data-driven approach and show that problems where purely data-driven approaches struggle can be handled by using knowledge about the domain, providing an alternative to simply collecting more data.

Published in 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

References in corpus (10)

• Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
• Deep Learning for Physical Processes: Incorporating Prior Scientific Knowledge
• The three-body problem
• Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Physics-Guided Deep Learning for Dynamical Systems: A Survey
• Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey
• A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
• PDE-Driven Spatiotemporal Disentanglement
• Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODE