LagNetViP: A Lagrangian Neural Network for Video Prediction

Abstract

The dominant paradigms for video prediction rely on opaque transition models where neither the equations of motion nor the underlying physical quantities of the system are easily inferred. The equations of motion, as defined by Newton's second law, describe the time evolution of a physical system state and can therefore be applied toward the determination of future system states. In this paper, we introduce a video prediction model where the equations of motion are explicitly constructed from learned representations of the underlying physical quantities. To achieve this, we simultaneously learn a low-dimensional state representation and system Lagrangian. The kinetic and potential energy terms of the Lagrangian are distinctly modelled and the low-dimensional equations of motion are explicitly constructed using the Euler-Lagrange equations. We demonstrate the efficacy of this approach for video prediction on image sequences rendered in modified OpenAI gym Pendulum-v0 and Acrobot environments.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Neural Ordinary Differential Equations
• Hamiltonian Neural Networks
• Towards a Definition of Disentangled Representations
• Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Lagrangian Neural Networks
• Hamiltonian Generative Networks
• Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control

Cited by in corpus (7)

• Discrete Lagrangian Neural Networks with Automatic Symmetry Discovery
• Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modeling
• Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Models
• Learning discrete Lagrangians for variational PDEs from data and detection of travelling waves
• Benchmarking Energy-Conserving Neural Networks for Learning Dynamics from Data
• SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision
• KeyCLD: Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images