SlotPi: Physics-informed Object-centric Reasoning Models

Abstract

Understanding and reasoning about dynamics governed by physical laws through visual observation, akin to human capabilities in the real world, poses significant challenges. Currently, object-centric dynamic simulation methods, which emulate human behavior, have achieved notable progress but overlook two critical aspects: 1) the integration of physical knowledge into models. Humans gain physical insights by observing the world and apply this knowledge to accurately reason about various dynamic scenarios; 2) the validation of model adaptability across diverse scenarios. Real-world dynamics, especially those involving fluids and objects, demand models that not only capture object interactions but also simulate fluid flow characteristics. To address these gaps, we introduce SlotPi, a slot-based physics-informed object-centric reasoning model. SlotPi integrates a physical module based on Hamiltonian principles with a spatio-temporal prediction module for dynamic forecasting. Our experiments highlight the model's strengths in tasks such as prediction and Visual Question Answering (VQA) on benchmark and fluid datasets. Furthermore, we have created a real-world dataset encompassing object interactions, fluid dynamics, and fluid-object interactions, on which we validated our model's capabilities. The model's robust performance across all datasets underscores its strong adaptability, laying a foundation for developing more advanced world models.

References in corpus (15)

• Natural Scales in Geographical Patterns
• Social physics
• PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain
• PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network
• Learning data driven discretizations for partial differential equations
• PhyCRNet: Physics-informed Convolutional-Recurrent Network for Solving Spatiotemporal PDEs
• Learning Physical Graph Representations from Visual Scenes
• SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos
• Thermodynamics-informed neural networks for physically realistic mixed reality
• PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers
• Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos
• ROOTS: Object-Centric Representation and Rendering of 3D Scenes
• Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language
• Machine learning structure preserving brackets for forecasting irreversible processes
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition