A Neural-Symbolic Framework for Mental Simulation
arXiv:2008.02356
Abstract
We present a neural-symbolic framework for observing the environment and continuously learning visual semantics and intuitive physics to reproduce them in an interactive simulation. The framework consists of five parts, a neural-symbolic hybrid network based on capsules for inverse graphics, an episodic memory to store observations, an interaction network for intuitive physics, a meta-learning agent that continuously improves the framework and a querying language that acts as the framework's interface for simulation. By means of lifelong meta-learning, the capsule network is expanded and trained continuously, in order to better adapt to its environment with each iteration. This enables it to learn new semantics using a few-shot approach and with minimal input from an oracle over its lifetime. From what it learned through observation, the part for intuitive physics infers all the required physical properties of the objects in a scene, enabling predictions. Finally, a custom query language ties all parts together, which allows to perform various mental simulation tasks, such as navigation, sorting and simulation of a game environment, with which we illustrate the potential of our novel approach.
arXiv commentDissertation
References in corpus (12)
- Auto-Encoding Variational Bayes
- How transferable are features in deep neural networks?
- Methods for Interpreting and Understanding Deep Neural Networks
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Graph networks as learnable physics engines for inference and control
- Non-Stationary Texture Synthesis by Adversarial Expansion
- Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications
- Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
- Machine Teaching: A New Paradigm for Building Machine Learning Systems
- Learning to Infer and Execute 3D Shape Programs
- CapsAcc: An Efficient Hardware Accelerator for CapsuleNets with Data Reuse
- OD-GCN: Object Detection Boosted by Knowledge GCN