Discovering Symbolic Models from Deep Learning with Inductive Biases

Abstract

We develop a general approach to distill symbolic representations of a learned deep model by introducing strong inductive biases. We focus on Graph Neural Networks (GNNs). The technique works as follows: we first encourage sparse latent representations when we train a GNN in a supervised setting, then we apply symbolic regression to components of the learned model to extract explicit physical relations. We find the correct known equations, including force laws and Hamiltonians, can be extracted from the neural network. We then apply our method to a non-trivial cosmology example-a detailed dark matter simulation-and discover a new analytic formula which can predict the concentration of dark matter from the mass distribution of nearby cosmic structures. The symbolic expressions extracted from the GNN using our technique also generalized to out-of-distribution data better than the GNN itself. Our approach offers alternative directions for interpreting neural networks and discovering novel physical principles from the representations they learn.

arXiv commentAccepted to NeurIPS 2020. 9 pages content + 16 pages appendix/references. Supporting code found at https://github.com/MilesCranmer/symbolic_deep_learning

References in corpus (29)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• The NumPy array: a structure for efficient numerical computation
• First Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Determination of Cosmological Parameters
• Discovering governing equations from data: Sparse identification of nonlinear dynamical systems
• Neural Message Passing for Quantum Chemistry
• Geometric deep learning: going beyond Euclidean data
• Fast Graph Representation Learning with PyTorch Geometric
• On the Number of Linear Regions of Deep Neural Networks
• GNNExplainer: Generating Explanations for Graph Neural Networks
• Interaction Networks for Learning about Objects, Relations and Physics
• Hamiltonian Neural Networks
• Learning to Simulate Complex Physics with Graph Networks
• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Graph networks as learnable physics engines for inference and control
• Neural Relational Inference for Interacting Systems
• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
• Flexible Neural Representation for Physics Prediction
• A Bayesian machine scientist to aid in the solution of challenging scientific problems
• DeepMoD: Deep learning for Model Discovery in noisy data
• What Can Neural Networks Reason About?
• Hamiltonian Graph Networks with ODE Integrators
• Feature engineering and symbolic regression methods for detecting hidden physics from sparse sensors
• Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
• Data-driven discovery of free-form governing differential equations
• Learning Symbolic Physics with Graph Networks
• Hamiltonian Generative Networks
• B-Spline CNNs on Lie Groups
• Neural-Guided Symbolic Regression with Asymptotic Constraints

Cited by in corpus (67)

• Enhancing Computational Fluid Dynamics with Machine Learning
• On scientific understanding with artificial intelligence
• SELFIES and the future of molecular string representations
• AI Poincaré: Machine Learning Conservation Laws from Trajectories
• Thermodynamically Consistent Machine-Learned Internal State Variable Approach for Data-Driven Modeling of Path-Dependent Materials
• Learning hydrodynamic equations for active matter from particle simulations and experiments
• Hypothesis Learning in Automated Experiment: Application to Combinatorial Materials Libraries
• Data-driven discovery of reduced plasma physics models from fully-kinetic simulations
• The GIGANTES dataset: precision cosmology from voids in the machine learning era
• Lagrangian Neural Networks
• Physics-Informed Deep Learning For Traffic State Estimation: A Survey and the Outlook
• The Ramanujan Machine: Automatically Generated Conjectures on Fundamental Constants
• Thermodynamics-informed graph neural networks
• Six decades of the FitzHugh-Nagumo model: A guide through its spatio-temporal dynamics and influence across disciplines
• An Interdisciplinary Survey on Origin-destination Flows Modeling: Theory and Techniques
• Artificial Neural Networks Modelling of Wall Pressure Spectra Beneath Turbulent Boundary Layers
• Physics-enhanced deep surrogates for partial differential equations
• Discovering interpretable elastoplasticity models via the neural polynomial method enabled symbolic regressions
• Physical Activation Functions (PAFs): An Approach for More Efficient Induction of Physics into Physics-Informed Neural Networks (PINNs)
• The Cosmic Graph: Optimal Information Extraction from Large-Scale Structure using Catalogues
• gLaSDI: Parametric Physics-informed Greedy Latent Space Dynamics Identification
• Deep reconstruction of strange attractors from time series
• Manifold embedding data-driven mechanics
• Discrete and continuous representations and processing in deep learning: Looking forward
• Augmenting astrophysical scaling relations with machine learning: application to reducing the Sunyaev-Zeldovich flux-mass scatter
• Symbolic regression outperforms other models for small data sets
• The SZ flux-mass ($Y$-$M$) relation at low halo masses: improvements with symbolic regression and strong constraints on baryonic feedback
• Neural Symbolic Regression that Scales
• Machine-Learning Non-Conservative Dynamics for New-Physics Detection
• AlphaEvolve: A Learning Framework to Discover Novel Alphas in Quantitative Investment
• Learning Closed-form Equations for Subgrid-scale Closures from High-fidelity Data: Promises and Challenges
• Machine-Guided Discovery of a Real-World Rogue Wave Model
• On the functional form of the radial acceleration relation
• A precise symbolic emulator of the linear matter power spectrum
• Back to the Formula -- LHC Edition
• Towards Human-like Perception: Learning Structural Causal Model in Heterogeneous Graph
• Machine Learning and Cosmology
• Modeling assembly bias with machine learning and symbolic regression
• Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
• Integrability ex machina
• Physics-Aware Neural Implicit Solvers for multiscale, parametric PDEs with applications in heterogeneous media
• Graph Laplacians, Riemannian Manifolds and their Machine-Learning
• Machine learning depinning of dislocation pileups
• Sparse Symplectically Integrated Neural Networks
• Physics-informed active learning with simultaneous weak-form latent space dynamics identification
• From Kepler to Newton: Explainable AI for Science
• How wavelength affects the hydrodynamic performance of two accelerating mirror-symmetric slender swimmers
• Deep learning complete intersection Calabi-Yau manifolds
• Utilising Graph Machine Learning within Drug Discovery and Development
• Discovering Nuclear Models from Symbolic Machine Learning
• Backpropagation on Dynamical Networks
• Neural Network Predicts Ion Concentration Profiles under Nanoconfinement
• On Computational Poisson Geometry II: Numerical Methods
• An Extensible Benchmark Suite for Learning to Simulate Physical Systems
• Kernel-based parameter estimation of dynamical systems with unknown observation functions
• Physics-Guided Discovery of Highly Nonlinear Parametric Partial Differential Equations
• Data-Driven Modeling of an Unsaturated Bentonite Buffer Model Test Under High Temperatures Using an Enhanced Axisymmetric Reproducing Kernel Particle Method
• Learning from learning machines: a new generation of AI technology to meet the needs of science
• A Bayesian neural network predicts the dissolution of compact planetary systems
• Hybrid analysis and modeling, eclecticism, and multifidelity computing toward digital twin revolution
• Reverse engineering learned optimizers reveals known and novel mechanisms
• There is No Such Thing as an "Index"! or: The next 500 Indexing Papers
• Grammar-based Ordinary Differential Equation Discovery
• Worrisome Properties of Neural Network Controllers and Their Symbolic Representations
• Identifying Physical Law of Hamiltonian Systems via Meta-Learning
• Discovering Latent Representations of Relations for Interacting Systems
• Physical discovery in representation learning via conditioning on prior knowledge: applications for ferroelectric domain dynamics