Neural Networks Trained to Solve Differential Equations Learn General Representations

Abstract

We introduce a technique based on the singular vector canonical correlation analysis (SVCCA) for measuring the generality of neural network layers across a continuously-parametrized set of tasks. We illustrate this method by studying generality in neural networks trained to solve parametrized boundary value problems based on the Poisson partial differential equation. We find that the first hidden layer is general, and that deeper layers are successively more specific. Next, we validate our method against an existing technique that measures layer generality using transfer learning experiments. We find excellent agreement between the two methods, and note that our method is much faster, particularly for continuously-parametrized problems. Finally, we visualize the general representations of the first layers, and interpret them as generalized coordinates over the input domain.

arXiv comment14 pages, 9 figures. Submitted to NIPS 2018

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• DGM: A deep learning algorithm for solving partial differential equations
• A unified deep artificial neural network approach to partial differential equations in complex geometries
• Convergent Learning: Do different neural networks learn the same representations?

Cited by in corpus (19)

• Weak Adversarial Networks for High-dimensional Partial Differential Equations
• Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learning
• Deep splitting method for parabolic PDEs
• Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers
• Hamiltonian neural networks for solving equations of motion
• An overview on deep learning-based approximation methods for partial differential equations
• Physical Symmetries Embedded in Neural Networks
• Overcoming the curse of dimensionality in the numerical approximation of Allen-Cahn partial differential equations via truncated full-history recursive multilevel Picard approximations
• Numerical simulations for full history recursive multilevel Picard approximations for systems of high-dimensional partial differential equations
• Uniform error estimates for artificial neural network approximations for heat equations
• Hypersolvers: Toward Fast Continuous-Depth Models
• Deep neural network approximations for Monte Carlo algorithms
• Inference of neutrino flavor evolution through data assimilation and neural differential equations
• Unsupervised Neural Networks for Quantum Eigenvalue Problems
• Multilevel Picard approximations for high-dimensional semilinear second-order PDEs with Lipschitz nonlinearities
• Semi-supervised Neural Networks solve an inverse problem for modeling Covid-19 spread
• Feature Engineering with Regularity Structures
• Revisiting Hidden Representations in Transfer Learning for Medical Imaging
• Local error quantification for Neural Network Differential Equation solvers