Differential equations as models of deep neural networks
arXiv:1909.03767
Abstract
In this work we systematically analyze general properties of differential equations used as machine learning models. We demonstrate that the gradient of the loss function with respect to to the hidden state can be considered as a generalized momentum conjugate to the hidden state, allowing application of the tools of classical mechanics. In addition, we show that not only residual networks, but also feedforward neural networks with small nonlinearities and the weights matrices deviating only slightly from identity matrices can be related to the differential equations. We propose a differential equation describing such networks and investigate its properties.
References in corpus (9)
- Improving neural networks by preventing co-adaptation of feature detectors
- PDE-Net: Learning PDEs from Data
- Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
- Invertible Residual Networks
- Latent ODEs for Irregularly-Sampled Time Series
- Reversible Architectures for Arbitrarily Deep Residual Neural Networks
- Augmented Neural ODEs
- Deep Residual Learning and PDEs on Manifold
- Recurrent Neural Networks in the Eye of Differential Equations