IMEXnet: A Forward Stable Deep Neural Network

Abstract

Deep convolutional neural networks have revolutionized many machine learning and computer vision tasks, however, some remaining key challenges limit their wider use. These challenges include improving the network's robustness to perturbations of the input image and the limited ``field of view'' of convolution operators. We introduce the IMEXnet that addresses these challenges by adapting semi-implicit methods for partial differential equations. Compared to similar explicit networks, such as residual networks, our network is more stable, which has recently shown to reduce the sensitivity to small changes in the input features and improve generalization. The addition of an implicit step connects all pixels in each channel of the image and therefore addresses the field of view problem while still being comparable to standard convolutions in terms of the number of parameters and computational complexity. We also present a new dataset for semantic segmentation and demonstrate the effectiveness of our architecture using the NYU Depth dataset.

References in corpus (1)

• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations

Cited by in corpus (13)

• Dissecting Neural ODEs
• Latent Space Data Assimilation by using Deep Learning
• Lipschitz Recurrent Neural Networks
• Constrained Neural Ordinary Differential Equations with Stability Guarantees
• PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations
• Robust learning with implicit residual networks
• LeanConvNets: Low-cost Yet Effective Convolutional Neural Networks
• Towards Robust and Stable Deep Learning Algorithms for Forward Backward Stochastic Differential Equations
• PNKH-B: A Projected Newton-Krylov Method for Large-Scale Bound-Constrained Optimization
• ResNet After All? Neural ODEs and Their Numerical Solution
• Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equations
• Stability of implicit neural networks for long-term forecasting in dynamical systems
• Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families