Translating Diffusion, Wavelets, and Regularisation into Residual Networks
arXiv:2002.02753
Abstract
Convolutional neural networks (CNNs) often perform well, but their stability is poorly understood. To address this problem, we consider the simple prototypical problem of signal denoising, where classical approaches such as nonlinear diffusion, wavelet-based methods and regularisation offer provable stability guarantees. To transfer such guarantees to CNNs, we interpret numerical approximations of these classical methods as a specific residual network (ResNet) architecture. This leads to a dictionary which allows to translate diffusivities, shrinkage functions, and regularisers into activation functions, and enables a direct communication between the four research communities. On the CNN side, it does not only inspire new families of nonmonotone activation functions, but also introduces intrinsically stable architectures for an arbitrary number of layers.
References in corpus (16)
- Deep Learning in Neural Networks: An Overview
- Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration
- Opening the Black Box of Deep Neural Networks via Information
- PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network
- Regularization for Deep Learning: A Taxonomy
- Essentially No Barriers in Neural Network Energy Landscape
- The loss surface of deep and wide neural networks
- Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
- The power of deeper networks for expressing natural functions
- Wavelet Convolutional Neural Networks
- Mathematics of Deep Learning
- PDE-based Group Equivariant Convolutional Neural Networks
- Parseval Proximal Neural Networks
- Deep Residual Learning and PDEs on Manifold
- Approximation spaces of deep neural networks
- Analysis of Invariance and Robustness via Invertibility of ReLU-Networks