Designing Stable Neural Networks using Convex Analysis and ODEs

Abstract

Motivated by classical work on the numerical integration of ordinary differential equations we present a ResNet-styled neural network architecture that encodes non-expansive (1-Lipschitz) operators, as long as the spectral norms of the weights are appropriately constrained. This is to be contrasted with the ordinary ResNet architecture which, even if the spectral norms of the weights are constrained, has a Lipschitz constant that, in the worst case, grows exponentially with the depth of the network. Further analysis of the proposed architecture shows that the spectral norms of the weights can be further constrained to ensure that the network is an averaged operator, making it a natural candidate for a learned denoiser in Plug-and-Play algorithms. Using a novel adaptive way of enforcing the spectral norm constraints, we show that, even with these constraints, it is possible to train performant networks. The proposed architecture is applied to the problem of adversarially robust image classification, to image denoising, and finally to the inverse problem of deblurring.

34 pages, 6 figures. This is the accepted version of a paper published in Physica D: Nonlinear Phenomena

References in corpus (8)

• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Spectral Normalization for Generative Adversarial Networks
• Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation
• An Online Plug-and-Play Algorithm for Regularized Image Reconstruction
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• Lipschitz regularity of deep neural networks: analysis and efficient estimation
• Parseval Proximal Neural Networks
• Deep neural networks on diffeomorphism groups for optimal shape reparameterization