Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures

Abstract

We present a general variational framework for the training of freeform nonlinearities in layered computational architectures subject to some slope constraints. The regularization that we add to the traditional training loss penalizes the second-order total variation of each trainable activation. The slope constraints allow us to impose properties such as 1-Lipschitz stability, firm non-expansiveness, and monotonicity/invertibility. These properties are crucial to ensure the proper functioning of certain classes of signal-processing algorithms (e.g., plug-and-play schemes, unrolled proximal gradient, invertible flows). We prove that the global optimum of the stated constrained-optimization problem is achieved with nonlinearities that are adaptive nonuniform linear splines. We then show how to solve the resulting function-optimization problem numerically by representing the nonlinearities in a suitable (nonuniform) B-spline basis. Finally, we illustrate the use of our framework with the data-driven design of (weakly) convex regularizers for the denoising of images and the resolution of inverse problems.

References in corpus (11)

• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Sparsity and Incoherence in Compressive Sampling
• Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration
• MoDL: Model Based Deep Learning Architecture for Inverse Problems
• A survey on modern trainable activation functions
• Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imaging
• The Role of Neural Network Activation Functions
• Insights into analysis operator learning: From patch-based sparse models to higher-order MRFs
• Splines are Universal Solutions of Linear Inverse Problems with Generalized-TV regularization
• Continuous-Domain Solutions of Linear Inverse Problems with Tikhonov vs. Generalized TV Regularization
• Learning Convex Regularizers for Optimal Bayesian Denoising