Parseval Proximal Neural Networks

Abstract

The aim of this paper is twofold. First, we show that a certain concatenation of a proximity operator with an affine operator is again a proximity operator on a suitable Hilbert space. Second, we use our findings to establish so-called proximal neural networks (PNNs) and stable tight frame proximal neural networks. Let $\mathcal{H}$ and $\mathcal{K}$ be real Hilbert spaces, $b\in \mathcal{K}$ and $T\in \mathcal{B}(\mathcal{H},\mathcal{K})$ have closed range and Moore-Penrose inverse $T^{†}$. Based on the well-known characterization of proximity operators by Moreau, we prove that for any proximity operator $\text{Prox}:\mathcal{K}\to \mathcal{K}$ the operator $T^{†}\text{Prox}(T\cdot +b)$ is a proximity operator on $\mathcal{H}$ equipped with a suitable norm. In particular, it follows for the frequently applied soft shrinkage operator $\text{Prox}=S_{\lambda }:{\ell }_2\to {\ell }_2$ and any frame analysis operator $T:\mathcal{H}\to {\ell }_2$ that the frame shrinkage operator $T^{†}{S}_{\lambda }T$ is a proximity operator on a suitable Hilbert space. The concatenation of proximity operators on ${\mathbb{R}}^d$ equipped with different norms establishes a PNN. If the network arises from tight frame analysis or synthesis operators, then it forms an averaged operator. Hence, it has Lipschitz constant 1 and belongs to the class of so-called Lipschitz networks, which were recently applied to defend against adversarial attacks. Moreover, due to its averaging property, PNNs can be used within so-called Plug-and-Play algorithms with convergence guarantee. In case of Parseval frames, we call the networks Parseval proximal neural networks (PPNNs). Then, the involved linear operators are in a Stiefel manifold and corresponding minimization methods can be applied for training. Finally, some proof-of-the concept examples demonstrate the performance of PPNNs.

arXiv commentarXiv admin note: text overlap with arXiv:1910.02843

References in corpus (16)

• Spectral Normalization for Generative Adversarial Networks
• Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation
• Unitary Evolution Recurrent Neural Networks
• An Online Plug-and-Play Algorithm for Regularized Image Reconstruction
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• On orthogonality and learning recurrent networks with long term dependencies
• The Singular Values of Convolutional Layers
• An Overview of Robust Subspace Recovery
• Full-Capacity Unitary Recurrent Neural Networks
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Sorting out Lipschitz function approximation
• Generalized BackPropagation, Étude De Cas: Orthogonality
• DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation
• Proximal Backpropagation

Cited by in corpus (15)

• Fixed Point Strategies in Data Science
• Convolutional Proximal Neural Networks and Plug-and-Play Algorithms
• Stabilizing Invertible Neural Networks Using Mixture Models
• Generalized Normalizing Flows via Markov Chains
• Inertial Stochastic PALM (iSPALM) and Applications in Machine Learning
• NESTANets: Stable, accurate and efficient neural networks for analysis-sparse inverse problems
• Designing Stable Neural Networks using Convex Analysis and ODEs
• Deep Transform and Metric Learning Network: Wedding Deep Dictionary Learning and Neural Networks
• Approximal operator with application to audio inpainting
• Motion Detection in Diffraction Tomography by Common Circle Methods
• Translating Diffusion, Wavelets, and Regularisation into Residual Networks
• Inversion of Integral Models: a Neural Network Approach
• Proximal Residual Flows for Bayesian Inverse Problems
• Stability Bounds for the Unfolded Forward-Backward Algorithm
• On $\alpha$-Firmly Nonexpansive Operators in $r$-Uniformly Convex Spaces