Nonlinear Power Method for Computing Eigenvectors of Proximal Operators and Neural Networks

Abstract

Neural networks have revolutionized the field of data science, yielding remarkable solutions in a data-driven manner. For instance, in the field of mathematical imaging, they have surpassed traditional methods based on convex regularization. However, a fundamental theory supporting the practical applications is still in the early stages of development. We take a fresh look at neural networks and examine them via nonlinear eigenvalue analysis. The field of nonlinear spectral theory is still emerging, providing insights about nonlinear operators and systems. In this paper we view a neural network as a complex nonlinear operator and attempt to find its nonlinear eigenvectors. We first discuss the existence of such eigenvectors and analyze the kernel of ReLU networks. Then we study a nonlinear power method for generic nonlinear operators. For proximal operators associated to absolutely one-homogeneous convex regularization functionals, we can prove convergence of the method to an eigenvector of the proximal operator. This motivates us to apply a nonlinear method to networks which are trained to act similarly as a proximal operator. In order to take the non-homogeneity of neural networks into account we define a modified version of the power method. We perform extensive experiments for different proximal operators and on various shallow and deep neural networks designed for image denoising. Proximal eigenvectors will be used for geometric analysis of graphs, as clustering or the computation of distance functions. For simple neural nets, we observe the influence of training data on the eigenvectors. For state-of-the-art denoising networks, we show that eigenvectors can be interpreted as (un)stable modes of the network, when contaminated with noise or other degradations.

arXiv commentAccepted for publication in SIAM Journal on Imaging Sciences

References in corpus (15)

• Deep Learning in Neural Networks: An Overview
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising
• Understanding Deep Convolutional Networks
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• An Inverse Power Method for Nonlinear Eigenproblems with Applications in 1-Spectral Clustering and Sparse PCA
• The Perron-Frobenius theorem for multi-homogeneous mappings
• Deep Limits of Residual Neural Networks
• Nonlinear Spectral Decompositions by Gradient Flows of One-Homogeneous Functionals
• Energy Models for Better Pseudo-Labels: Improving Semi-Supervised Classification with the 1-Laplacian Graph Energy
• Structural analysis of an $L$-infinity variational problem and relations to distance functions
• Computing the norm of nonnegative matrices and the log-Sobolev constant of Markov chains
• An Optimal Control Approach to Early Stopping Variational Methods for Image Restoration
• Revealing Stable and Unstable Modes of Generic Denoisers through Nonlinear Eigenvalue Analysis
• Polyhedron under Linear Transformations

Cited by in corpus (2)

• Gradient Flows and Nonlinear Power Methods for the Computation of Nonlinear Eigenfunctions
• The Infinity Laplacian eigenvalue problem: reformulation and a numerical scheme