Theoretical Interpretation of Learned Step Size in Deep-Unfolded Gradient Descent

Abstract

Deep unfolding is a promising deep-learning technique in which an iterative algorithm is unrolled to a deep network architecture with trainable parameters. In the case of gradient descent algorithms, as a result of the training process, one often observes the acceleration of the convergence speed with learned non-constant step size parameters whose behavior is not intuitive nor interpretable from conventional theory. In this paper, we provide a theoretical interpretation of the learned step size of deep-unfolded gradient descent (DUGD). We first prove that the training process of DUGD reduces not only the mean squared error loss but also the spectral radius related to the convergence rate. Next, we show that minimizing the upper bound of the spectral radius naturally leads to the Chebyshev step which is a sequence of the step size based on Chebyshev polynomials. The numerical experiments confirm that the Chebyshev steps qualitatively reproduce the learned step size parameters in DUGD, which provides a plausible interpretation of the learned parameters. Additionally, we show that the Chebyshev steps achieve the lower bound of the convergence rate for the first-order method in a specific limit without learning parameters or momentum terms.

arXiv comment12 pages, 12 figures, typos are fixed

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Deep Convolutional Neural Network for Inverse Problems in Imaging
• AMP-Inspired Deep Networks for Sparse Linear Inverse Problems
• Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures
• Trainable ISTA for Sparse Signal Recovery
• Learned D-AMP: Principled Neural Network based Compressive Image Recovery
• Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds
• Neural Proximal Gradient Descent for Compressive Imaging
• Learning a Compressed Sensing Measurement Matrix via Gradient Unrolling
• Chebyshev Inertial Iteration for Accelerating Fixed-Point Iterations
• Degrees of Freedom Analysis of Unrolled Neural Networks
• Chebyshev Inertial Landweber Algorithm for Linear Inverse Problems

Cited by in corpus (5)

• Learning to Optimize: A Primer and A Benchmark
• One-Bit Compressive Sensing: Can We Go Deep and Blind?
• Chebyshev Inertial Landweber Algorithm for Linear Inverse Problems
• A Design Space Study for LISTA and Beyond
• Deep Unfolded Multicast Beamforming