Learning to Optimize: A Primer and A Benchmark

Abstract

Learning to optimize (L2O) is an emerging approach that leverages machine learning to develop optimization methods, aiming at reducing the laborious iterations of hand engineering. It automates the design of an optimization method based on its performance on a set of training problems. This data-driven procedure generates methods that can efficiently solve problems similar to those in the training. In sharp contrast, the typical and traditional designs of optimization methods are theory-driven, so they obtain performance guarantees over the classes of problems specified by the theory. The difference makes L2O suitable for repeatedly solving a certain type of optimization problems over a specific distribution of data, while it typically fails on out-of-distribution problems. The practicality of L2O depends on the type of target optimization, the chosen architecture of the method to learn, and the training procedure. This new paradigm has motivated a community of researchers to explore L2O and report their findings. This article is poised to be the first comprehensive survey and benchmark of L2O for continuous optimization. We set up taxonomies, categorize existing works and research directions, present insights, and identify open challenges. We also benchmarked many existing L2O approaches on a few but representative optimization problems. For reproducible research and fair benchmarking purposes, we released our software implementation and data in the package Open-L2O at https://github.com/VITA-Group/Open-L2O.

References in corpus (80)

• Conditional Random Fields as Recurrent Neural Networks
• Neural Architecture Search: A Survey
• AutoML: A Survey of the State-of-the-Art
• Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration
• Learning Combinatorial Optimization Algorithms over Graphs
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network
• Denoising Prior Driven Deep Neural Network for Image Restoration
• AMP-Inspired Deep Networks for Sparse Linear Inverse Problems
• Model-Driven Deep Learning for MIMO Detection
• Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Search
• Compressed Sensing using Generative Models
• Plug and play methods for magnetic resonance imaging (long version)
• Real-time Power System State Estimation and Forecasting via Deep Neural Networks
• Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures
• Trainable ISTA for Sparse Signal Recovery
• An Online Plug-and-Play Algorithm for Regularized Image Reconstruction
• Neural Optimizer Search with Reinforcement Learning
• Learned D-AMP: Principled Neural Network based Compressive Image Recovery
• Learning to Learn without Gradient Descent by Gradient Descent
• Compressed Sensing with Deep Image Prior and Learned Regularization
• Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds
• Learned Convolutional Sparse Coding
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• Unrolled Optimization with Deep Priors
• Neural Proximal Gradient Descent for Compressive Imaging
• Maximal Sparsity with Deep Networks?
• Learned Optimizers that Scale and Generalize
• Plug-and-Play Methods Provably Converge with Properly Trained Denoisers
• Recurrent Inference Machines for Solving Inverse Problems
• Input Warping for Bayesian Optimization of Non-stationary Functions
• Learning to Optimize Neural Nets
• Learning Converged Propagations with Deep Prior Ensemble for Image Enhancement
• Deep Mean-Shift Priors for Image Restoration
• A Model for Learned Bloom Filters, and Optimizing by Sandwiching
• One Network to Solve Them All --- Solving Linear Inverse Problems using Deep Projection Models
• The large learning rate phase of deep learning: the catapult mechanism
• Learned SPARCOM: Unfolded Deep Super-Resolution Microscopy
• Learning Gradient Descent: Better Generalization and Longer Horizons
• Understanding and correcting pathologies in the training of learned optimizers
• Plug-and-play ISTA converges with kernel denoisers
• Learning a Compressed Sensing Measurement Matrix via Gradient Unrolling
• Learning an Adaptive Learning Rate Schedule
• Neural Networks-based Regularization for Large-Scale Medical Image Reconstruction
• RNA Secondary Structure Prediction By Learning Unrolled Algorithms
• DeepRED: Deep Image Prior Powered by RED
• Better Software Analytics via "DUO": Data Mining Algorithms Using/Used-by Optimizers
• End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction
• Learning to Defend by Learning to Attack
• Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training
• Differentiable Linearized ADMM
• Learning-Based Low-Rank Approximations
• Learning Efficient Structured Sparse Models
• Learned convex regularizers for inverse problems
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Tuning-free Plug-and-Play Proximal Algorithm for Inverse Imaging Problems
• Using learned optimizers to make models robust to input noise
• Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding
• Total Deep Variation: A Stable Regularizer for Inverse Problems
• The Two Regimes of Deep Network Training
• Automated Synthetic-to-Real Generalization
• Theoretical Interpretation of Learned Step Size in Deep-Unfolded Gradient Descent
• Understanding the Learned Iterative Soft Thresholding Algorithm with matrix factorization
• Learning to Learn by Zeroth-Order Oracle
• Deep Learning-Aided Trainable Projected Gradient Decoding for LDPC Codes
• Learnable Descent Algorithm for Nonsmooth Nonconvex Image Reconstruction
• Degrees of Freedom Analysis of Unrolled Neural Networks
• Training Stronger Baselines for Learning to Optimize
• Async-RED: A Provably Convergent Asynchronous Block Parallel Stochastic Method using Deep Denoising Priors
• Guarantees for Tuning the Step Size using a Learning-to-Learn Approach
• Scalable Plug-and-Play ADMM with Convergence Guarantees
• A Generalizable Approach to Learning Optimizers
• Safeguarded Learned Convex Optimization
• Iterative Amortized Policy Optimization
• Data Consistent CT Reconstruction from Insufficient Data with Learned Prior Images
• A Design Space Study for LISTA and Beyond
• Deep Unfolding of a Proximal Interior Point Method for Image Restoration
• Learning the Step-size Policy for the Limited-Memory Broyden-Fletcher-Goldfarb-Shanno Algorithm
• Understanding Deep Architectures with Reasoning Layer
• Consensus Multiplicative Weights Update: Learning to Learn using Projector-based Game Signatures

Cited by in corpus (15)

• An Adaptive and Robust Deep Learning Framework for THz Ultra-Massive MIMO Channel Estimation
• Scalable Optimal Transport Methods in Machine Learning: A Contemporary Survey
• Learning to Optimize in Model Predictive Control
• Expert-Calibrated Learning for Online Optimization with Switching Costs
• Operator Splitting for Learning to Predict Equilibria in Convex Games
• TFPnP: Tuning-free Plug-and-Play Proximal Algorithm with Applications to Inverse Imaging Problems
• Feasibility-based Fixed Point Networks
• SF-SGL: Solver-Free Spectral Graph Learning from Linear Measurements
• Methods for Recovering Conditional Independence Graphs: A Survey
• Meta-Learning the Search Distribution of Black-Box Random Search Based Adversarial Attacks
• Primitive Agentic First-Order Optimization
• Self-Learning Tuning for Post-Silicon Validation
• Learning to Learn Graph Topologies
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Robust Data-Driven Accelerated Mirror Descent