Data-driven Algorithm Selection and Parameter Tuning: Two Case studies in Optimization and Signal Processing
arXiv:1905.13404
Abstract
Machine learning algorithms typically rely on optimization subroutines and are well-known to provide very effective outcomes for many types of problems. Here, we flip the reliance and ask the reverse question: can machine learning algorithms lead to more effective outcomes for optimization problems? Our goal is to train machine learning methods to automatically improve the performance of optimization and signal processing algorithms. As a proof of concept, we use our approach to improve two popular data processing subroutines in data science: stochastic gradient descent and greedy methods in compressed sensing. We provide experimental results that demonstrate the answer is ``yes'', machine learning algorithms do lead to more effective outcomes for optimization problems, and show the future potential for this research direction.
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- ADADELTA: An Adaptive Learning Rate Method
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
- Optimally Tuned Iterative Reconstruction Algorithms for Compressed Sensing
- No More Pesky Learning Rates
- Barzilai-Borwein Step Size for Stochastic Gradient Descent
- OBOE: Collaborative Filtering for AutoML Model Selection
- Learning to Branch
- WNGrad: Learn the Learning Rate in Gradient Descent
- Big Batch SGD: Automated Inference using Adaptive Batch Sizes
- Practical approximate projection schemes in greedy signal space methods
- Speed learning on the fly
- AdaBatch: Efficient Gradient Aggregation Rules for Sequential and Parallel Stochastic Gradient Methods