Neural Conditional Gradients
arXiv:1803.04300
Abstract
The move from hand-designed to learned optimizers in machine learning has been quite successful for gradient-based and -free optimizers. When facing a constrained problem, however, maintaining feasibility typically requires a projection step, which might be computationally expensive and not differentiable. We show how the design of projection-free convex optimization algorithms can be cast as a learning problem based on Frank-Wolfe Networks: recurrent networks implementing the Frank-Wolfe algorithm aka. conditional gradients. This allows them to learn to exploit structure when, e.g., optimizing over rank-1 matrices. Our LSTM-learned optimizers outperform hand-designed as well learned but unconstrained ones. We demonstrate this for training support vector machines and softmax classifiers.
arXiv commentarXiv admin note: text overlap with arXiv:1610.05120 by other authors
References in corpus (9)
- Matching Networks for One Shot Learning
- Deep Learning using Linear Support Vector Machines
- Learning to reinforcement learn
- Learning to Learn without Gradient Descent by Gradient Descent
- OptNet: Differentiable Optimization as a Layer in Neural Networks
- Learning to Optimize Neural Nets
- Frank-Wolfe Bayesian Quadrature: Probabilistic Integration with Theoretical Guarantees
- A Distributed Frank-Wolfe Framework for Learning Low-Rank Matrices with the Trace Norm
- Complexity Issues and Randomization Strategies in Frank-Wolfe Algorithms for Machine Learning