Learning to Optimize with Dynamic Mode Decomposition

Abstract

Designing faster optimization algorithms is of ever-growing interest. In recent years, learning to learn methods that learn how to optimize demonstrated very encouraging results. Current approaches usually do not effectively include the dynamics of the optimization process during training. They either omit it entirely or only implicitly assume the dynamics of an isolated parameter. In this paper, we show how to utilize the dynamic mode decomposition method for extracting informative features about optimization dynamics. By employing those features, we show that our learned optimizer generalizes much better to unseen optimization problems in short. The improved generalization is illustrated on multiple tasks where training the optimizer on one neural network generalizes to different architectures and distinct datasets.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• A Data-Driven Approximation of the Koopman Operator: Extending Dynamic Mode Decomposition
• On Dynamic Mode Decomposition: Theory and Applications
• Learning to Learn without Gradient Descent by Gradient Descent
• Learning Gradient Descent: Better Generalization and Longer Horizons
• Self-referential basis of undecidable dynamics: from The Liar Paradox and The Halting Problem to The Edge of Chaos
• Online dynamic mode decomposition for time-varying systems
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Applications of Koopman Mode Analysis to Neural Networks
• Safeguarded Learned Convex Optimization