A Survey of Optimization Methods from a Machine Learning Perspective

Abstract

Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization methods in machine learning face more and more challenges. A lot of work on solving optimization problems or improving optimization methods in machine learning has been proposed successively. The systematic retrospect and summary of the optimization methods from the perspective of machine learning are of great significance, which can offer guidance for both developments of optimization and machine learning research. In this paper, we first describe the optimization problems in machine learning. Then, we introduce the principles and progresses of commonly used optimization methods. Next, we summarize the applications and developments of optimization methods in some popular machine learning fields. Finally, we explore and give some challenges and open problems for the optimization in machine learning.

References in corpus (38)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Continuous control with deep reinforcement learning
• Decoupled Weight Decay Regularization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• ADADELTA: An Adaptive Learning Rate Method
• Prototypical Networks for Few-shot Learning
• Playing Atari with Deep Reinforcement Learning
• An overview of gradient descent optimization algorithms
• Recurrent Neural Network Regularization
• On the Convergence of Adam and Beyond
• Pixel Recurrent Neural Networks
• Theoretical Models of Learning to Learn
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• DRAW: A Recurrent Neural Network For Image Generation
• Recent Advances in Recurrent Neural Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Deep Reinforcement Learning: An Overview
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Stochastic Gradient Hamiltonian Monte Carlo
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Gaussian Processes for Big Data
• Improving Generalization Performance by Switching from Adam to SGD
• Scalable Variational Gaussian Process Classification
• New insights and perspectives on the natural gradient method
• Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees
• RES: Regularized Stochastic BFGS Algorithm
• A Linearly-Convergent Stochastic L-BFGS Algorithm
• Global Convergence of Online Limited Memory BFGS
• Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring
• Parallelizing Stochastic Gradient Descent for Least Squares Regression: mini-batching, averaging, and model misspecification
• Stochastic Conjugate Gradient Algorithm with Variance Reduction
• Sub-Sampled Newton Methods II: Local Convergence Rates
• High Dimensional Expectation-Maximization Algorithm: Statistical Optimization and Asymptotic Normality
• Provable Burer-Monteiro factorization for a class of norm-constrained matrix problems
• Normalized Direction-preserving Adam
• An ADMM Algorithm for a Class of Total Variation Regularized Estimation Problems

Cited by in corpus (4)

• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
• CoRe Optimizer: An All-in-One Solution for Machine Learning
• A Continuous Convolutional Trainable Filter for Modelling Unstructured Data
• Effective Multi-Stage Training Model For Edge Computing Devices In Intrusion Detection